AI Engineer Europe 2026
Agentic Search for Context Engineering
Read the talk
Agentic Search for Context Engineering
Choosing what enters an agent’s context window requires more than vector search: the right tools must support exact lookup, query generation, aggregation and retrieval from files.
From a talk by Leonie Monigatti
Before you start: Familiarity with basic RAG, LLM tool calling and Python will help you follow the examples.
What belongs in the context window?
An agent can draw on code files, databases, the web and memory. Which pieces should enter its context window so it can answer the current question? That is the opening problem in Leonie Monigatti’s workshop. Working at Elastic, the company behind Elasticsearch, she focuses on the mechanism inside the arrow between available context and selected context: search tools.
Her framing—that context engineering is about eighty percent agentic search—is a personal hot take, not a measured allocation of engineering work. Its practical point is that choosing context depends on how the agent can find it. A larger collection of sources does little good if the available search interface cannot retrieve the relevant material.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a fixed retrieval step to a choice of tools
The familiar starting point is a fixed RAG pipeline: use the user’s message almost verbatim as a vector-search query, retrieve chunks, and send those chunks alongside the message to the LLM. This makes retrieval mandatory even when the question needs no additional context. Unnecessary results can confuse the model. Conversely, a single retrieval step can be insufficient: a returned chunk may reveal a fact that requires a second search.
Agentic RAG turns retrieval into a callable tool. The agent can decide whether to search, inspect the response, rewrite the query and search again. Extending that design to context engineering adds more sources: local code, scratchpads and plan.md; skill folders; enterprise databases; the web; and long-term memory. Monigatti deliberately leaves open whether that memory belongs in files or a database.
| Context source | Typical retrieval interface |
|---|---|
| Local files | File search |
| Skills | Skill loading |
| Database | Semantic search or query execution |
| Web | Web search |
| Long-term memory | Dedicated memory tool |
These interfaces expose different operations. A semantic-search tool accepts a topic; a general database tool may accept a complete SQL query. Both retrieve context, but they demand different work from the agent.
A terminal interface cuts across these boundaries. LangChain calls it a shell tool, Anthropic a bash tool, and OpenClaw an exec tool. Through it, an agent can run ls and grep, invoke a database CLI, write a script that connects to a database, or use curl against an HTTPS endpoint. The same interface can support file exploration, database access and web requests.
That versatility raises the central design question: is a shell enough, or should the agent also have specialized search tools? Search itself remains difficult. Keyword search, dense embeddings, sparse embeddings, multi-vector embeddings and indexing choices address different retrieval needs. Choose the search stack around the required search behavior and latency, rather than assuming that one interface supplies every useful retrieval capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make tool selection and parameters manageable
The apparent simplicity of agentic search hides several failure points: the agent must choose a tool, generate its parameters, interpret the response and answer. Monigatti highlights three recurring failures from work with teams building Elasticsearch agents:
- No tool call: the agent answers from parametric knowledge when it should retrieve context.
- Wrong tool: it chooses an unsuitable source. One colleague struggled to make an agent search a database instead of the web.
- Wrong parameters: it selects the right operation but generates arguments that do not express the intended search.
Improve descriptions incrementally. Start with the tool’s core purpose. If additional tools or parameters make selection unreliable, add conditions explaining when to use it and when not to. Then describe relationships: load a particular skill before executing a query, or obtain confirmation before performing an operation. If the description still does not guide the agent reliably, reinforce the instruction in the system prompt.
| Interface | What the agent must generate |
|---|---|
| Customer lookup | A valid customer ID |
| Semantic search | A query string |
| Filtered semantic search | Query, filters and possibly top K |
| Query execution | A complete SQL or ES|QL statement |
Parameter complexity is part of the failure surface. Adding filters and result limits increases the number of decisions; generating an entire query also requires knowledge of a language and the underlying schema. Models vary in how well they handle that responsibility, so expressive interfaces may need more assistance than simple lookup tools.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A semantic search tool for conference sessions
The workshop demonstrations follow three interfaces over conference-session data: semantic retrieval from a local Elasticsearch cluster, general database query execution, and shell search over local files with a semantic CLI extension. The database already contains chunked sessions; ingestion and chunking are outside the demonstration.
LangChain supplies the agent abstractions, shell tool and examples for skill loading. The first agent uses GPT-5.4 nano. Its system prompt asks it to decide whether additional context is needed before answering and explains the indexed schema. The embedded text combines each session’s title and description. Day, time, room and speaker are metadata: in this setup they can be filtered, but they are not part of the semantic representation.
Jina Embeddings V5 embeds the query at search time through the Elasticsearch vector store. The tool calls similarity_search with a fixed k=3, so each invocation can return only three results. LangChain’s @tool decorator derives the tool name from the Python function name and its description from the docstring. A factory makes that small interface explicit:
python
from langchain.tools import tool
def make_semantic_search(vector_store):
@tool
def semantic_search(query: str) -> str:
"""Find conference sessions related to a topic."""
documents = vector_store.similarity_search(query, k=3)
return "\n\n".join(
f"{doc.page_content}\nMetadata: {doc.metadata}"
for doc in documents
)
return semantic_search
The short description is sufficient for this demonstration because the agent has only one search tool to choose from.
A direct search for regulatory constraints finds Bilge’s session on engineering AI systems under sovereignty constraints, along with talks by Tejas and Pedro. Monigatti then combines the LLM, system prompt and tool into an agent, intentionally omitting memory. Asked which sessions discuss regulatory constraints in AI systems, the agent searches, rewrites its query, receives similar results and answers with the relevant talks. The loop works for this conceptual question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
GEPA exposes two different retrieval failures
A successful topic search leaves substantial gaps. An audience member suggests asking for something absent from the database. Exact keywords and metadata filters offer other ways to break this tool: filters exist in the data model, but the tool has not exposed them. Monigatti asks which sessions she should attend to learn about GEPA. The semantic tool returns material about DeepMind’s Gemma models, harness engineering and another unrelated session, despite a known GEPA session in the dataset. She speculates about tokenization similarity between GEPA and Gemma, but does not establish the cause.
The next interface accepts a whole database query instead of a topic. Monigatti pairs that more expressive tool with a skill-loading tool, then switches to GPT-5.4 mini because she expects query generation to be harder. Initially, the system prompt stays unchanged.
ES|QL, Elasticsearch’s piped query language, supports filtering, transformation and analysis. A direct query finds Samuel’s GEPA session. Wrapping the client’s query method exposes those capabilities to the agent. This time the wrapper also catches exceptions and returns the error as a tool response, giving the agent a chance to revise its query instead of terminating the interaction:
python
from langchain.tools import tool
def make_query_tool(client):
@tool
def execute_esql(query: str) -> dict:
"""Execute ES|QL against the indexed conference schedule."""
try:
response = client.esql.query(query=query)
return {
"columns": response["columns"],
"values": response["values"],
}
except Exception as exc:
return {"error": str(exc)}
return execute_esql
Returning errors supplies feedback for correction; it does not guarantee that every unsuccessful search will produce an error.
Asked the GEPA question again, the agent generates a query that looks plausible but uses SQL-style % wildcards. In ES|QL pattern matching, the wildcard is *. The demonstrated %GEPA% pattern searches for literal percent signs around GEPA, so it returns no rows.
| Pattern | Meaning in this ES|QL search |
|---|---|
%GEPA% | Literal percent signs surrounding GEPA |
*GEPA* | GEPA with any preceding or following characters |
An empty result is not proof that the requested information is absent. It can also mean that a syntactically acceptable query expressed the wrong search. The exception handler cannot repair this by itself because there is no database exception to return.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Load query knowledge, then let the database calculate
A richer tool description or system prompt could explain the wildcard rule. For a language with more documentation than a few lines, Monigatti instead introduces an agent skill. Although official Elastic skills are available, she uses a short custom skill. Its name and description enter the system prompt; its body loads only when needed. This progressive disclosure keeps detailed instructions out of the initial context.
The custom skill covers basic query structure, double-quoted string literals and asterisk wildcards. LangChain’s skill-loading tool and middleware handle loading. The execution tool’s description now requires the Elasticsearch ES|QL skill before query execution, and the system prompt repeats that dependency. The resulting sequence is:
- Load the ES|QL skill into context.
- Generate the query using its syntax rules.
- Execute the query against the conference index.
- Use the returned session information to answer.
On the next GEPA attempt, the agent follows that order, uses asterisk wildcards and identifies the session scheduled for 10:40.
The same interface can answer an analytical question: how many sessions are on April 8? After loading the skill, the agent generates a date filter and a count aggregation. The database aggregation returns 27 sessions for April 8 in the workshop’s indexed dataset.
Returning every matching session and asking the LLM to count would consume context and rely on the model’s counting ability. The database can perform the calculation and return the compact result instead. This is a broader benefit of query execution: the retrieval layer can compute the answer’s supporting facts, rather than merely supply documents from which the model must derive them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find the same session through the shell
For the filesystem demonstration, the conference data moves into a session-data directory. Session-type subfolders, such as keynotes and workshops, contain one file per session, with its title, metadata and description. Monigatti returns to GPT-5.4 nano, which she considers sufficient for this navigation task, and replaces the database-schema explanation in the prompt with the file layout.
Terminal access also permits destructive operations. Monigatti warns that the demonstrated LangChain shell tool has no default safeguards and recommends a sandbox. After importing and instantiating the tool, she illustrates its commands parameter with an echo command that prints a greeting. That simple test establishes the interface: the agent supplies shell commands and receives their output.
The GEPA lookup now becomes a sequence of file operations. The agent inspects the directory structure, searches with grep using an initial first-fifty limit, then searches the full dataset after finding one session. It finds the same session again, reads the complete matching file and uses its contents to recommend the session. The final file read matters: locating a matching word is only the first step toward obtaining the metadata and description needed for an answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the shell a semantic search operation
grep matches text and regular expressions, but an agent can approximate conceptual search by generating related terms. Asked again about regulatory constraints, the agent first searches for regulat, then expands into compliance, constraints, GDPR, governance and sovereignty. The basic operation looks like this:
bash
grep -RniE 'regulat|compliance|constraints|GDPR|governance|sovereignty' session_data/
Repeated searches eventually locate Bilge’s session. The agent has supplied the semantic expansion; grep still performs lexical matching.
This can work, but enumerating terms is an awkward general search strategy. Monigatti illustrates the limitation with movies about animal superheroes: should the agent list every possible animal to find all relevant films? The problem is not whether it can generate another synonym, but how much searching it must do to cover a concept.
Semantic CLIs add another operation behind the same terminal interface. Monigatti names LlamaIndex’s SemTools, LightOn’s ColGREP, which uses multi-vector embeddings, and JinaGrep. For JinaGrep, she installs the CLI and updates the existing system prompt to explain its availability, usage and examples.
JinaGrep also has classification and reranking modes, but this demonstration uses semantic search. The prompt distinguishes the two search paths: retain grep for exact matches and use JinaGrep for semantic or fuzzy queries. Asked about regulatory constraints, the agent explores the folders and then successfully invokes JinaGrep. In this demonstration, its first JinaGrep search finds Bilge’s session while requesting a top K of 10. The returned set includes other sessions, and the agent uses the results to answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a low floor and a high ceiling
The demonstrations suggest a mixed toolset rather than a single universal search tool. Monigatti describes the balance using two interface-design ideas:
- Low floor: specialized tools make routine operations easy. A customer lookup by ID or a simple semantic search exposes few parameters, can suit a smaller model and avoids unnecessary iterations.
- High ceiling: shell access and complete-query execution let the agent handle complex or unexpected requests that narrow tools cannot express. Their flexibility can require more attempts before the agent reaches a useful answer.
When query behavior is unknown, begin with a general-purpose tool and log what the agent does. Monigatti treats repeated sequences of four or five tool calls for a question as a reason to investigate whether the interface is too difficult—not as a universal limit. Inspect what those calls are trying to accomplish, then consider exposing that recurring operation through a specialized tool.
In her OpenClaw experiment, she logged database interactions through exec and later asked the agent to identify patterns. It recommended dedicated database search tools. The design process is therefore empirical: start with an expressive interface, observe the behavior and failures, and add purpose-built interfaces for the operations that recur.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model capability, latency and query repair
The Q&A first asks how much the toolset depends on model capability. Monigatti recalls a substantial reduction in parameter errors with stronger models in internal testing, but does not recall the figures. A stronger model can help with general-purpose tools; it does not remove the need to handle mistakes.
Latency raises a separate architectural question: should a system retain a simple RAG path for fast answers and use agentic retrieval only for harder questions? Monigatti does not offer a routing algorithm. She emphasizes that ordinary RAG remains effective for many applications, while selecting between retrieval paths may itself require decision-making logic. The existence of an agentic alternative is not, by itself, a reason to replace a working fixed pipeline.
Returning to the GEPA failure, an audience member proposes deterministically replacing SQL-style percent signs with ES|QL asterisks. Monigatti clarifies that a skill was not necessary to fix that particular example: a simple instruction against percent-sign wildcards worked while she built the demo. Her concern is what happens next. Repeatedly adding isolated syntax instructions can eventually recreate the query language’s documentation inside the system prompt. The skill provides a place to load that knowledge when needed; her response does not implement or evaluate the proposed deterministic repair.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cross-checking results without accumulating noise
Could an agent use both database and shell tools, then validate results across them? Monigatti agrees that the single-search-tool demonstrations simplify the real problem of selecting among several interfaces. She points to Testing if “bash is all you need”, an experiment comparing database, Bash, filesystem and hybrid approaches. Her recollection highlights their complementary strengths: database operations for analytical questions, file search for quick lookup, and shell-based checking of database results. The published experiment’s hybrid conclusion is about more consistent accuracy through cross-checking, rather than a uniquely higher maximum score.
More retrieval also introduces more irrelevant material. Asked whether semantic search should use a conservative score threshold, Monigatti notes that in the final JinaGrep example she considered only the first result relevant. Models can often reject irrelevant results while composing an answer, but those results may remain in context and confuse later turns. Threshold choice therefore depends both on immediate relevance judgment and on how the application manages retrieved material over a long conversation.
An audience question then raises search subagents and relevance evaluation, mentioning Ragas. Monigatti has not experimented with subagents herself. She tentatively recalls Claude Code delegating niche product questions to a specialist helper and sees that division of expertise as interesting, but offers no tested design or performance recommendation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Loaded context needs a way out
Progressive skill loading postpones context growth, but a long session can still accumulate many full skill bodies. The final substantive question asks when and how to clear them. Monigatti passes the question to her Elastic colleague Joe, who describes using a file store to offload content as context grows. Loaded material and messages can be replaced with references to their stored locations.
In Joe’s account, the agent retains skill names, descriptions and file-store locations, loads a body when needed, and offloads it as the conversation progresses. He applies the same approach to context compaction and previous tool results: keep the material in the file store and provide tools such as grep to recover it. Search then serves both sides of context management—bringing useful material into the active window and making offloaded material retrievable again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Setup instructions, conference data, notebooks for the three search demonstrations, and a slide PDF.
An evaluation of SQL, Bash, filesystem and hybrid agents answering questions over GitHub issues and pull requests.
Semantic search, reranking and classification from the terminal using local Jina embeddings on Apple Silicon.
Command-line tools for document parsing and local semantic search, with Unix pipeline examples.
Local semantic code search combining regex filtering with multi-vector ranking.
Reusable agent skills for working with Elastic products.
Further reading
Reference examples for WHERE, LIKE, regular expressions and full-text filtering.
Updates since the talk
Current guidance on built-in and custom subagents, separate context windows and tool permissions.
Read the complete timestamped transcript
- 0:00
[upbeat music] Everyone ready?
- 0:16
Yes.
- 0:17
Yes.
- 0:17
Awesome. Welcome to AI Engineer. Thanks for joining my session.
- 0:23
We are going to be talking about agentic search for context engineering today. My name is Leonie. I work at Elastic, the company behind Elasticsearch,
- 0:35
and usually I like to talk about retrieval on Twitter. Today, I'm super excited to be doing this in person. Uh, a little bit of housekeeping. If you want to access the slides and the code we will be looking at, you can scan the QR code.
- 0:51
So, let's start with why I'm excited about search and retrieval and hopefully why you are excited about it by the end of this workshop as well.
- 1:02
Who here has built an agent or some form of it before?
- 1:07
Awesome. Then you've probably... Then you're probably not intimidated by this, uh, image. You've probably seen some, some alternative to this one before. This is essentially what context engineering looks like.
- 1:21
So context engineering, when you talk about it, is the art or engineering techniques about how, from all of the possible context sources we have, how do we actually decide what goes into the context window so our LLMs can, uh, generate the best responses.
- 1:40
Often when we talk about this, we talk about context curation, and we mean this little, this little arrow from context sources to context window. But we're not giving this little arrow right there enough, uh,
- 1:55
credit, in my opinion, because what's powering this is the search tool or search tools that actually decide what goes from context sources to the context window. And today, we're going to be looking at the different search tools we have.
- 2:10
So this is my personal hot take. I like to th- say that context engineering is about eighty percent agentic search because it's this little box right here.
- 2:21
All right. Let's start with a little bit of history, and when I say history, I mean the last three years.
- 2:28
RAG. Um, when we started with RAG, the original idea was that we had a fixed retrieval pipeline, so the user message would usually mo-more or less word verbatim be used as a search query to be used usually as a vector search query to pull some data or chunks, uh, from a database.
- 2:53
And together with the retrieved context, it would-- the user message would go into the context window, and then it would be fed to the LLM.
- 3:02
Nice. This has clearly many limitations. So since this is a fixed pipeline, whether or not you actually need any context, you're still retrieving additional information, and in the worst case, that can actually confuse your LLM, right?
- 3:20
On the other hand, if you're only retrieving once, let's say you need some multi-hop retrieval. You're asking your LLM something more complex. Then if you're re-- only retrieving once, maybe the retrieved chunks, um, reveal some information about another search query you need.
- 3:38
Then you would actually might want to have a second round of search, right? So that's why we then moved on to agentic RAG.
- 3:48
So we replaced the fixed pipeline with now a search tool. So
- 3:54
now the agent can decide by himself, by itself, uh, whether or not to call the search tool and retrieve some information. So we don't have the problem anymore of do I actually need any information, and when I actually retrieve information, is this even relevant?
- 4:12
Do I need to retrieve more? Um, do I actually have to retrieve something, rewrite the search query? Um, yeah.
- 4:25
So we still only have one context source in this case, one database. Now, when we look at context engineering, the context lies in many different places, right? So we have context sources in local files.
- 4:41
So wh- when you think about your coding agent, you probably have your, um, coding project in or code files lying around in your local file system. Maybe you're using, um, some kind of working memory like a scratch pad.
- 4:55
So you're planning with your, um, agent what you want to do. Then you probably have something like a plan.md file.
- 5:04
When you have agent skills, you also have them usually in a local folder somewhere. We still have databases because many enterprises have their data stored in databases. Um, we have the web as another context source, and I know this is a super controversial, uh, image here because I did not commit to having the long-term memory in the
- 5:27
local file system or the database. This is, I think, currently a very big discussion. We can get into this in the Q&A if you like. But we also have long-term memory as another context source, right?
- 5:41
So how do we actually retrieve context from these? Usually, we have a set of, um, let's say, context source native search tools. So for the local files, you usually have something like, um, a search files, um, tool.
- 5:59
For skills, you usually have a skill loading tool. When you think about databases- We have a little bit more custom tools, so, um, something like a semantic search tool.
- 6:12
Maybe you also have something more, um, general purpose, like a tool that lets you execute entire search queries against a database like SQL, for example. For web, you have web search tools, and for memory, you have something like a dedicated memory tool.
- 6:30
If that's not overwhelming enough, we now also have something called a shell tool. Um, LangChain calls it shell tool. Anthropic calls it the bash tool. If you've experienced, uh, if you've played around with OpenCloud, it's called the exec tool.
- 6:47
But what all of these tools do is they let your agent run commands in the terminal, and that actually makes them super, uh, versatile because now you can let your agent, um, use CLIs to br- navigate your, um, and explore your local files.
- 7:04
So you can just run LS and grep to find, uh, data in your local file system.
- 7:12
If your database has a custom CLI, you can actually let the agent also use the shell tool, um, and interact with the database.
- 7:23
You could also let your agent write an entire ser-- uh, entire script from scratch, like connect to your database, run a search query.
- 7:32
Um, if your database is exposed via HTTPS, you can just run a curl command, interact with the database through the shell tool.
- 7:41
Speaking of curl commands, you can also, um, do web searches if you like. So this shell tool is super versatile, right? So the question and the topic of today is: what search tool do we actually need?
- 7:58
Do we only need a ser-- a shell tool? Do I need all of these?
- 8:04
And if you take home only one thing from today is that doing good search is incredibly difficult, and that's why we have many different techniques to do search, right?
- 8:16
We have vector search. We have keyword search. Even in vector search, we have dense embeddings, sparse embeddings, multi-vector embeddings. Then we have many different indexing techniques. So
- 8:29
depending on what kind of search requirements and latency requirements you have,
- 8:36
you will need to curate your own stack of search tools, right?
- 8:42
So today we're gonna be looking at a few of these. Unfortunately, we only have one hour, so I [laughs] cannot show you all of them. Um,
- 8:50
before we get into some code, I want to, um, give you a few fundamentals, um, of bui- building good search tools because
- 9:02
agentic search at the surface level seems very straightforward. The user makes a request. The agent calls the right tool with the right parameters. I see someone laughing. Um, the retrieval tool gives you the tool response, and then your agent, uh, responds to you with the correct answer.
- 9:25
At Elastic, we help a lot of internal and external teams build, uh, agents based to interact with Elasticsearch data, and the reality is that this can break in many different ways.
- 9:38
I'm just going to show you three, um, today. So the first is the agent doesn't call any tool. So this means the agent decides, "I actually can answer this question based on my parametric knowledge.
- 9:52
I don't need to u- use any, um, context retrieval tool." And the other, uh, problem is that the agent calls the wrong tool. I was recently talking to a colleague of mine.
- 10:03
I was asking him, "What was your, the most challenging aspect of your project?" And he was like, "You won't believe it, but it was really difficult to get the agent to actually not call the web search tool but call the database search tool."
- 10:18
Right. And then depending on how complex your parameters are for your search tools, it can also be quite challenging to get your agent to generate the right, um, search parameters, right?
- 10:34
There's many more failure cases, but we're [laughs] gonna limit this to, to th- these three today. So
- 10:41
I personally hate this slide because I feel like everyone in this room probably knows, um, that the tool description is the most important aspect. But anytime I see a tool description, it's like the least effort, one sentence, and then you're wondering why your agent isn't calling the right tool.
- 11:00
So arguably, this is a very long tool description. I'm not saying you have to write it like this. I'm just saying if you-- Just start with a core purpose.
- 11:10
If it works fine, great. But if you add more parameters or more tools and your agent is starting to struggle with calling the right, uh, tool, then maybe add some trigger condition.
- 11:26
When should this tool be used? When should this tool not be used? Especially if you have multiple tools, um, adding something like relationships is super important. Like, first call this agent skill before you actually call this tool or get some confirmation before you call this tool.
- 11:46
If you have the perfect tool description and your agent still doesn't call the right tool, then reinforce it in the agent system prompt. That should actually, um, help out in most cases.
- 11:58
Then I want to quickly touch on some, on, on the parameter complexity.
- 12:06
If you have a search tool that's just very simple in, in the sense that something like get customer by ID- It should be fairly straightforward for the agent to generate an ID parameter, given that it's a valid ID.
- 12:20
Um, same for if you're doing a semantic search, right? Gen-generating some valid string should be-- shouldn't cause any, any issues. But let's say if you want to have a semantic search tool, and instead of just giving, um, a topic, you now also want to give it some filter conditions.
- 12:41
Maybe you want to define the top K, then you start to have more parameters, right? This isn't re-- very complex here, but the longer the list of parameters you have, uh, the more difficult, um, it's going to get for the agent to generate the right ones.
- 12:57
And I think a very complex one for an agent is to-- when you have something that's more general purpose, like letting the agent, um, execute, um, entire search queries against a database.
- 13:10
So here I have ESQL, which is the Elasticsearch Query Language. Could be a se-- Uh, could be SQL as well. So letting the agent write an entire SQL query from scratch can be quite challenging.
- 13:21
Most are pretty good, but, um, some aren't. So just keep in mind, um,
- 13:29
let's say the, the complexity of the parameter also is kind of a failure mode, and you can, uh, you, you might need to help the agent out with, um, a few of these if they, they are more complex.
- 13:44
Good. Let's look at some code. Okay. So
- 14:03
quick show of hands, who here has built some sort of agentic RAG, agentic search before?
- 14:09
Okay, about half. That's good. Um, so we're going to be looking at three things. Uh, I'm going to give you a quick recap or intro to the very vanilla agentic search demo.
- 14:23
And then I'm going to show you how easy it is to break this usual demo, and then we're replacing the semantic search tool with something more general purpose. So we're letting the agent write an entire search query from scratch.
- 14:39
And for these two examples, we will be using, um, a local Elasticsearch cluster as a context source. And then for the third part, I'm switching gears, and I'm going to be showing you how search over a local file system works, um, with the bash tool.
- 14:57
And then I'm also going to show... Or the shell tool. And then I'm also going to show you, um, some limitations of the shell tool and how you can expand it with custom CLIs.
- 15:10
All right. The example that we'll be doing today is I have the conference session data of this conference here. And let me sh-- let me start, show you this one.
- 15:24
So just a quick recap. We have an Elasticsearch database, and we're gonna be writing a semantic search, um, search tool. And for-- In the database, I have the conference session, um, already chunked.
- 15:45
You probably already know how to chunk and store data in a database, so we're skipping this part. This is not the important aspect.
- 15:55
So what do we need for an agent? An LLM? Oh, sorry.
- 16:01
We're gonna be using LangChain for this, um, session just because it wraps a lot of the complexity and, uh, I don't-- Like, it lets us concentrate on the high level concepts.
- 16:15
Also, it has some nice built-in features like, uh, the shell tool is built in, and it has some, uh, code samples for skill loading tools which we will be looking at later.
- 16:27
Okay. Switching back. I'm using GPT 5.4 Nano for this demo.
- 16:34
Then we're defining a very simple system prompt. So the usual, "You are a search agent tasked with answering questions.
- 16:44
Um, you have access to different context retrieval tools and before answering a question," oops, "decide whether or not you need to retrieve additional context." To help the agent a little bit, I have some information about how the data is, um, structured in Elasticsearch.
- 17:01
So here we have a text field. It's comprised of the title of each session and the description of each se-session, and the text field is, uh, what actually gets embedded, um, as the vector embeddings for semantic search.
- 17:16
And then I also have some metadata fields. So for example, the day, the time, the room, um, the speaker's name. So since the metadata is not embedded, I can only run, um, filters over them, but not any semantic search, just for your information.
- 17:35
Okay. Now the interesting part. Let's build a semantic search tool. So how that works in LangChain is I have to first define an embedding model. In this case, I'm using the new Gina Embeddings V5 model.
- 17:53
Um, this is used to embed the search queries at query time. And the embedding model I'm putting in with-- putting it into my, um, Elasticsearch store together with the Elastic data to create a vector store.
- 18:12
And then- I can create a search tool. So in this case, I have-- all I have to do is I call the similarity search method. It takes in a search query, and in this case, I'm setting the limit or the top K to three.
- 18:32
This is a little bit of foreshadowing because I'm limiting the capabilities of this tool to just returning three search results, right?
- 18:43
What's nice in LangChain as well is that when you use the tool decorator up here, it lets you convert any Python function into a search tool-- Uh, sorry, into an agent tool.
- 18:56
So by default, it takes the, um, function's, uh, Python function's name as the tool name, and the doc string down here is going to, uh, convert-- g-going to get converted into the tool description.
- 19:12
You can see I'm breaking my own rule by having a very short tool description here. Why this works is because I only have one search tool here, right? So you will see I'm adding a few things later on, but it's not going to get, uh, very descriptive in this demo here.
- 19:32
So now we can, um, run a test and test it for a search query of regulatory constraints. And you can see it finds a talk by my friend Bilge on engineering AI systems under sovereignty constraints, and it also finds some more talks, one by Tejas and one by Pedro.
- 19:55
All right, let's plug it in. So we're plugging in the LLM, the system prompt, and the search tool. I'm leaving out memory. Obviously, this would be another core component of an agent.
- 20:08
In this case, I'm leaving it out to keep it kind of concise.
- 20:14
Now I can, uh, run a simple question like which sessions discuss regulatory constraints in AI systems? And you can see the agent first, uh, calls my semantic search tool.
- 20:31
It wrote a quite extensive search query, in my opinion, but it works. Um, it finds the right talk by Bilge.
- 20:41
Then it decided that that apparently wasn't enough, so it rewrote the search query, uh, but decided-- but got the s-very similar search results back. So after that, it decided that...
- 20:57
Sorry. It decided that it's now able to, um, respond with the right talks.
- 21:06
This is where most agentic search demos fail, but this is very brittle. Does anyone have an idea how we can break this?
- 21:19
Ask for something that's not in the database.
- 21:22
Yes, that's a good idea. Anything else? Asking it something that's not in the database would be something great. What about asking it, um, something where semantic search actually falls short?
- 21:41
Maybe something where we want to look for a keyword, a specific keyword.
- 21:47
Um, also doing something like filtering because we're-- in this search tool, we don't have any filters implemented, right?
- 21:56
So my, my choice of search query is which sessions should I visit to learn more about GEPA?
- 22:08
I'm not even sure... Like, I've heard people talk about GEPA. I'm not even sure if I'm pronouncing it correctly. Sorry. That's why I need to definitely attend this session. [chuckles]
- 22:20
So what you can see is the agent now calls the search-- semantic search tool, and this time it's looking for GEPA. So far, so good.
- 22:32
But now you can see it's actually returning a talk for, um, DeepMind's Gemma models. I guess from token-- from a tokenization perspective, it could be similar to GEPA or GEPA.
- 22:48
I don't know. Then it returns something on harness engineering. Not sure if that's necessarily related. And then a third one. I-- Clearly, none of these are related to GEPA.
- 23:04
Spoiler alert, I know there's a talk about GEPA or GEPA. Again, I think it's, like, right after this one. So we can see, uh, the search tool we just created, it's not very useful, or at least useful only for a very narrow scope of use cases, right?
- 23:26
What if we let the agent now write an entire search query from scratch? Let me show you how we can do this. So
- 23:35
we're now replacing the database tool w- that we had with an execute query tool. So we're letting the agent not only take in a, like, a, a ser-- like, a topic, but this time we're giving it an entire, uh, the search tool, an entire search query.
- 23:53
And I'm going to show you... Because this is quite difficult for an agent, uh, we're also combining it with a skill loading tool.
- 24:04
So doing the same thing. I'm setting up my LLM. You've probably noticed I'm switching to a little bit more powerful model here. So I'm switching from the GPT 5.4 Nano to, um, the Mini.
- 24:20
Because I am now anticipating that writing search queries is a little bit more difficult, so the Nano is probably not powerful enough.
- 24:30
I'm using the exact same system prompt as before.
- 24:35
And now I'm creating a general purpose database query tool. Since I'm using Elasticsearch, I'm gonna be using the Elasticsearch query language, which is a piped query language for filtering, transforming, and analyzing data.
- 24:53
It looks something like this. Maybe it reminds you of SQL. It's a little bit different. It has different capabilities. Not important for this [chuckles] session. Um, but you can see when I connect to my client and then I use this query method from the ESQL class and run this query.
- 25:13
You can actually see that there is a session by Samuel which talks a lot about GEPA. You can see here is a match. Here is another match.
- 25:26
So let's wrap this into a search tool.
- 25:34
You can see, um, this time I just used the, the query method again here, and the agent takes in the ESQL query, um, as a parameter.
- 25:47
And I exchanged the tool description with something that said-- that's called execute an ESQL query against the conference schedule indexed in Elasticsearch.
- 25:59
Notice anything different about how I wrote this search tool versus the other one?
- 26:13
This time I added a try except block here for error handling. Generally speaking, you should have error handling, but since I'm anticipating that writing a good ESQL query or a valid one is gonna cause more problems for the, the tool, um, I don't want the agent to just fail and then the whole system to crash.
- 26:37
So instead of-- Instead, I return the error response to the agent so it can kind of self-correct, rewrite the query.
- 26:47
Generally speaking, super important to have this, right, so the agent can self-correct. So
- 26:53
when we-- we can test this here. This is not important. Uh, then I'm plugging in the LLM, the system prompt, and my no-new search tool into the agent again.
- 27:03
And when I now, um, ask it the exact same question as before, "Which session should I visit to learn more about GEPA?"
- 27:13
You can see it calls the execute ESQL query tool, and it generates something that looks like valid ESQL.
- 27:23
I'm not expecting anyone to be very familiar with ESQL. What's wrong with this is that
- 27:30
ESQL doesn't use the percentage sign as a wildcard character. In ESQL, you would use the asterisk. So in this case, it's actually looking for percentage sign GEPA percentage sign in the data as an exact match.
- 27:47
So that's why it's actually returning zero search results. And this is when you're working with, uh, search tools, also super important to think about. Is returning zero search results actually a valid response, or is it a failure mode, right?
- 28:07
Okay. How could I overcome this? I could probably write, um, a more descriptive tool description, give it a little bit more help on how to write better, um, parameters.
- 28:22
I could refo- uh, reinforce it in the system prompt, give it more instructions there. Or I could use an agent skill because ESQL ha- uh, you need more documentation than just like a one-liner, right?
- 28:37
So now I'm going to show you how to add an agent skill.
- 28:44
So in this case, I'm going to be writing my own very short custom, uh, agent skill. Quick question. Who has u-used and played with agent skills before?
- 28:57
Okay. Good amount. So I'm gonna be writing a very short one here. Um,
- 29:08
there is official Elasticsearch agent skills available if you want to play around with it. In this case, I'm just, uh, using my own custom ones. So how that works is you have the, um,
- 29:22
the skill name and then also skill description, which gets injected into the system prompt. So only the, uh, if, if you write it in, in Markdown with the, I think, the front matter, right, that gets injected into, to the system prompt.
- 29:37
And then when you need it, um, more information on the agent skill is loaded into the context window, right? So it's called something like progressive disclosure, where you kind of add more information about the skill as you, as needed.
- 29:54
So in this case, I have some minimal instructions like, "Here's the basic, um, structure of an ESQL query.
- 30:05
Um, ESQL uses double quotes for string literals." Just some very basic syntax rules. And I also added some more information about the wildcard pattern, so it's not making this mistake again.
- 30:21
And then as I mentioned, LangChain has some boilerplate code you can just copy and reuse for, um, using agent skills. So I'm skipping over this. All you have to know is we have a tool...
- 30:35
a, um, skill loading tool, and it's get- it's getting inject, um... Sorry. It gets combined with a s- something called a skill middleware.
- 30:45
Skipping over this because this is not relevant for our session.
- 30:50
And now all I have to do is, um, edit, edit the tool description of my general purpose search tool. So
- 31:03
this is the exact same tool that I had before, except this time I'm now adding some relationship to... I'm saying always use Elastic, the Elasticsearch ESQL skill to generate the ESQL query before using this tool, because otherwise if the agent then still uses this tool first without, with- without,
- 31:28
um, using the agent skill, then that would be a shame, right?
- 31:34
I'm doing the exact same. So I'm reinforcing this now in the system prompt. I'm saying the same thing to use the, um, Elasticsearch a- agent skill first before calling the general purpose search tool.
- 31:47
And now I'm plugging it into the agent again. So this time LLM system prompt. For the skill loading tool, I have the skill middleware and then my general purpose ESQL query tool.
- 32:01
And now when I let the, um, ask... When I ask the agent which session should I visit to learn more about GEPA, you can see it first, first loads the skill.
- 32:12
So it actually loads everything that's kind of in the body part of the skill into my context window.
- 32:23
And then it generates, this time, a very valid ESQL with the asterisks as my percentage, uh, as my wildcard characters. And you can see it actually finds the right session.
- 32:39
And now it tells me that at ten forty, so after this session, I should be going to this session to learn more about GEPA and learn how to pronounce it correctly.
- 32:52
Okay. What's also cool about this is now the agent can techn- do a lot of things, right? It can also do aggregations.
- 33:05
So if I ask it something like how many sessions are on April 8th, you can see again it loads the, um, Elasticsearch ESQL tool.
- 33:17
Um, and then it writes an ESQL query. Whoops.
- 33:23
It writes an ESQL query that's using a filter, so it's filtering for April 8th. And then it also does an aggregation, some, some counting, and tells me today there are twenty-seven sessions.
- 33:40
This is nice because if I just do a search, let's say I ask it to tell me which sessions are on April 8th, and it only runs a filtered search, right?
- 33:51
So just imagine it would give me a list of all twenty-seven sessions that are today, and we let the agent count how many sessions there are.
- 34:03
That would probably not be so good because we all know agents or LLMs are notoriously bad at counting things. Um, and also it would, um, fill up your context window, right?
- 34:14
So by letting the agent do its own calculation, so letting... Like outsourcing the calculation part into the search tool is actually quite an efficient way to do this, right?
- 34:30
Any questions so far? Okay. Let's switch gears. Um, this is a very prominent topic at the moment.
- 34:49
Maybe you've heard the discussion about all an agent needs is a shell tool and a file system. So I work at Elastic, but I don't discriminate. Let's look at file systems and how to do this, 'cause I think it's a very interesting, um, topic in general.
- 35:07
So what I did here, um, I prepared the data this time in a local file system. So I have a folder called session data, and in here I have, uh, for each type of session, like, uh, keynotes and workshops, I have another folder.
- 35:26
And in there there's, uh, per session one file. Looks something like this, so with a title, some metadata and a description.
- 35:38
And now, um, I'm going to show you how you can use the shell tool with this.
- 35:48
So I'm switching back to the GPT 5.4 Nano because LLMs are just generally good at, um, navigating file systems, writing, uh, shell commands. So GPT 5.4 Nano is sufficient in this case.
- 36:05
I define another system prompt. So the first part of the system prompt is, uh, exactly the same as the one we had before.
- 36:18
And what I'm replacing this time is instead of explaining how the data is structured in Elasticsearch, I'm explaining how the data is structured in my local file system. Um-
- 36:31
Okay. So let's use the shell tool. I have to give you a disclaimer. Using the shell tool can be risky since giving your agent access to a terminal
- 36:47
can make it delete files or do other things you don't want it to do. So always recommended to, uh, use it in a sandbox environment. Also, in LangChain it doesn't have any safeguards by default, so please be careful when using this.
- 37:05
But other than that, it's very easy to use. So you can just use, uh, import the shell tool and instantiate it here. And here you can see how you would use it.
- 37:16
So it takes in the commands parameter. So when I say echo hello world, you can see down here it actually prints hello world into my terminal.
- 37:28
And then all you have to do, again, plug in the LLM, the system prompt, and the shell tool. And now you can ask it this exact same thing we had earlier.
- 37:40
So are there any sessions about GEPA? And you can say, as you can see here, it's called the terminal here, but it's the, it's ca-- it, the agent calls the, uh, shell tool, and it actually writes a few commands.
- 37:55
So first it's looking at the folder structure, and then it runs, um, some grep commands. So it's looking for GEPA in the session data, and I think it's looking for the first fifty entries.
- 38:10
And so you can see it saw the, um, the folder structure, and then it also found the one session we were talking about earlier. But since I was only looking at the, the first fifty and only fou-found one session, it decided that it should probably look at the entire session data.
- 38:31
So it finds the exact same session again. So this time it decides, okay, then I should probably look at the contents of this session.
- 38:40
So now it reads the entire file content,
- 38:45
and here you can see the, the session information, um, as a tool response. And then at the end, the agent tells me which session I should visit.
- 39:01
Okay. Grep works based on exact matches, right? And, um, regex. And I just want to show you this because I think it's funny how, like, surprisingly good agents are with Bash because they kind of can cheat at semantic search.
- 39:23
I, I... Let me show you this. So when I ask it which sessions discuss handling regulatory constraints, this was our semantic search query from the beginning.
- 39:39
You can see it again looks at the folder structure, and then it, the first command it, or the first search it does, it's looking for regulat-- It's fair. It's looking for regulation, for regulatory.
- 39:55
I guess that's, that's a fair start. But then it goes ahead, and now it just chains a bunch of synonyms together. So it's looking for compliance. It's looking for constraints.
- 40:06
It's looking for GDPR. It's looking for governance. Yeah, I guess that's fair.
- 40:13
Um, I think it actually finds the, finds a bunch of sessions. So it's like, okay, let me try a bunch of other synonyms. So now it's looking again for, uh, regulate, compliance, GDPR, sovereignty.
- 40:29
Um, I think, yeah, the list goes on. It's just looking at a bunch of different synonyms. And it actually is successful with this, and it finds the session by Birgit and returns the session information and then, um, is able to respond correctly.
- 40:51
I guess it works. Is that the most efficient way to do this?
- 40:56
Probably not. I mean, just as an example. So let's say you, you want to search for something like movies with animal superheroes or something. Do you really want to do-- your agent to search for a list of all the animals possible when you find all the superhero movies with animal superheroes?
- 41:18
Probably not. So it works. Is it the best? I let you decide.
- 41:27
So at the moment, there's many different semantic search alternatives to grep. Um,
- 41:36
I think there's one by LlamaIndex called Semtools. There's a really cool one by Lighton which is called Colgrep, based on multi-vector embeddings.
- 41:47
Al- also, there's one by our own Jina that's called JinaGrep. So today I'm going to show you how easy it is to actually, um, use this together with your agent.
- 41:59
So all you have to do is go ahead and install the JinaGrep CLI.
- 42:06
And then all, all you have to do is tell your agent that it now has access to this tool, or to this CLI. So this is the exact same system prompt as I had earlier.
- 42:20
Now with the difference that I'm explaining to it, um, that it has JinaGrep and how JinaGrep works, how it should use it.
- 42:32
Here are some examples of how you would use JinaGrep.
- 42:37
Just a disclaimer, GinaGreп has many different modes. You can use it for, um, classification, you can use it for re-ranking. Today, I'm just showing you how to use it for semantic search.
- 42:49
And at the end, I'm also explaining to the agent when it should use Greп and when it should use GinaGreп, just so it knows for exact matches, you probably still want to use Greп, and for, um, more semantic search or fuzzy queries, use, um, GinaGreп.
- 43:07
And then plugging this in to my agent again.
- 43:13
And when I now run the exact same semantic search query we had earlier, so which... Oops.
- 43:21
Which sessions discuss handling regulatory constraints? You can see the same behavior. So it calls the terminal tool. It first explores the, uh, folder structure, and then it actually, on the first try, is able to correctly use GinaGreп.
- 43:37
So it's looking for regulatory constraints, and boom, it actually finds, um, the session by Bilge on the first, first try.
- 43:48
Ba, ba, ba. Finds a few others because it's looking for 10. Um, it says returned top k- top k of 10.
- 43:56
And then it's able to answer me correctly. Nice.
- 44:02
All right. Any questions so far? Good. Then I'm switching back. [sniffs] [coughs]
- 44:23
So we were looking at a bunch of different tools today. We saw how big the tool landscape is. Um, I showed you a few of the, um, search tools we have.
- 44:34
Now, some practical recommendations on when should you actually use what. [sniffs]
- 44:42
So maybe let's start with this. If you're looking for just one silver bullet tool, that's probably not the right way to go.
- 44:52
Again, if you're-- if you think about it, doing good search is incredibly difficult. So ideally, you want to have or curate the right set of search tools for your agent's search behaviors.
- 45:09
And you want to have a combination of specialized tools and a combination, uh, and, uh, general purpose tools. So specialized tools are something that the agent can use out of the box, something with a very simple c- uh, parameter.
- 45:26
Something where you're not h- where you don't need a very powerful LLM. You know the agent isn't going to make a lot of mistakes. The agent can just use this tool out of the box.
- 45:36
So at Elastic, we like to think about this, about of having a low floor. So this is a concept from user experience where the agent can just
- 45:47
u- use a tool, doesn't make many mistakes. It's also efficient, so it doesn't have to run your tool multiple times. You can think about this as the semantic search tool we had earlier.
- 45:59
Maybe you need to look up customers by ID a lot of the time, then having a specialized tool for that exact operation would be helpful.
- 46:10
But then you also want to give the agent a high ceiling. That means
- 46:16
for unexpected queries, for complex questions, you want the agent to still be able to handle these questions, right? And not be-- have, like, these limited, uh, specialized tools and be like, "I c- I cannot solve this."
- 46:33
So for this case, something like a shell tool or the very, um, general purpose one we had earlier of, um, q- a query execution tool would be very helpful.
- 46:45
But the problem with the query execution tool or the shell tool you saw ear- earlier is, since it's so general purpose, the agent sometimes might need more iteration to, iterations to actually get to the right answer, right?
- 47:02
So this is my, my practical recommendation of having a balanced set of search tools, um, of a low floor and high ceiling.
- 47:14
This is all nice when you already know your agent's behavior.
- 47:18
But if you don't know your agent's query behavior yet, then I would recommend to start with a general purpose tool.
- 47:28
Then log your agent's behavior. Generally speaking, logging your agent's behavior [chuckles] I recommend it. Um, but if you notice maybe your agent is taking four or five tool calls per question, that's too many tool calls.
- 47:47
Then you probably-- that's probably an indicator that the tool your agent has is too difficult for it to use. Then definitely look at what the agent's actually trying to solve.
- 47:58
Maybe scope out something more specialized in that case. Right. [sniffs]
- 48:06
Also, if you notice specific, um, query behaviors. This is what I personally did with my, [chuckles] with my test OpenClau. I was-- Uh, it has the exec tool, and I, um, started logging its behavior, and obviously, I was playing around with databases.
- 48:23
So after three days, I was asking it, "What kind of interesting patterns do you see?" And it was recommending me actually to, um, implement some specific, uh, search tools, uh, to interact with the database because it was out of the box only using the, the exec tool.
- 48:40
All right. Start with general purpose tools if you don't know your user's behavior yet, log what breaks, and add purpose-built interfaces.
- 48:49
Yeah, that's [chuckles] was a lot to take in. Um,
- 48:54
I'm sure you have lots of questions, so I'm opening it up for Q&A. And otherwise, on your way out, don't forget to grab yourself some stickers, and then thank you for joining my session. [audience applauding]
- 49:16
Yes, I think there's a mic coming.
- 49:20
Thank you. Um, so would you say the tool stack you need is also mostly dependent on the model you're willing to use? So if you're using a very good model, it might be fine using the shell tool or search tool, but a very small model, a light agent might need more specialized tools, or?
- 49:37
Yeah. Actually, uh, we-- I think in our internal testing, we noticed that a more powerful tool actually reduces the error rate of, um, for the parameters by a f- I don't know the numbers exactly, but it was a very big amount where it reduced the error rate.
- 49:54
So having a stronger model definitely helps, um, for the general purpose tools. But I think you cannot expect, uh, just because you have a very strong model that there's gonna be no, no errors, if that makes sense.
- 50:08
It does.
- 50:09
Yeah.
- 50:19
Thanks for the nice talk. Um, I have one question. Maybe it's slightly off topic, but, um, so now we are talking about agentic RAG, but it comes with the drawback of having higher latency against typical RAG.
- 50:33
So would you recommend of having, like, a second pathway for simple RAG for fast answers? And how would you, like,
- 50:43
uh, guide the agent to actually choose the right one? Because I think it's hard to say which question should be answered by agentic RAG or by simple RAG.
- 50:54
That's a good question. I, I don't think I have a, have a good answer on, like, on the top of my head right now. I'm thinking-- I was-- Maybe something related I was asked recently, if you have a RAG system, should you replace it with agentic RAG?
- 51:13
And I guess this is kind of going in the same direction of when do you actually need agentic RAG, right? Um,
- 51:25
probably for a lot of use cases. I know RAG has been killed many times, but I think the reality is that RAG is still very effective for many use cases.
- 51:38
Um, how would you actually switch between RAG and agentic RAG? I'm not sure because I assume it's, again, it needs some kind of almost agentic logic of switching between them.
- 51:52
So I'm not sure. I'm sorry.
- 52:08
In such cases, uh, like, um, when you have, uh, the wildcard in the GEPA, uh, example, uh, why can't we just, um, perform a hybrid tool that maybe search and replaces, uh, common wrong, uh, wildcard symbols coming from
- 52:33
SQL with the correct ones, for instance?
- 52:38
Mm, I'm not sure if I understood your question correctly.
- 52:42
I mean, um, there are cases in which, uh, the agent does- doesn't know the how to write the, uh, correct query because maybe he thinks, uh, the placeholders, uh, coming from SQL, um, apply it to ESQL.
- 53:04
But, uh, so why don't we perform a hybrid tool that, um, determinately, um, uh, search and replaces the wrong, um, placeholder, the, the percentage symbol with the asterisk?
- 53:23
Yeah. Actually, so the, the example I showed you of using the agent skill wasn't necessarily the, the ne- necessary solution for it. You can also, um, add just some very simple instructions on, for ESQL, don't use, um, the percentage sign as a wildcard character.
- 53:44
It actually works. I tried it when I was building the, the demo. Um, but then when the agent now runs in the next issue, then you start adding the next piece of documentation, then you can kind of start writing the entire ESQL document- mentation from scratch into your system prompt.
- 54:03
And yes, for the demo purposes, it, it would have worked, but it's probably not how you would do it necessarily when you're building something more robust, right? Because if you just a- add like little, like, band-aids every time, uh, you run into an error, then
- 54:21
what happens when you run into the next edge case? Does that make sense? Yeah.
- 54:33
Hi. Thank you for a really wonderful, um, presentation. I have a question. Uh, in the demo, we have Walk through the agentic search with DB Curious tool and also another one with shell tool.
- 54:45
Would you recommend in the practical use we can also kind of, uh, use-- combine both tool and then we validate the result from each of tool, and then we kind of add the confidence of the result from the LM?
- 54:59
Would you like recommend doing this in a practical use?
- 55:02
Yes. Yes. That, that's a great question. Also, again, I'm kind of cheating in this demo, right? Because I'm only showing you one tool per demo. In reality, you would have something more like a bunch of different tools where you then have to, um, decide which-- or the agent has to decide which tool to use.
- 55:21
I think there was a very interesting blog post by Vercel, I believe, and they did an experiment. I think it's called... If you want to look it up, I think it's called Testing is-- If Bash Is All You Need is the title, I think, of the blog post.
- 55:36
And they actually kind of benchmark or tested an agent with a Bash tool, an agent with a f-- just file search tools, I believe, and an agent with database tools.
- 55:50
And in the end, they also had one agent with a Bash tool and the database tool, and they noticed that was super interesting. Uh, for a specific set of queries, um,
- 56:04
where you have analytical queries, this is a specific use case. Actually, the database tool was more effective, but on the other hand, the file search tool was very effective, as you saw, for just quickly finding things.
- 56:19
But the very interesting aspect was the hybrid agent with the Bash tool and the, um, database tool was actually achieving the highest, um, like, highest accuracy because at first, I believe it was first using the database tool and then verifying the results with the shell, with the shell tool, and that led the agent to actually achieve better
- 56:44
accuracy. So I think that was a very interesting way, um, and behavior to see in, in agents.
- 56:51
Thanks for sharing.
- 57:04
Um, one second question. Um, so if we use the semantic search tool, I think in practice you probably would use some kind of threshold to cut the results to not get something if there's no answer.
- 57:20
But in the agen-agentic regime, would you then say, "Okay, let's put a conservative threshold such that we don't confuse our agent," or would you say the agent is smart enough, even if we retrieve results that are not really relevant, it will good enough to, to, to notice that?
- 57:40
Yeah. That's a great question, actually. Um, so in the examples, you probably saw some where the agent was returning-- I think in the last Gina Grab example, you see it's actually returning the top K results, where only the first one is the actual relevant one.
- 57:57
And I think, um, because the agent does a little bit of reasoning over whether the search results are relevant to the search query, I think it's much better or they're much better today at kind of weeding out what's not relevant.
- 58:11
But then you kind of run into the risk if you have longer running conversations that kind of these search results sit in your context window long term could have the problem of confusing your agent long term.
- 58:24
So I think it kind of-- it depends on h- your use case of how,
- 58:33
um, your-- like, how your agent can handle, like, irrelevant search results. But generally speaking, based on search results, it can filter out what's irrelevant.
- 58:46
Thanks.
- 58:57
Uh, thank you for the talk. Amazing. Um, are you utilizing sub-agents for these search queries? Because, yeah, if we let them decide to, is it relevant to use a question like, yeah, it's done in Ragas framework, for example, for evaluation.
- 59:13
I mean, sub-agents would help a lot. Do you have any experience with them?
- 59:17
Unfortunately not. I have not played around with sub-agents yet. I can only tell you that I know, for example, I believe in Claude Code, they're using sub-agents for doing specific search tasks.
- 59:29
I think there was a blog post on how they're actually using a sub-agent to, um, answer specific questions about Claude Code because it's kind of like a niche question a user would ask.
- 59:42
So in this case, they kind of outsourced, um, the expertise to a sub-agent. So having a sub-agent for specific niche questions I think would be interesting, but I don't have too much ex-- like, experience on it.
- 59:58
Okay, thanks. Uh, can I ask another question?
- 1:00:01
Sure.
- 1:00:02
Um, damn, I forget. Sorry. [laughs]
- 1:00:07
No worries.
- 1:00:07
I try to catch up later. [laughs] [laughs]
- 1:00:26
Um, yeah. Uh, kindly, yeah, off topic, but, um, you talked about skills, and the big benefit of skills is just have the description in the, uh, system prompt, and whenever needed, we need to load this full skill.
- 1:00:42
Um, do you have any recommendation when and how to clear the system prompt again? So because we want to keep the context window small and maybe for a long, um, session we might have up to ten skills, full skills in the context.
- 1:00:59
Mm.
- 1:00:59
And yeah.
- 1:01:00
I'm not sure. Joe, do you have a better answer, like have an idea?
- 1:01:04
So what we're doing behind the scenes is that we load the skills, like the definitions within the prompt, but then we use the file store to like offload like things within the context.
- 1:01:15
So that when the context is building up, when you're loading ten or eleven or twelve skills, we basically refer to, we replace the context and the messages with the referral where, where it is in the file store, and yeah.
- 1:01:27
So we keep the context very small. Sorry. I, I've finished that bit, yeah. [laughs]
- 1:01:36
Um, but yeah, so like the way that we are doing it behind the scenes is that we're, we're, we're providing like that kind of progressive disclosure of skills. So we're providing those, um, the skill names and descriptions, the location within the file store, and then from the file store we're loading into the context window when we need
- 1:01:57
that skill, and then we offload it once it's, once it's the prog- you know, the context window progresses ahead of time around that. So we have this kind of more on-demand, uh, one around that.
- 1:02:09
And that's the same with our like compaction, like com- of context, and that's what I would advise you to do. Like for some of the questions too, try and use the file store as much as you can and have those tools as well, like being able to grep the file store for when you wanna see previous tool
- 1:02:25
results and then, and then, uh, use it from there.
- 1:02:33
Thanks. That's my colleague Joe from Elastic as well. [laughs]
- 1:02:40
Awesome. If there are no more question, I will let you guys go into the coffee break. Again, don't forget to grab yourself some stickers, and happy to catch up in the halls if anyone's interested.
- 1:02:54
Thanks so much. [clapping] [outro jingle]