Agentic Search for Context Engineering

Talk transcript

1. 0:00

   [upbeat music] Everyone ready?

2. 0:16

   Yes.

3. 0:17

   Yes.

4. 0:17

   Awesome. Welcome to AI Engineer. Thanks for joining my session.

5. 0:23

   We are going to be talking about agentic search for context engineering today. My name is Leonie. I work at Elastic, the company behind Elasticsearch,

6. 0:35

   and usually I like to talk about retrieval on Twitter. Today, I'm super excited to be doing this in person. Uh, a little bit of housekeeping. If you want to access the slides and the code we will be looking at, you can scan the QR code.

7. 0:51

   So, let's start with why I'm excited about search and retrieval and hopefully why you are excited about it by the end of this workshop as well.

8. 1:02

   Who here has built an agent or some form of it before?

9. 1:07

   Awesome. Then you've probably... Then you're probably not intimidated by this, uh, image. You've probably seen some, some alternative to this one before. This is essentially what context engineering looks like.

10. 1:21

    So context engineering, when you talk about it, is the art or engineering techniques about how, from all of the possible context sources we have, how do we actually decide what goes into the context window so our LLMs can, uh, generate the best responses.

11. 1:40

    Often when we talk about this, we talk about context curation, and we mean this little, this little arrow from context sources to context window. But we're not giving this little arrow right there enough, uh,

12. 1:55

    credit, in my opinion, because what's powering this is the search tool or search tools that actually decide what goes from context sources to the context window. And today, we're going to be looking at the different search tools we have.

13. 2:10

    So this is my personal hot take. I like to th- say that context engineering is about eighty percent agentic search because it's this little box right here.

14. 2:21

    All right. Let's start with a little bit of history, and when I say history, I mean the last three years.

15. 2:28

    RAG. Um, when we started with RAG, the original idea was that we had a fixed retrieval pipeline, so the user message would usually mo-more or less word verbatim be used as a search query to be used usually as a vector search query to pull some data or chunks, uh, from a database.

16. 2:53

    And together with the retrieved context, it would-- the user message would go into the context window, and then it would be fed to the LLM.

17. 3:02

    Nice. This has clearly many limitations. So since this is a fixed pipeline, whether or not you actually need any context, you're still retrieving additional information, and in the worst case, that can actually confuse your LLM, right?

18. 3:20

    On the other hand, if you're only retrieving once, let's say you need some multi-hop retrieval. You're asking your LLM something more complex. Then if you're re-- only retrieving once, maybe the retrieved chunks, um, reveal some information about another search query you need.

19. 3:38

    Then you would actually might want to have a second round of search, right? So that's why we then moved on to agentic RAG.

20. 3:48

    So we replaced the fixed pipeline with now a search tool. So

21. 3:54

    now the agent can decide by himself, by itself, uh, whether or not to call the search tool and retrieve some information. So we don't have the problem anymore of do I actually need any information, and when I actually retrieve information, is this even relevant?

22. 4:12

    Do I need to retrieve more? Um, do I actually have to retrieve something, rewrite the search query? Um, yeah.

23. 4:25

    So we still only have one context source in this case, one database. Now, when we look at context engineering, the context lies in many different places, right? So we have context sources in local files.

24. 4:41

    So wh- when you think about your coding agent, you probably have your, um, coding project in or code files lying around in your local file system. Maybe you're using, um, some kind of working memory like a scratch pad.

25. 4:55

    So you're planning with your, um, agent what you want to do. Then you probably have something like a plan.md file.

26. 5:04

    When you have agent skills, you also have them usually in a local folder somewhere. We still have databases because many enterprises have their data stored in databases. Um, we have the web as another context source, and I know this is a super controversial, uh, image here because I did not commit to having the long-term memory in the

27. 5:27

    local file system or the database. This is, I think, currently a very big discussion. We can get into this in the Q&A if you like. But we also have long-term memory as another context source, right?

28. 5:41

    So how do we actually retrieve context from these? Usually, we have a set of, um, let's say, context source native search tools. So for the local files, you usually have something like, um, a search files, um, tool.

29. 5:59

    For skills, you usually have a skill loading tool. When you think about databases- We have a little bit more custom tools, so, um, something like a semantic search tool.

30. 6:12

    Maybe you also have something more, um, general purpose, like a tool that lets you execute entire search queries against a database like SQL, for example. For web, you have web search tools, and for memory, you have something like a dedicated memory tool.

31. 6:30

    If that's not overwhelming enough, we now also have something called a shell tool. Um, LangChain calls it shell tool. Anthropic calls it the bash tool. If you've experienced, uh, if you've played around with OpenCloud, it's called the exec tool.

32. 6:47

    But what all of these tools do is they let your agent run commands in the terminal, and that actually makes them super, uh, versatile because now you can let your agent, um, use CLIs to br- navigate your, um, and explore your local files.

33. 7:04

    So you can just run LS and grep to find, uh, data in your local file system.

34. 7:12

    If your database has a custom CLI, you can actually let the agent also use the shell tool, um, and interact with the database.

35. 7:23

    You could also let your agent write an entire ser-- uh, entire script from scratch, like connect to your database, run a search query.

36. 7:32

    Um, if your database is exposed via HTTPS, you can just run a curl command, interact with the database through the shell tool.

37. 7:41

    Speaking of curl commands, you can also, um, do web searches if you like. So this shell tool is super versatile, right? So the question and the topic of today is: what search tool do we actually need?

38. 7:58

    Do we only need a ser-- a shell tool? Do I need all of these?

39. 8:04

    And if you take home only one thing from today is that doing good search is incredibly difficult, and that's why we have many different techniques to do search, right?

40. 8:16

    We have vector search. We have keyword search. Even in vector search, we have dense embeddings, sparse embeddings, multi-vector embeddings. Then we have many different indexing techniques. So

41. 8:29

    depending on what kind of search requirements and latency requirements you have,

42. 8:36

    you will need to curate your own stack of search tools, right?

43. 8:42

    So today we're gonna be looking at a few of these. Unfortunately, we only have one hour, so I [laughs] cannot show you all of them. Um,

44. 8:50

    before we get into some code, I want to, um, give you a few fundamentals, um, of bui- building good search tools because

45. 9:02

    agentic search at the surface level seems very straightforward. The user makes a request. The agent calls the right tool with the right parameters. I see someone laughing. Um, the retrieval tool gives you the tool response, and then your agent, uh, responds to you with the correct answer.

46. 9:25

    At Elastic, we help a lot of internal and external teams build, uh, agents based to interact with Elasticsearch data, and the reality is that this can break in many different ways.

47. 9:38

    I'm just going to show you three, um, today. So the first is the agent doesn't call any tool. So this means the agent decides, "I actually can answer this question based on my parametric knowledge.

48. 9:52

    I don't need to u- use any, um, context retrieval tool." And the other, uh, problem is that the agent calls the wrong tool. I was recently talking to a colleague of mine.

49. 10:03

    I was asking him, "What was your, the most challenging aspect of your project?" And he was like, "You won't believe it, but it was really difficult to get the agent to actually not call the web search tool but call the database search tool."

50. 10:18

    Right. And then depending on how complex your parameters are for your search tools, it can also be quite challenging to get your agent to generate the right, um, search parameters, right?

51. 10:34

    There's many more failure cases, but we're [laughs] gonna limit this to, to th- these three today. So

52. 10:41

    I personally hate this slide because I feel like everyone in this room probably knows, um, that the tool description is the most important aspect. But anytime I see a tool description, it's like the least effort, one sentence, and then you're wondering why your agent isn't calling the right tool.

53. 11:00

    So arguably, this is a very long tool description. I'm not saying you have to write it like this. I'm just saying if you-- Just start with a core purpose.

54. 11:10

    If it works fine, great. But if you add more parameters or more tools and your agent is starting to struggle with calling the right, uh, tool, then maybe add some trigger condition.

55. 11:26

    When should this tool be used? When should this tool not be used? Especially if you have multiple tools, um, adding something like relationships is super important. Like, first call this agent skill before you actually call this tool or get some confirmation before you call this tool.

56. 11:46

    If you have the perfect tool description and your agent still doesn't call the right tool, then reinforce it in the agent system prompt. That should actually, um, help out in most cases.

57. 11:58

    Then I want to quickly touch on some, on, on the parameter complexity.

58. 12:06

    If you have a search tool that's just very simple in, in the sense that something like get customer by ID- It should be fairly straightforward for the agent to generate an ID parameter, given that it's a valid ID.

59. 12:20

    Um, same for if you're doing a semantic search, right? Gen-generating some valid string should be-- shouldn't cause any, any issues. But let's say if you want to have a semantic search tool, and instead of just giving, um, a topic, you now also want to give it some filter conditions.

60. 12:41

    Maybe you want to define the top K, then you start to have more parameters, right? This isn't re-- very complex here, but the longer the list of parameters you have, uh, the more difficult, um, it's going to get for the agent to generate the right ones.

61. 12:57

    And I think a very complex one for an agent is to-- when you have something that's more general purpose, like letting the agent, um, execute, um, entire search queries against a database.

62. 13:10

    So here I have ESQL, which is the Elasticsearch Query Language. Could be a se-- Uh, could be SQL as well. So letting the agent write an entire SQL query from scratch can be quite challenging.

63. 13:21

    Most are pretty good, but, um, some aren't. So just keep in mind, um,

64. 13:29

    let's say the, the complexity of the parameter also is kind of a failure mode, and you can, uh, you, you might need to help the agent out with, um, a few of these if they, they are more complex.

65. 13:44

    Good. Let's look at some code. Okay. So

66. 14:03

    quick show of hands, who here has built some sort of agentic RAG, agentic search before?

67. 14:09

    Okay, about half. That's good. Um, so we're going to be looking at three things. Uh, I'm going to give you a quick recap or intro to the very vanilla agentic search demo.

68. 14:23

    And then I'm going to show you how easy it is to break this usual demo, and then we're replacing the semantic search tool with something more general purpose. So we're letting the agent write an entire search query from scratch.

69. 14:39

    And for these two examples, we will be using, um, a local Elasticsearch cluster as a context source. And then for the third part, I'm switching gears, and I'm going to be showing you how search over a local file system works, um, with the bash tool.

70. 14:57

    And then I'm also going to show... Or the shell tool. And then I'm also going to show you, um, some limitations of the shell tool and how you can expand it with custom CLIs.

71. 15:10

    All right. The example that we'll be doing today is I have the conference session data of this conference here. And let me sh-- let me start, show you this one.

72. 15:24

    So just a quick recap. We have an Elasticsearch database, and we're gonna be writing a semantic search, um, search tool. And for-- In the database, I have the conference session, um, already chunked.

73. 15:45

    You probably already know how to chunk and store data in a database, so we're skipping this part. This is not the important aspect.

74. 15:55

    So what do we need for an agent? An LLM? Oh, sorry.

75. 16:01

    We're gonna be using LangChain for this, um, session just because it wraps a lot of the complexity and, uh, I don't-- Like, it lets us concentrate on the high level concepts.

76. 16:15

    Also, it has some nice built-in features like, uh, the shell tool is built in, and it has some, uh, code samples for skill loading tools which we will be looking at later.

77. 16:27

    Okay. Switching back. I'm using GPT 5.4 Nano for this demo.

78. 16:34

    Then we're defining a very simple system prompt. So the usual, "You are a search agent tasked with answering questions.

79. 16:44

    Um, you have access to different context retrieval tools and before answering a question," oops, "decide whether or not you need to retrieve additional context." To help the agent a little bit, I have some information about how the data is, um, structured in Elasticsearch.

80. 17:01

    So here we have a text field. It's comprised of the title of each session and the description of each se-session, and the text field is, uh, what actually gets embedded, um, as the vector embeddings for semantic search.

81. 17:16

    And then I also have some metadata fields. So for example, the day, the time, the room, um, the speaker's name. So since the metadata is not embedded, I can only run, um, filters over them, but not any semantic search, just for your information.

82. 17:35

    Okay. Now the interesting part. Let's build a semantic search tool. So how that works in LangChain is I have to first define an embedding model. In this case, I'm using the new Gina Embeddings V5 model.

83. 17:53

    Um, this is used to embed the search queries at query time. And the embedding model I'm putting in with-- putting it into my, um, Elasticsearch store together with the Elastic data to create a vector store.

84. 18:12

    And then- I can create a search tool. So in this case, I have-- all I have to do is I call the similarity search method. It takes in a search query, and in this case, I'm setting the limit or the top K to three.

85. 18:32

    This is a little bit of foreshadowing because I'm limiting the capabilities of this tool to just returning three search results, right?

86. 18:43

    What's nice in LangChain as well is that when you use the tool decorator up here, it lets you convert any Python function into a search tool-- Uh, sorry, into an agent tool.

87. 18:56

    So by default, it takes the, um, function's, uh, Python function's name as the tool name, and the doc string down here is going to, uh, convert-- g-going to get converted into the tool description.

88. 19:12

    You can see I'm breaking my own rule by having a very short tool description here. Why this works is because I only have one search tool here, right? So you will see I'm adding a few things later on, but it's not going to get, uh, very descriptive in this demo here.

89. 19:32

    So now we can, um, run a test and test it for a search query of regulatory constraints. And you can see it finds a talk by my friend Bilge on engineering AI systems under sovereignty constraints, and it also finds some more talks, one by Tejas and one by Pedro.

90. 19:55

    All right, let's plug it in. So we're plugging in the LLM, the system prompt, and the search tool. I'm leaving out memory. Obviously, this would be another core component of an agent.

91. 20:08

    In this case, I'm leaving it out to keep it kind of concise.

92. 20:14

    Now I can, uh, run a simple question like which sessions discuss regulatory constraints in AI systems? And you can see the agent first, uh, calls my semantic search tool.

93. 20:31

    It wrote a quite extensive search query, in my opinion, but it works. Um, it finds the right talk by Bilge.

94. 20:41

    Then it decided that that apparently wasn't enough, so it rewrote the search query, uh, but decided-- but got the s-very similar search results back. So after that, it decided that...

95. 20:57

    Sorry. It decided that it's now able to, um, respond with the right talks.

96. 21:06

    This is where most agentic search demos fail, but this is very brittle. Does anyone have an idea how we can break this?

97. 21:19

    Ask for something that's not in the database.

98. 21:22

    Yes, that's a good idea. Anything else? Asking it something that's not in the database would be something great. What about asking it, um, something where semantic search actually falls short?

99. 21:41

    Maybe something where we want to look for a keyword, a specific keyword.

100. 21:47

     Um, also doing something like filtering because we're-- in this search tool, we don't have any filters implemented, right?

101. 21:56

     So my, my choice of search query is which sessions should I visit to learn more about GEPA?

102. 22:08

     I'm not even sure... Like, I've heard people talk about GEPA. I'm not even sure if I'm pronouncing it correctly. Sorry. That's why I need to definitely attend this session. [chuckles]

103. 22:20

     So what you can see is the agent now calls the search-- semantic search tool, and this time it's looking for GEPA. So far, so good.

104. 22:32

     But now you can see it's actually returning a talk for, um, DeepMind's Gemma models. I guess from token-- from a tokenization perspective, it could be similar to GEPA or GEPA.

105. 22:48

     I don't know. Then it returns something on harness engineering. Not sure if that's necessarily related. And then a third one. I-- Clearly, none of these are related to GEPA.

106. 23:04

     Spoiler alert, I know there's a talk about GEPA or GEPA. Again, I think it's, like, right after this one. So we can see, uh, the search tool we just created, it's not very useful, or at least useful only for a very narrow scope of use cases, right?

107. 23:26

     What if we let the agent now write an entire search query from scratch? Let me show you how we can do this. So

108. 23:35

     we're now replacing the database tool w- that we had with an execute query tool. So we're letting the agent not only take in a, like, a, a ser-- like, a topic, but this time we're giving it an entire, uh, the search tool, an entire search query.

109. 23:53

     And I'm going to show you... Because this is quite difficult for an agent, uh, we're also combining it with a skill loading tool.

110. 24:04

     So doing the same thing. I'm setting up my LLM. You've probably noticed I'm switching to a little bit more powerful model here. So I'm switching from the GPT 5.4 Nano to, um, the Mini.

111. 24:20

     Because I am now anticipating that writing search queries is a little bit more difficult, so the Nano is probably not powerful enough.

112. 24:30

     I'm using the exact same system prompt as before.

113. 24:35

     And now I'm creating a general purpose database query tool. Since I'm using Elasticsearch, I'm gonna be using the Elasticsearch query language, which is a piped query language for filtering, transforming, and analyzing data.

114. 24:53

     It looks something like this. Maybe it reminds you of SQL. It's a little bit different. It has different capabilities. Not important for this [chuckles] session. Um, but you can see when I connect to my client and then I use this query method from the ESQL class and run this query.

115. 25:13

     You can actually see that there is a session by Samuel which talks a lot about GEPA. You can see here is a match. Here is another match.

116. 25:26

     So let's wrap this into a search tool.

117. 25:34

     You can see, um, this time I just used the, the query method again here, and the agent takes in the ESQL query, um, as a parameter.

118. 25:47

     And I exchanged the tool description with something that said-- that's called execute an ESQL query against the conference schedule indexed in Elasticsearch.

119. 25:59

     Notice anything different about how I wrote this search tool versus the other one?

120. 26:13

     This time I added a try except block here for error handling. Generally speaking, you should have error handling, but since I'm anticipating that writing a good ESQL query or a valid one is gonna cause more problems for the, the tool, um, I don't want the agent to just fail and then the whole system to crash.

121. 26:37

     So instead of-- Instead, I return the error response to the agent so it can kind of self-correct, rewrite the query.

122. 26:47

     Generally speaking, super important to have this, right, so the agent can self-correct. So

123. 26:53

     when we-- we can test this here. This is not important. Uh, then I'm plugging in the LLM, the system prompt, and my no-new search tool into the agent again.

124. 27:03

     And when I now, um, ask it the exact same question as before, "Which session should I visit to learn more about GEPA?"

125. 27:13

     You can see it calls the execute ESQL query tool, and it generates something that looks like valid ESQL.

126. 27:23

     I'm not expecting anyone to be very familiar with ESQL. What's wrong with this is that

127. 27:30

     ESQL doesn't use the percentage sign as a wildcard character. In ESQL, you would use the asterisk. So in this case, it's actually looking for percentage sign GEPA percentage sign in the data as an exact match.

128. 27:47

     So that's why it's actually returning zero search results. And this is when you're working with, uh, search tools, also super important to think about. Is returning zero search results actually a valid response, or is it a failure mode, right?

129. 28:07

     Okay. How could I overcome this? I could probably write, um, a more descriptive tool description, give it a little bit more help on how to write better, um, parameters.

130. 28:22

     I could refo- uh, reinforce it in the system prompt, give it more instructions there. Or I could use an agent skill because ESQL ha- uh, you need more documentation than just like a one-liner, right?

131. 28:37

     So now I'm going to show you how to add an agent skill.

132. 28:44

     So in this case, I'm going to be writing my own very short custom, uh, agent skill. Quick question. Who has u-used and played with agent skills before?

133. 28:57

     Okay. Good amount. So I'm gonna be writing a very short one here. Um,

134. 29:08

     there is official Elasticsearch agent skills available if you want to play around with it. In this case, I'm just, uh, using my own custom ones. So how that works is you have the, um,

135. 29:22

     the skill name and then also skill description, which gets injected into the system prompt. So only the, uh, if, if you write it in, in Markdown with the, I think, the front matter, right, that gets injected into, to the system prompt.

136. 29:37

     And then when you need it, um, more information on the agent skill is loaded into the context window, right? So it's called something like progressive disclosure, where you kind of add more information about the skill as you, as needed.

137. 29:54

     So in this case, I have some minimal instructions like, "Here's the basic, um, structure of an ESQL query.

138. 30:05

     Um, ESQL uses double quotes for string literals." Just some very basic syntax rules. And I also added some more information about the wildcard pattern, so it's not making this mistake again.

139. 30:21

     And then as I mentioned, LangChain has some boilerplate code you can just copy and reuse for, um, using agent skills. So I'm skipping over this. All you have to know is we have a tool...

140. 30:35

     a, um, skill loading tool, and it's get- it's getting inject, um... Sorry. It gets combined with a s- something called a skill middleware.

141. 30:45

     Skipping over this because this is not relevant for our session.

142. 30:50

     And now all I have to do is, um, edit, edit the tool description of my general purpose search tool. So

143. 31:03

     this is the exact same tool that I had before, except this time I'm now adding some relationship to... I'm saying always use Elastic, the Elasticsearch ESQL skill to generate the ESQL query before using this tool, because otherwise if the agent then still uses this tool first without, with- without,

144. 31:28

     um, using the agent skill, then that would be a shame, right?

145. 31:34

     I'm doing the exact same. So I'm reinforcing this now in the system prompt. I'm saying the same thing to use the, um, Elasticsearch a- agent skill first before calling the general purpose search tool.

146. 31:47

     And now I'm plugging it into the agent again. So this time LLM system prompt. For the skill loading tool, I have the skill middleware and then my general purpose ESQL query tool.

147. 32:01

     And now when I let the, um, ask... When I ask the agent which session should I visit to learn more about GEPA, you can see it first, first loads the skill.

148. 32:12

     So it actually loads everything that's kind of in the body part of the skill into my context window.

149. 32:23

     And then it generates, this time, a very valid ESQL with the asterisks as my percentage, uh, as my wildcard characters. And you can see it actually finds the right session.

150. 32:39

     And now it tells me that at ten forty, so after this session, I should be going to this session to learn more about GEPA and learn how to pronounce it correctly.

151. 32:52

     Okay. What's also cool about this is now the agent can techn- do a lot of things, right? It can also do aggregations.

152. 33:05

     So if I ask it something like how many sessions are on April 8th, you can see again it loads the, um, Elasticsearch ESQL tool.

153. 33:17

     Um, and then it writes an ESQL query. Whoops.

154. 33:23

     It writes an ESQL query that's using a filter, so it's filtering for April 8th. And then it also does an aggregation, some, some counting, and tells me today there are twenty-seven sessions.

155. 33:40

     This is nice because if I just do a search, let's say I ask it to tell me which sessions are on April 8th, and it only runs a filtered search, right?

156. 33:51

     So just imagine it would give me a list of all twenty-seven sessions that are today, and we let the agent count how many sessions there are.

157. 34:03

     That would probably not be so good because we all know agents or LLMs are notoriously bad at counting things. Um, and also it would, um, fill up your context window, right?

158. 34:14

     So by letting the agent do its own calculation, so letting... Like outsourcing the calculation part into the search tool is actually quite an efficient way to do this, right?

159. 34:30

     Any questions so far? Okay. Let's switch gears. Um, this is a very prominent topic at the moment.

160. 34:49

     Maybe you've heard the discussion about all an agent needs is a shell tool and a file system. So I work at Elastic, but I don't discriminate. Let's look at file systems and how to do this, 'cause I think it's a very interesting, um, topic in general.

161. 35:07

     So what I did here, um, I prepared the data this time in a local file system. So I have a folder called session data, and in here I have, uh, for each type of session, like, uh, keynotes and workshops, I have another folder.

162. 35:26

     And in there there's, uh, per session one file. Looks something like this, so with a title, some metadata and a description.

163. 35:38

     And now, um, I'm going to show you how you can use the shell tool with this.

164. 35:48

     So I'm switching back to the GPT 5.4 Nano because LLMs are just generally good at, um, navigating file systems, writing, uh, shell commands. So GPT 5.4 Nano is sufficient in this case.

165. 36:05

     I define another system prompt. So the first part of the system prompt is, uh, exactly the same as the one we had before.

166. 36:18

     And what I'm replacing this time is instead of explaining how the data is structured in Elasticsearch, I'm explaining how the data is structured in my local file system. Um-

167. 36:31

     Okay. So let's use the shell tool. I have to give you a disclaimer. Using the shell tool can be risky since giving your agent access to a terminal

168. 36:47

     can make it delete files or do other things you don't want it to do. So always recommended to, uh, use it in a sandbox environment. Also, in LangChain it doesn't have any safeguards by default, so please be careful when using this.

169. 37:05

     But other than that, it's very easy to use. So you can just use, uh, import the shell tool and instantiate it here. And here you can see how you would use it.

170. 37:16

     So it takes in the commands parameter. So when I say echo hello world, you can see down here it actually prints hello world into my terminal.

171. 37:28

     And then all you have to do, again, plug in the LLM, the system prompt, and the shell tool. And now you can ask it this exact same thing we had earlier.

172. 37:40

     So are there any sessions about GEPA? And you can say, as you can see here, it's called the terminal here, but it's the, it's ca-- it, the agent calls the, uh, shell tool, and it actually writes a few commands.

173. 37:55

     So first it's looking at the folder structure, and then it runs, um, some grep commands. So it's looking for GEPA in the session data, and I think it's looking for the first fifty entries.

174. 38:10

     And so you can see it saw the, um, the folder structure, and then it also found the one session we were talking about earlier. But since I was only looking at the, the first fifty and only fou-found one session, it decided that it should probably look at the entire session data.

175. 38:31

     So it finds the exact same session again. So this time it decides, okay, then I should probably look at the contents of this session.

176. 38:40

     So now it reads the entire file content,

177. 38:45

     and here you can see the, the session information, um, as a tool response. And then at the end, the agent tells me which session I should visit.

178. 39:01

     Okay. Grep works based on exact matches, right? And, um, regex. And I just want to show you this because I think it's funny how, like, surprisingly good agents are with Bash because they kind of can cheat at semantic search.

179. 39:23

     I, I... Let me show you this. So when I ask it which sessions discuss handling regulatory constraints, this was our semantic search query from the beginning.

180. 39:39

     You can see it again looks at the folder structure, and then it, the first command it, or the first search it does, it's looking for regulat-- It's fair. It's looking for regulation, for regulatory.

181. 39:55

     I guess that's, that's a fair start. But then it goes ahead, and now it just chains a bunch of synonyms together. So it's looking for compliance. It's looking for constraints.

182. 40:06

     It's looking for GDPR. It's looking for governance. Yeah, I guess that's fair.

183. 40:13

     Um, I think it actually finds the, finds a bunch of sessions. So it's like, okay, let me try a bunch of other synonyms. So now it's looking again for, uh, regulate, compliance, GDPR, sovereignty.

184. 40:29

     Um, I think, yeah, the list goes on. It's just looking at a bunch of different synonyms. And it actually is successful with this, and it finds the session by Birgit and returns the session information and then, um, is able to respond correctly.

185. 40:51

     I guess it works. Is that the most efficient way to do this?

186. 40:56

     Probably not. I mean, just as an example. So let's say you, you want to search for something like movies with animal superheroes or something. Do you really want to do-- your agent to search for a list of all the animals possible when you find all the superhero movies with animal superheroes?

187. 41:18

     Probably not. So it works. Is it the best? I let you decide.

188. 41:27

     So at the moment, there's many different semantic search alternatives to grep. Um,

189. 41:36

     I think there's one by LlamaIndex called Semtools. There's a really cool one by Lighton which is called Colgrep, based on multi-vector embeddings.

190. 41:47

     Al- also, there's one by our own Jina that's called JinaGrep. So today I'm going to show you how easy it is to actually, um, use this together with your agent.

191. 41:59

     So all you have to do is go ahead and install the JinaGrep CLI.

192. 42:06

     And then all, all you have to do is tell your agent that it now has access to this tool, or to this CLI. So this is the exact same system prompt as I had earlier.

193. 42:20

     Now with the difference that I'm explaining to it, um, that it has JinaGrep and how JinaGrep works, how it should use it.

194. 42:32

     Here are some examples of how you would use JinaGrep.

195. 42:37

     Just a disclaimer, GinaGreп has many different modes. You can use it for, um, classification, you can use it for re-ranking. Today, I'm just showing you how to use it for semantic search.

196. 42:49

     And at the end, I'm also explaining to the agent when it should use Greп and when it should use GinaGreп, just so it knows for exact matches, you probably still want to use Greп, and for, um, more semantic search or fuzzy queries, use, um, GinaGreп.

197. 43:07

     And then plugging this in to my agent again.

198. 43:13

     And when I now run the exact same semantic search query we had earlier, so which... Oops.

199. 43:21

     Which sessions discuss handling regulatory constraints? You can see the same behavior. So it calls the terminal tool. It first explores the, uh, folder structure, and then it actually, on the first try, is able to correctly use GinaGreп.

200. 43:37

     So it's looking for regulatory constraints, and boom, it actually finds, um, the session by Bilge on the first, first try.

201. 43:48

     Ba, ba, ba. Finds a few others because it's looking for 10. Um, it says returned top k- top k of 10.

202. 43:56

     And then it's able to answer me correctly. Nice.

203. 44:02

     All right. Any questions so far? Good. Then I'm switching back. [sniffs] [coughs]

204. 44:23

     So we were looking at a bunch of different tools today. We saw how big the tool landscape is. Um, I showed you a few of the, um, search tools we have.

205. 44:34

     Now, some practical recommendations on when should you actually use what. [sniffs]

206. 44:42

     So maybe let's start with this. If you're looking for just one silver bullet tool, that's probably not the right way to go.

207. 44:52

     Again, if you're-- if you think about it, doing good search is incredibly difficult. So ideally, you want to have or curate the right set of search tools for your agent's search behaviors.

208. 45:09

     And you want to have a combination of specialized tools and a combination, uh, and, uh, general purpose tools. So specialized tools are something that the agent can use out of the box, something with a very simple c- uh, parameter.

209. 45:26

     Something where you're not h- where you don't need a very powerful LLM. You know the agent isn't going to make a lot of mistakes. The agent can just use this tool out of the box.

210. 45:36

     So at Elastic, we like to think about this, about of having a low floor. So this is a concept from user experience where the agent can just

211. 45:47

     u- use a tool, doesn't make many mistakes. It's also efficient, so it doesn't have to run your tool multiple times. You can think about this as the semantic search tool we had earlier.

212. 45:59

     Maybe you need to look up customers by ID a lot of the time, then having a specialized tool for that exact operation would be helpful.

213. 46:10

     But then you also want to give the agent a high ceiling. That means

214. 46:16

     for unexpected queries, for complex questions, you want the agent to still be able to handle these questions, right? And not be-- have, like, these limited, uh, specialized tools and be like, "I c- I cannot solve this."

215. 46:33

     So for this case, something like a shell tool or the very, um, general purpose one we had earlier of, um, q- a query execution tool would be very helpful.

216. 46:45

     But the problem with the query execution tool or the shell tool you saw ear- earlier is, since it's so general purpose, the agent sometimes might need more iteration to, iterations to actually get to the right answer, right?

217. 47:02

     So this is my, my practical recommendation of having a balanced set of search tools, um, of a low floor and high ceiling.

218. 47:14

     This is all nice when you already know your agent's behavior.

219. 47:18

     But if you don't know your agent's query behavior yet, then I would recommend to start with a general purpose tool.

220. 47:28

     Then log your agent's behavior. Generally speaking, logging your agent's behavior [chuckles] I recommend it. Um, but if you notice maybe your agent is taking four or five tool calls per question, that's too many tool calls.

221. 47:47

     Then you probably-- that's probably an indicator that the tool your agent has is too difficult for it to use. Then definitely look at what the agent's actually trying to solve.

222. 47:58

     Maybe scope out something more specialized in that case. Right. [sniffs]

223. 48:06

     Also, if you notice specific, um, query behaviors. This is what I personally did with my, [chuckles] with my test OpenClau. I was-- Uh, it has the exec tool, and I, um, started logging its behavior, and obviously, I was playing around with databases.

224. 48:23

     So after three days, I was asking it, "What kind of interesting patterns do you see?" And it was recommending me actually to, um, implement some specific, uh, search tools, uh, to interact with the database because it was out of the box only using the, the exec tool.

225. 48:40

     All right. Start with general purpose tools if you don't know your user's behavior yet, log what breaks, and add purpose-built interfaces.

226. 48:49

     Yeah, that's [chuckles] was a lot to take in. Um,

227. 48:54

     I'm sure you have lots of questions, so I'm opening it up for Q&A. And otherwise, on your way out, don't forget to grab yourself some stickers, and then thank you for joining my session. [audience applauding]

228. 49:16

     Yes, I think there's a mic coming.

229. 49:20

     Thank you. Um, so would you say the tool stack you need is also mostly dependent on the model you're willing to use? So if you're using a very good model, it might be fine using the shell tool or search tool, but a very small model, a light agent might need more specialized tools, or?

230. 49:37

     Yeah. Actually, uh, we-- I think in our internal testing, we noticed that a more powerful tool actually reduces the error rate of, um, for the parameters by a f- I don't know the numbers exactly, but it was a very big amount where it reduced the error rate.

231. 49:54

     So having a stronger model definitely helps, um, for the general purpose tools. But I think you cannot expect, uh, just because you have a very strong model that there's gonna be no, no errors, if that makes sense.

232. 50:08

     It does.

233. 50:09

     Yeah.

234. 50:19

     Thanks for the nice talk. Um, I have one question. Maybe it's slightly off topic, but, um, so now we are talking about agentic RAG, but it comes with the drawback of having higher latency against typical RAG.

235. 50:33

     So would you recommend of having, like, a second pathway for simple RAG for fast answers? And how would you, like,

236. 50:43

     uh, guide the agent to actually choose the right one? Because I think it's hard to say which question should be answered by agentic RAG or by simple RAG.

237. 50:54

     That's a good question. I, I don't think I have a, have a good answer on, like, on the top of my head right now. I'm thinking-- I was-- Maybe something related I was asked recently, if you have a RAG system, should you replace it with agentic RAG?

238. 51:13

     And I guess this is kind of going in the same direction of when do you actually need agentic RAG, right? Um,

239. 51:25

     probably for a lot of use cases. I know RAG has been killed many times, but I think the reality is that RAG is still very effective for many use cases.

240. 51:38

     Um, how would you actually switch between RAG and agentic RAG? I'm not sure because I assume it's, again, it needs some kind of almost agentic logic of switching between them.

241. 51:52

     So I'm not sure. I'm sorry.

242. 52:08

     In such cases, uh, like, um, when you have, uh, the wildcard in the GEPA, uh, example, uh, why can't we just, um, perform a hybrid tool that maybe search and replaces, uh, common wrong, uh, wildcard symbols coming from

243. 52:33

     SQL with the correct ones, for instance?

244. 52:38

     Mm, I'm not sure if I understood your question correctly.

245. 52:42

     I mean, um, there are cases in which, uh, the agent does- doesn't know the how to write the, uh, correct query because maybe he thinks, uh, the placeholders, uh, coming from SQL, um, apply it to ESQL.

246. 53:04

     But, uh, so why don't we perform a hybrid tool that, um, determinately, um, uh, search and replaces the wrong, um, placeholder, the, the percentage symbol with the asterisk?

247. 53:23

     Yeah. Actually, so the, the example I showed you of using the agent skill wasn't necessarily the, the ne- necessary solution for it. You can also, um, add just some very simple instructions on, for ESQL, don't use, um, the percentage sign as a wildcard character.

248. 53:44

     It actually works. I tried it when I was building the, the demo. Um, but then when the agent now runs in the next issue, then you start adding the next piece of documentation, then you can kind of start writing the entire ESQL document- mentation from scratch into your system prompt.

249. 54:03

     And yes, for the demo purposes, it, it would have worked, but it's probably not how you would do it necessarily when you're building something more robust, right? Because if you just a- add like little, like, band-aids every time, uh, you run into an error, then

250. 54:21

     what happens when you run into the next edge case? Does that make sense? Yeah.

251. 54:33

     Hi. Thank you for a really wonderful, um, presentation. I have a question. Uh, in the demo, we have Walk through the agentic search with DB Curious tool and also another one with shell tool.

252. 54:45

     Would you recommend in the practical use we can also kind of, uh, use-- combine both tool and then we validate the result from each of tool, and then we kind of add the confidence of the result from the LM?

253. 54:59

     Would you like recommend doing this in a practical use?

254. 55:02

     Yes. Yes. That, that's a great question. Also, again, I'm kind of cheating in this demo, right? Because I'm only showing you one tool per demo. In reality, you would have something more like a bunch of different tools where you then have to, um, decide which-- or the agent has to decide which tool to use.

255. 55:21

     I think there was a very interesting blog post by Vercel, I believe, and they did an experiment. I think it's called... If you want to look it up, I think it's called Testing is-- If Bash Is All You Need is the title, I think, of the blog post.

256. 55:36

     And they actually kind of benchmark or tested an agent with a Bash tool, an agent with a f-- just file search tools, I believe, and an agent with database tools.

257. 55:50

     And in the end, they also had one agent with a Bash tool and the database tool, and they noticed that was super interesting. Uh, for a specific set of queries, um,

258. 56:04

     where you have analytical queries, this is a specific use case. Actually, the database tool was more effective, but on the other hand, the file search tool was very effective, as you saw, for just quickly finding things.

259. 56:19

     But the very interesting aspect was the hybrid agent with the Bash tool and the, um, database tool was actually achieving the highest, um, like, highest accuracy because at first, I believe it was first using the database tool and then verifying the results with the shell, with the shell tool, and that led the agent to actually achieve better

260. 56:44

     accuracy. So I think that was a very interesting way, um, and behavior to see in, in agents.

261. 56:51

     Thanks for sharing.

262. 57:04

     Um, one second question. Um, so if we use the semantic search tool, I think in practice you probably would use some kind of threshold to cut the results to not get something if there's no answer.

263. 57:20

     But in the agen-agentic regime, would you then say, "Okay, let's put a conservative threshold such that we don't confuse our agent," or would you say the agent is smart enough, even if we retrieve results that are not really relevant, it will good enough to, to, to notice that?

264. 57:40

     Yeah. That's a great question, actually. Um, so in the examples, you probably saw some where the agent was returning-- I think in the last Gina Grab example, you see it's actually returning the top K results, where only the first one is the actual relevant one.

265. 57:57

     And I think, um, because the agent does a little bit of reasoning over whether the search results are relevant to the search query, I think it's much better or they're much better today at kind of weeding out what's not relevant.

266. 58:11

     But then you kind of run into the risk if you have longer running conversations that kind of these search results sit in your context window long term could have the problem of confusing your agent long term.

267. 58:24

     So I think it kind of-- it depends on h- your use case of how,

268. 58:33

     um, your-- like, how your agent can handle, like, irrelevant search results. But generally speaking, based on search results, it can filter out what's irrelevant.

269. 58:46

     Thanks.

270. 58:57

     Uh, thank you for the talk. Amazing. Um, are you utilizing sub-agents for these search queries? Because, yeah, if we let them decide to, is it relevant to use a question like, yeah, it's done in Ragas framework, for example, for evaluation.

271. 59:13

     I mean, sub-agents would help a lot. Do you have any experience with them?

272. 59:17

     Unfortunately not. I have not played around with sub-agents yet. I can only tell you that I know, for example, I believe in Claude Code, they're using sub-agents for doing specific search tasks.

273. 59:29

     I think there was a blog post on how they're actually using a sub-agent to, um, answer specific questions about Claude Code because it's kind of like a niche question a user would ask.

274. 59:42

     So in this case, they kind of outsourced, um, the expertise to a sub-agent. So having a sub-agent for specific niche questions I think would be interesting, but I don't have too much ex-- like, experience on it.

275. 59:58

     Okay, thanks. Uh, can I ask another question?

276. 1:00:01

     Sure.

277. 1:00:02

     Um, damn, I forget. Sorry. [laughs]

278. 1:00:07

     No worries.

279. 1:00:07

     I try to catch up later. [laughs] [laughs]

280. 1:00:26

     Um, yeah. Uh, kindly, yeah, off topic, but, um, you talked about skills, and the big benefit of skills is just have the description in the, uh, system prompt, and whenever needed, we need to load this full skill.

281. 1:00:42

     Um, do you have any recommendation when and how to clear the system prompt again? So because we want to keep the context window small and maybe for a long, um, session we might have up to ten skills, full skills in the context.

282. 1:00:59

     Mm.

283. 1:00:59

     And yeah.

284. 1:01:00

     I'm not sure. Joe, do you have a better answer, like have an idea?

285. 1:01:04

     So what we're doing behind the scenes is that we load the skills, like the definitions within the prompt, but then we use the file store to like offload like things within the context.

286. 1:01:15

     So that when the context is building up, when you're loading ten or eleven or twelve skills, we basically refer to, we replace the context and the messages with the referral where, where it is in the file store, and yeah.

287. 1:01:27

     So we keep the context very small. Sorry. I, I've finished that bit, yeah. [laughs]

288. 1:01:36

     Um, but yeah, so like the way that we are doing it behind the scenes is that we're, we're, we're providing like that kind of progressive disclosure of skills. So we're providing those, um, the skill names and descriptions, the location within the file store, and then from the file store we're loading into the context window when we need

289. 1:01:57

     that skill, and then we offload it once it's, once it's the prog- you know, the context window progresses ahead of time around that. So we have this kind of more on-demand, uh, one around that.

290. 1:02:09

     And that's the same with our like compaction, like com- of context, and that's what I would advise you to do. Like for some of the questions too, try and use the file store as much as you can and have those tools as well, like being able to grep the file store for when you wanna see previous tool

291. 1:02:25

     results and then, and then, uh, use it from there.

292. 1:02:33

     Thanks. That's my colleague Joe from Elastic as well. [laughs]

293. 1:02:40

     Awesome. If there are no more question, I will let you guys go into the coffee break. Again, don't forget to grab yourself some stickers, and happy to catch up in the halls if anyone's interested.

294. 1:02:54

     Thanks so much. [clapping] [outro jingle]