Architecting and Testing Controllable Agents

Talk transcript

1. 0:00

   [on hold music] Good to be here. Good to see a fair number of people.

2. 0:16

   It's early, so I wasn't sure if anyone would come, but thank you for coming. Um, one quick note, I just put my slides on Twitter. I wasn't sure of the best way to access everyone.

3. 0:26

   I'm [REDACTED:username]. Um, if there's another way I can get everyone slides, um, then... Yeah, I see some people opening. Um, so the slides will link to a few different, uh, it'll link to a Colab and other notebooks I'm providing, so all the code will be available for you.

4. 0:44

   Um, yeah, okay, good. As I see people finding it, that's fantastic. Um, may- if there's a better way, let me know, but, uh, figured this is somewhat easy. Um, well, it's great to be here.

5. 0:54

   We have a bit of time. So I think the format is I'll, like, lay out some slides to kind of set the preliminaries and, like, kind of give the big picture.

6. 1:02

   And then there'll be a bunch of time where I can just walk around, talk to people, and because we have, I guess, three hours. So I think the idea of this will be a hands-on workshop.

7. 1:10

   I provide a bunch of starter code. And also, one of my slide shows, it'll be kind of a g- choose your own adventure format. Um, so why don't I kick it off?

8. 1:18

   I think maybe one or two people are still coming in, but... So the, the theme here is building, uh, and testing reliable agents. And let me go to slideshow mode here.

9. 1:29

   Um, and maybe I'll just kind of start with, like, the very basics. You know, LLM applications follow a general control flow of some sort. You start, usually it's a user input, there's some set of steps, and then you end.

10. 1:43

    Um, and you've heard a lot about chains. You know, when you build applications, oftentimes we talk about this idea of chains, and chain is just basically, you know, it is some control flow set by the developer.

11. 1:54

    Uh, again, you start, proceed through some steps, and you end. So retrieval-augmented generation is a super popular application many-- some of you may be familiar with, um, basically refers to retrieving documents from an index and passing them to an LLM.

12. 2:09

    This is a good example of a chain. It's a control flow set by a user. The question's, you know, provided, um, vector store retrieves them, passed to an LLM, LLM produces an answer.

13. 2:20

    So this is kind of a classic chain.

14. 2:23

    Now, when you get into agents, there's a lot of different confusing interpretations. What is an agent? Here's the way I might think about it, which is just a really simple kind of framing, is agent is just one of the control flows set by an LLM.

15. 2:35

    And so you can imagine we talked about this process of you start your app, step one, step two. In this case, I have an LLM in there. An LLM looks at the output of step one, makes a decision.

16. 2:45

    Do I go back and do I proceed? So that's, like, the way to-- simple way to think about an agent.

17. 2:50

    So again, chains, developer-defined control flow. I set it ahead of time. I follow some set of steps every time. An agent, an LLM kind of determines the control flow.

18. 2:59

    An LLM makes a decision about where to go inside my application. That's one simple way to think about it.

19. 3:06

    Now, you hear about function calling a lot, and this is kind of a confusing topic, so I wanna talk through it kinda carefully. Um, agents typically use function calling to kind of determine what step to go to.

20. 3:18

    So usually, the way this works is, what you do is you basically give the LLM awareness of some number of tools or steps that it can take. So in my little example here, I define this tool and this little decorator as a LangChain thing, but the point is, I have some step, it's some function.

21. 3:35

    I'm defining it as a tool, and I'm wanna-- I'm binding it to the LLM. So then the LLM has awareness of this tool, and here's the key point. When it sees an input, like what is the output of step two, it actually produces the payload needed to run that tool.

22. 3:49

    Now, this is often confusing. Remember, LLMs are just string to string. They don't have the ability to magically call some function. What they can do is produce the payload or arguments needed to run that function and the function name.

23. 4:00

    So really think about tool calling or function calling as just an LLM producing a structured output. Still, you know, obviously a string, but it's a structured output that then can be used to call a tool.

24. 4:10

    So that's all that function calling is. And you might have heard of re-react agents. So the way to think about this is, it's just basically binding some set of tools to my LLM.

25. 4:21

    And again, we talked about tool calling. So LLM makes decisions about what step or what tool to use. And you have some node that will call that tool. So LLM says, "Okay, run step one."

26. 4:32

    I have some node that runs step one and passes the output of step one back to my agent. React typically stands for, like, basically action. So the LLM chooses the action.

27. 4:42

    A tool is gen-- a tool is run. It observes the output. That's what goes back to the agent. Observe that tool response, thinks about what to do next, maybe runs another tool, and this runs in a loop until you end.

28. 4:52

    And usually, the end condition is the LLM just outputs a, a string response, not a tool call. So this is the way to think about a classic react agent.

29. 5:01

    Um, and it's really flexible. That's the nice thing about a react agent. So basically, it can implement many different control flows. It can do step one only, step two, one, two, two, one.

30. 5:12

    That's the beauty of, of these open-ended style react agents.

31. 5:16

    And these have a lot of promise. These kind of flexible tool calling agents were really hyped last year. They're still really hyped. It's really exciting because they're flexible. They're open-ended.

32. 5:25

    You can give them a task, give them some tools, and they can just execute arbitrary control flows, given those tools, to solve open-ended problems. The catch is, and this is kinda the crux of what we're getting to with this workshop, is they do have poor reliability, or they can.

33. 5:40

    So you can get caught if you've played with agents. Sometimes you can see, they kinda get caught on one step, and they keep calling the same tool.

34. 5:47

    And, you know, really, this is often caused by LLM non-determinism. LLMs are not deterministic. And also errors in tool calling. So tool calling is kind of tricky. If you think about it, LLM has to basically pick the right tool given the input, and it has to pick the right payload.

35. 6:03

    So it has to produce the right inputs needed to run the tool. And these both can break. So here's a good example. The tool I'm passing is step two.

36. 6:10

    And the LLM is saying the tool name to run is step three, so that's obviously wrong. Or I'm passing what is step two of the input three, and the LLM says, "Okay, pass four."

37. 6:21

    So these-- both of these errors can happen. Tool calling is a tricky thing, and it's exacerbated. If you pass an LLM five tools, ten tools, it actually gets worse.

38. 6:29

    If you have very long dialogues, it gets worse. And so this idea of open-ended tool calling agents is really promising, it's really exciting, but it's really challenging because of these issues that we were mentioning here.

39. 6:41

    So this is kind of the build-up here. Like, so can we envision something in the middle? So again, we talked about chains. They are not flexible, but they're very reliable.

40. 6:49

    This chain will always run step one, two in order. We talked about, like, ReAct agents on the other extreme. They're extremely flexible. They can run any sequence of tool calls, you know, can run step one, two, step one only, two only, two, one, but they do have reliability issues.

41. 7:03

    So can we imagine something in the middle that's both flexible and reliable?

42. 7:08

    So here's kinda the setup, and it's, like, kinda the intuition. Like, a lot of times in many applications, you have some idea of what you want the thing to do, um, every time.

43. 7:16

    So some parts of the application may be fixed. Like, the developer can set, okay, I always wanna run step one, and I wanna end with step two.

44. 7:24

    And you can inject an LLM in certain places that you want there to be some kinda branching or kind of optionality in the control flow. Okay. So this is the motivation for what we call LangGraph.

45. 7:35

    So LangGraph is basically a, a library from the LangChain team that can be used to express control flows as graphs. And it is a very general tool, and I put out a bunch of videos on it, and we're gonna use it today.

46. 7:48

    And by the end of this, uh, you will all have an agent that runs reliably using LangGraph, hopefully. And then we'll, we'll, we'll see. So you could-- you should test me on that.

47. 7:57

    If it, if, if things don't work for you, then, then we'll work it out. But the idea is kind of this. This graph has some set of nodes and edges.

48. 8:06

    So nodes you can think about are basically... Well, maybe I should start with this. This graph has something called state, so it's like short-term memory that lives across the lifetime of this graph that contains things you wanna operate on.

49. 8:19

    Nodes modify the state in some way. So basically, each node can, like, call a tool and can modify the state. Edges just make decisions about what node to go to next, okay?

50. 8:28

    So you basically have this idea of memory, and this is the same as common agents, right? Me- agents are characterized by having tool calling, um, and, and short-term memory as well as planning.

51. 8:37

    Those same things are present in LangGraph. Memory is the state that lives across your graph. Tools exist within your nodes. And planning, uh, basically, you can w- incorporate LLM-dictated decision-making in the edges of your graph.

52. 8:51

    So, like, why is this interesting, and where has this been cropping up? We've actually been seeing this theme crop up a lot of places. Um, so there's a really interesting paper.

53. 8:59

    There's actually a few I really like. This one's called Corrective RAG. And the idea is pretty simple. Like, with a na- naive RAG pipeline, you're doing a retrieval, you're taking retrieved docs, and you're generating your answer.

54. 9:09

    Corrective RAG, like, is doing one step more where it's saying, "Well, why don't we reflect on the docs we retrieved and ask, are they actually relevant?" You can have lots of issues with retrieval.

55. 9:17

    You can reflect on the documents, see if they're relevant. If they're not relevant, you can do different things. You can kick out and do a web search. So it makes your application a lot more dynamic to poor quality retrieval.

56. 9:26

    So this is one of the first videos I put out on LangGraph back in February. It was, it was very popular, and I basically showed you can build Corrective RAG inside LangGraph, and it's super simple.

57. 9:34

    This is what the graph looks like. I do retrieval, I grade my documents, and we're gonna... Actually, we're gonna do this today, and I have a bunch of code for you that does exactly this.

58. 9:42

    So we're gonna go way in detail on this one. Um, but this is kinda the setup, and I showed this working. I showed it works locally with Ollama using, in that-- at that time, it was, um, Mistral 7B, um, and it works really well.

59. 9:56

    So this is like one simple illustration of how you can use LangGraph to build kind of a self-reflective or corrective RAG application. Now, another cool paper was called SelfRAG, which actually looked at the an- the generation.

60. 10:07

    So basically, we're all familiar with the idea of hallucinations. It's a real problem. Um, instead of just allowing hallucinations to propagate to the user, you can actually reflect on the answer relative to the documents and catch hallucinations.

61. 10:19

    If there's hallucinations, you can, like, basically do different things, and they sh- they propose a few ideas here. Um, I implemented this, and this is a, this is actually our most popular video of all time.

62. 10:28

    So this was showing LangGraph and Llama three, um, implementing three different things: Corrective RAG, which we just talked about, the SelfRAG thing of hallucination checking, and this Adaptive RAG thing.

63. 10:38

    So I can kinda walk through it. This all runs in LangGraph locally, uh, and I have the notebook here. If you wanna test that today, you definitely could. Um, so that's the point.

64. 10:47

    It's reliable enough to run this whole thing locally. So what's happening here is I take a question, I route it either to my index or to web search. Um, I then retrieve documents.

65. 10:56

    I grade them for relevance. If any are not relevant, I kick out and do a web search to supplement my retrieval. If they're relevant, I generate my answer, I check it for hallucinations, and then I finally check it for answer relevance.

66. 11:08

    So basically, does it have hallucinations relative to my documents, and does it answer my question? If all that passes, I go-- I finish and return that to the user.

67. 11:17

    So this is kind of like a complex RAG flow, but with LangGraph, you can actually run this on your laptop. Um, it is reliable enough to run a laptop with LangGraph.

68. 11:25

    And the intuition, again, is that you're constraining the control flow. You're allowing the LLM to make certain decisions but at very discrete points. If you implement this as a RAG agent, this could be very open-ended and a lot of opportunities for breakage.

69. 11:37

    And so that's the real intuition here. Um, now a final theme is Karpathy kind of mentioned this idea of flow engineering, uh, related to this AlphaCodium paper, a really nice paper on code generation.

70. 11:48

    And the intuition here is, um, produce a code solution. They tested this on a bunch of coding challenges. Produce a code solution and check it against a number of unit tests, auto-generated or pre-existing.

71. 11:59

    And basically, if it fails unit tests, feed back those errors to LLM and try again. Really simple idea Uh, I implemented this in LangGraph. Again, the code is here.

72. 12:07

    Um, and this works really well. So I basically, I share a blog post as well. I ran this on our internal, uh, coding. W-We have an internal application for RAG, uh, at LangChain, and we're actually working on implementing this right now in production, uh, because the performance is way better.

73. 12:24

    And a common thing this can fix with code generation and code solutions is hallucinations at import. So we see that a lot with our RAG app. Um, so what I did was I very simply implemented a unit test for import checks.

74. 12:37

    Just run that, it significantly improves performance, um, relative to without doing it. And so we're actually working on implementing this in our internal RAG system. So super simple idea that can really improve code generation.

75. 12:50

    So, you know, if, if I kinda like back up what, what did we talk about? I mean, we talked about chains. They are not flexible, which is fine in some cases.

76. 12:58

    But a lot of interesting newer papers with RAG, for example, this idea of self-reflection is really beneficial. Uh, the ability to kind of self-correct applications can be really beneficial, uh, not, beyond RAG as well for coding.

77. 13:10

    Um, so chains are very reliable, but they're not flexible. Now, if you go to the other end, like a classic ReAct agent is very flexible. It can implement any sequence control flows through your different tools, uh, but it does have reliability problems due to things we talked about, nondeterminism, tool calling errors.

78. 13:26

    And LangGraph kinda sits in the middle where you can actually implement these user-defined pl- slash LLM-gated control flows, um, and they can actually be extremely reliable because of that constraint.

79. 13:38

    Um, they are less flexible than a classic ReAct agent, so that is true. So for a very open-ended task, I do agree, maybe you do need, you know, a very open-ended, more like autonomous style ReAct agent.

80. 13:49

    But for a lot of applications that our customers are, are like, are working on and seeing, um, these kinds of like hybrid flows are sufficient and you gain a lot of reliability.

81. 14:00

    And so we've talked to a lot of companies that have implemented LangGraph successfully, um, for agentic flows for this reason, 'cause reliability is just incredibly important in a production setting.

82. 14:09

    Um, so this gets into, if you look at the slides, I have a few different notebooks. Um, and what I show is we talked about corrective RAG. These notebooks show how to build corrective RAG yourself, and I thought that's a fun starting application.

83. 14:24

    It's a really popular one. Um, it's super simple. There are not many dependencies. Um, you can use your own webs- whatever tool you wanna use for web search. You can use other things as well.

84. 14:33

    You sh- have a look at the notebooks and I'll kinda walk around. Um, and we're gonna, we're gonna keep going. I just wanna like, this is just like a placeholder here.

85. 14:41

    Um, but so if you wanna test this locally, if you have a laptop capable of running things locally, then we have a notebook to support that. I use Ollama.

86. 14:49

    I can talk a lot about that. That's a really cool thing. Um, if you don't, then I have two options for you. So one is a Colab, so that's probably the easiest.

87. 15:00

    Um, if there's issues, let me know, but I've tested it. Um, so if you have a Google account, you can spin up a Colab. All you need is a few API keys depending on what models you wanna use.

88. 15:10

    It's all kinda there. You can set those accordingly. And I also have a notebook. So this just kind of is like a, kind of a, gives you a roadmap of the different things you can try today since this is a workshop format.

89. 15:21

    And I'll, I'll just be walking around and we'll, we'll do questions for a while. But I wanna talk about the second half of this, of this, you know, story.

90. 15:29

    Um, so one of the themes we're seeing a lot, I think you're gonna hear a lot at, at this conference, is the challenge of testing and evaluation. And this is a real pain point.

91. 15:39

    Like for example, how do I actually know that my LangGraph agent is more reliable than the ReAct agent? Um, how do I know what LLM to use? How do I know what prompt to use, right?

92. 15:49

    So testing agents is, is-- Testing in general is really hard, and agents in particular is challenging.

93. 15:56

    So there's kinda three types of testing loops I like to think about. Uh, one is this in-app error correction, and that's actually what we just talked about. So LangGraph and LangGraph agents are really good for that.

94. 16:07

    So basically, in-app error handling, uh, where you can catch and fix errors is really useful for code generation, for RAG. We just talked about that, so that's like placeholder one.

95. 16:16

    Now we get into this idea of pre-production testing. Um, and then finally production monitoring, and I wanna introduce a few ideas on the latter two. Um, so we just talked through this.

96. 16:26

    W-we're, here we're gonna build Corrective RAG a few different ways, and I just showed the choose your own adventure stuff, and so this is just kind of reiterating that.

97. 16:33

    Um, but I wanna show you some other things. So LangSmith is a tool from the LangChain team that supports testing and evaluation as well as monitoring and tracing. And so we've seen a lot of interest in this, and it's, it's quite popular.

98. 16:47

    Um, it is really useful for doing these types of testings and evaluations. So the key idea behind LangSmith, and the notebooks actually have this, so this is totally optional.

99. 16:57

    If you just wanna build an agent, that's totally fine. If you wanna also test it, you can, you can use LangSmith. You don't have to, of course, but I have it all set up to use LangSmith if you want.

100. 17:06

     It's free to use, of course. Um, and so the idea is there's kind of four components that I like to think about when it comes to testing slash evaluation.

101. 17:14

     You have some datasets. That's some set of examples you wanna test on. So say you have a RAG app. That's like a set of ground truth question-answer pairs you've built.

102. 17:21

     Like you're testing your system. You have question-answer pairs that you know are correct. Can your system produce those answers? You know, how many will it actually get right? You have your application.

103. 17:29

     That's your agent, that's your RAG app, that's your code app, whatever that is. That's your application.

104. 17:34

     Now, the thing that's often the trickiest, you have this evaluator thing, and I'm gonna, I-- And the notebooks show you in detail, but this evaluator is something as simple as a, a user-defined function that can implement a few different things.

105. 17:45

     You can think about using an LLM to actually judge your output. So in that case, let's take RAG as an example. My application produces an answer. I have a ground truth answer.

106. 17:53

     You can actually have an LLM look at those two answers jointly and reason, is it correct? And this is often very effective. It requires some prompt engineering. I have some nice templates in the notebooks to show you.

107. 18:03

     But this is something that's very popular. This idea of LLM as judge evaluators is very interesting. A lot of peop- You actually, you'll probably hear about it this week.

108. 18:09

     Uh, it's a really good theme. It's still kind of in development, but that's like one placeholder to keep in mind. So for one option for kind of testing is this idea of using LLMs themselves.

109. 18:20

     The other is building your own heuristic evaluator, so a custom evaluator of some sort. And actually, the notebooks that I share have both. Um,

110. 18:30

     and so we're actually gonna-- The notebooks actually show how to evaluate an agent specifically. And there's a few different things you can look at with an agent. So one is, if you go to that far right in blue, the end-to-end performance.

111. 18:41

     So our notebooks are basically gonna be a RAG agent. The eval set has five questions. Um, and I basically have a set of question-answer pairs. So basically, I'm gonna compare my agent answers to reference answers, and we'll walk through that in the notebook.

112. 18:55

     But that's kinda one thing. I just wanna introduce the ideas. So one big idea is you can, you can evaluate the end-to-end performance of your agent, right? You don't care anything what's happening inside the end-to-end performance.

113. 19:05

     The other, which I actually like to look at a lot, is this thing on top. What are actually the tool calls that the agent executed? This is how you can actually test the agent's reasoning.

114. 19:13

     So what you see often with agents is they can make some weird trajectory of tool calls that's highly inefficient, but still gets to the right answer. You don't get that if you only look at the answer and say, "Okay, it's got the right answer."

115. 19:22

     But if you look at the trajectory, it's some crazy path, and so you wanna actually look at both, like how efficient, how correct is the trajectory, and does it get the right answer, right?

116. 19:29

     And so the notebooks I share actually do both. Um, now this is, uh, actually an evaluation that I ran, and this data set is public. This is on the agents that we actually just talked about.

117. 19:41

     So this is kinda what you see when you open LangSmith. So these are different experiment names. Um, this is just saying, like, I've run three rep-replicates of each experiment, and these are my aggregate scores.

118. 19:52

     So this first score is basically the answer correctness thing, and the second score is like the tool use trajectory or like does it doos-- use the right reasoning trace.

119. 20:01

     And I can go through my experiments. So this top one is actually... This is kinda cool. This is actually my local agent running on my laptop with LangGraph, okay?

120. 20:10

     It's a five-question eval set, small eval set, just a bunch of... It's some very small test examples. But basically, my local agent does not... does fine. It does 60% in terms of the ultimate answer, so that's not amazing.

121. 20:22

     Uh, but it does do very well in terms of the tool calling trajectory, so it's very reliable in terms of reasoning. Um, it's an eight billion parameter model, so I basically, I think the quality of its outputs are a little bit lower than you might see with larger models.

122. 20:36

     Um, now Fire Function v2 is another option. It's, it's basically a fine-tuned Llama 70B from Fireworks. This one with LangGraph, so this is actually shown in this top, actually gets up to 80%.

123. 20:47

     So very strong performance in terms of answers and 100%, again, in terms of tool calling. So the key observation here is the tool calling or reasoning is consistent, whether you're using a local model or a 70 billion parameter model with LangGraph.

124. 21:00

     So you get very high consistency in your tool calling. The answer quality degrades. That's more an LLM capacity problem, but the reasoning of the agent is consistent. So that's the key point.

125. 21:11

     Now, here's where it gets interesting. Fire Function v2, again, that's Llama 70B, this is with, uh, a ReAct agent. What you can see here is the answer quality is degraded, but here's the interesting thing.

126. 21:22

     The tool calling trajectories are really bad, and this again gets back to that problem with ReAct agents. They're open-ended. They can choose arbitrary sequences of tool calls, and you can deviate really quickly from your expected trajectory.

127. 21:33

     So that's the key intuition here. Now, the final two are GPT-4os. That's obviously a very flagship model. It's maybe number two now relative-- again, sonnet on the chatbot arena at least.

128. 21:44

     You know, again, answers ultimately are, are strong. The tool calling though, even here, is degraded. So basically, it follows some weird trajectories to get to its answers that are unexpected.

129. 21:54

     So what's the high level point here? The high level point is LangGraph allows you to significantly constrain the control flow of your app and get higher reliability. And if you look at these tool calling scores, it's very kinda consistent going all the way down to local models.

130. 22:07

     It follows the same se-sequence every time. ReAct agents kinda go off the rails much more easily. The answer performance is really a function of the model capacity. So using an eight billion parameter model locally, the answer quality is lower than a 70 billion.

131. 22:21

     That's to be expected. But the reasoning of my app is consistent and strong, so that's the key thing that you kind of get with LangGraph, and this is all public.

132. 22:30

     And hopefully, some of you will actually, you know, implement this or, or reproduce this today. Um, and this is just walking through those same insights I just mentioned. Um, uh, and then deployment, we're gonna be talking later this week.

133. 22:42

     We have an announcement related to deployment of LangGraph. So this is actually a very good setup. Um, if you're playing with LangGraph, you enjoy working with it, we're gonna have some really nice options for deploying later this week.

134. 22:51

     And so Harrison will be here on Thursday to give a keynote on that one. Um, and if you've deployed, we also have some really nice tools in LangSmith to actually monitor deployment.

135. 23:00

     And this is not as relevant for this workshop. It's something just to be aware of, um, I can talk about if you're interested. Um, so maybe to, to close out, so this is a really nice write-up.

136. 23:11

     Um, these guys are actually gonna give a keynote later this week. It's, uh, Jason and company, um, Hamel and others. And they kinda made a really nice point that the model is not the moat.

137. 23:21

     Like, LLMs are always changing. The moat is really the systems you build around your application. That's what we talked about today. Like, do you have an orchestration framework, for example, like LangGraph?

138. 23:30

     Do you have an evaluation chassis like LangSmith? And again, you don't have to use LangGraph. You don't have to use LangSmith for these things, but this workshop will introduce these ideas to you.

139. 23:38

     And frankly, I think it's important just to understand the ideas rather than the implementation. Whether or not you use LangGraph, whether or not you use LangSmith, I think understanding these, these principles is still helpful.

140. 23:48

     Um, but you know, an evaluation chassis, guardrails, data flywheel, these are like the components that, that give you the ability to improve your app over time. That's the, that's really the big idea.

141. 23:58

     That's the goal. Um, and I think you'll hear more on that later this week. The, the goal here is how are you measuring improvement of your app and ensuring it always gets better?

142. 24:05

     That's what we're actually trying to achieve here. Um, and that's kinda what evaluation is giving you. And yeah, this, this is kinda my last slide, and then maybe we can just move into maybe some Q&A.

143. 24:16

     I can actually show the notebooks themselves if you wanna walk through them together. Um, I mean, I'll just do that, and I'll let you guys Kind of hack on them in parallel as I walk through them.

144. 24:28

     Um, and then I can just go walk around and talk to people, something like that. So, you know, the three types of feedback loops. You have this design phase feedback, some of it LangGraph.

145. 24:38

     In-app error handling, that's kind of step one we talked about. Cool examples there for coding, for RAG. A lot of nice papers, really promising. I'm very excited about anything you can do here with terms of kind of agentic self-correction, self-reflection in your app itself.

146. 24:53

     Pre-production testing, we just talked through that. Building evaluation sets, running evaluations, testing for an agent, like your tool use trajectory, your answer quality, all really interesting and important. And then in production phase, production monitoring, um, this gets into...

147. 25:08

     We didn't talk about it too much, but basically this stuff. So basically, you can have evaluators running on your app in production, looking at inputs, looking at outputs, tagging them accordingly, and they can go back and look, um, later.

148. 25:20

     So that's kind of the, the setup here. I know that's probably a lot, and we went about half an hour. So I-- if there's any questions, I can just open it up, um, and we can kind of talk through stuff.

149. 25:31

     I can also start ripping through some of the notebooks just to kind of give you an overview of the code itself. Uh, but if there's any questions here, maybe, um, you know, happy to take a few.

150. 25:40

     Give, give you a minute to digest all that.

151. 25:43

     Yeah.

152. 25:44

     Is there a non-Twitter link to your slides here?

153. 25:47

     Uh, that's a good point. Let's see. Um-

154. 25:51

     Question.

155. 25:52

     Uh, is there a non-Twitter link to the slides? Uh, let's see if the conference organizers give me some... I don't know if I have an email list for everyone in here.

156. 26:02

     Someone can just post to the general channel.

157. 26:04

     Is there a-- Okay. Is there a Slack? Yeah, if someone can put... I actually didn't know there's a Slack, so that's very helpful. Yeah, if there's a Slack or an app for this conference, then please someone post.

158. 26:13

     I appreciate that. I actually didn't know that.

159. 26:17

     Thank you for that question. Yes, sir.

160. 26:21

     So about the, uh, testing and the evaluation, uh, does it really scale to, uh, predict what, what, uh, the num- the exact sequence of these agents? Uh, really, if it's, uh, smart enough dealing with a complex problem, it's, uh, it's hard to, to say exactly how it is it achieved that task, right?

161. 26:41

     Okay. This is a very good point, so I'm gonna repeat the question now. Um, how do you evaluate an agent's, like, reasoning trajectory if it's a large and open-ended problem that can be solved in many different ways?

162. 26:55

     In that particular case, you are right. It is hard to enumerate a specific trajectory, uh, that is actually reasonable. For really open-ended, long, uh, running type problems, trajectory evaluation may not be appropriate.

163. 27:09

     So one, one thing I would think about is, can you define a few canonical trajectories of tool use through your application? So it, it depends on the stat. If it's a very long-running agent, I think it's probably infeasible.

164. 27:22

     If it is a shorter run agent where it's like you expect something in the order of maybe five to ten steps, you can probably enumerate some set of reasonable trajectories and basically check does it follow any of these.

165. 27:32

     You can also do things... So you can do things like this. You can do things like check for the re-repeat of certain tool calls. You can be very flexible with this.

166. 27:43

     I mean, it's kind of open to you. So you can look for, like, is it repeating certain tool calls? Um, you can look at recall. Like, is it for sure calling this tool or not?

167. 27:52

     Um, so you can actually be very flexible. And actually, the way we set up Lang-LangSmith evaluator for this, it's just a simple function that you can define yourself. So it's a very good point.

168. 28:01

     You can be incred- you can be arbitrarily creative about how you evaluate that. But I would say for very long running, you're right. You can't really articulate step one, two, and three.

169. 28:10

     Um, but I would then think about more like evaluating is it repeating steps? Can you evaluate for clearly aberrant behaviors, excessive number of tool use repeats, excessive number of overall tool calls?

170. 28:22

     So kind of like guardrails related to, like, kind of clear aberrant behavior. If it's very short term, you can actually enumerate the trajectory specifically, but that's a good question.

171. 28:30

     So in the code, you can see we actually lay out a custom function. You can define that yourself. So that's a, that's a very good point, though. Yeah. Yep.

172. 28:40

     Can you also mention the cases with the human in the loop?

173. 28:45

     Yeah. Human in the loop is a good one. The workshop, uh, notebooks I shared do not have that, but LangGraph does have some good support for human in the loop, and I can share with you some notebooks that showcase that.

174. 28:58

     Also, what we have, uh, shipping on Thursday has very good support for human in the loop. So, um, I will share some notebooks with you for that and wait for Thursday for even more there. [sniffs]

175. 29:11

     So, um, if we're building a RAG-like application, right, pre-production, we can do this testing framework using a known set of this is the question, this is the proper answer it should have been to evaluate things end to end.

176. 29:26

     Um, what I've been struggling to figure out is, like, the right way to approach that in production where we don't know what the right answer is, and I'm wondering if there's like certain best materials.

177. 29:38

     Yeah. So the question was, this is actually a really good one. Um, for RAG, I'm just gonna go to our docs because I actually wrote a doc on this recently.

178. 29:47

     So, um, okay, you can still see my slides. For RAG, in a pre-production setting, it's easy to define, or may- not even easy, but you can define a set of question-answer pairs and evaluate them.

179. 29:59

     When you're in production, though, how do you evaluate your app 'cause you don't have a ground truth answer? So what are other things you can actually evaluate for a RAG app that don't require a reference?

180. 30:07

     Yeah. So there is a, a conceptual guide that I will share. So this is actually our RAG section. I have kind of a nice overview of this. Um, there's actually a few different things you can evaluate for RAG that don't require a reference, um, that are very useful.

181. 30:26

     Yeah. So it's this right here. So this is like a typical RAG flow.

182. 30:34

     So I have a question, I retrieve documents, I pass them to an LLM, I get an answer, right? What we just talked about and we showed is comparing your answer to some reference answer.

183. 30:43

     Now, this is, to be honest, pretty hard to do. You have to build an eval set of question-answer pairs. Very important to do, but it's not easy. So what else can you do?

184. 30:52

     So some of these we've seen that are really easy and actually pretty popular. There's three different types of grading you can do that don't require a reference, that are like internal checks you can run.

185. 31:02

     Y- I mean, you can run them online. So one is retrieval grading. So basically looking at the docu-- your retrieved documents relative to your question. So like an internal con- self-consistency check.

186. 31:12

     So this is actually a great check to run, and actually, the Corrective RAG paper that, or the Corrective RAG thing that is in the cookbooks that I share here does this.

187. 31:21

     So you can play with the prompt and all that. Um, but basically, this is just checking the consistency be- between your retrieved docs relative to your question. You can do that, and we have some really good prompts for that.

188. 31:32

     Another one I like is just comparing your answer to your question. Have an LLM look at, "Here's my answer, here's the question. Is this sane? Are they related?" And this is a really nice check, uh, just for like, you know, of course, you don't have a reference answer, but, like, you can still sanity check and say, "Does

189. 31:46

     this deviate significantly from what the questioner is asking?" The other, this is a great one, is hallucination, and this is, this is super intuitive. Compare your answer to the retrieved documents.

190. 31:56

     So if the LLM went off the rails and didn't ground the answer properly and you hallucinated, you can catch that really easily. Um, and so, um, I need to get on this Slack 'cause I wanna share this link with you.

191. 32:09

     Uh, I'll figure that out. I'll, I'll find you. But, uh,

192. 32:13

     this is in our LangSmith docs. If you search LangSmith evaluation concepts, we have-- I actually have a bunch of videos that showcase how to do this, and, um, I have a bunch of code as well.

193. 32:22

     So but those are three things you can do that don't require a reference, and we do run those in as online evaluation with our application. So-

194. 32:31

     Awesome.

195. 32:31

     Yeah. Yep.

196. 32:33

     Do you have any thoughts on unit testing with LangGraph or LangSmith?

197. 32:37

     Yeah, unit testing. So, um, y- yeah, okay. So do we have any thoughts on unit testing? So LangSmith supports pytest as a unit test, but basically

198. 32:51

     it depends what you mean by unit test. Typically, like conventional software engineering unit tests, um,

199. 32:57

     are, are very effectively done in things like pytest, and there's a lot of frameworks for that. What I like to think about in unit testing with respect to LLM apps is, um, kind of like what we show, um,

200. 33:09

     in this, uh, code generation example. Um, here we use some really simple unit tests just for like imports and code execution. Um, simple unit tests like this can actually run in your app itself.

201. 33:21

     So basically one place you can think about putting unit tests are actually in your app itself within LangGraph for self, for kind of in-app error handling or self-correction. So that's like one place for unit tests that's kind of interesting and new with LLM applications.

202. 33:33

     They can live inside your app itself. Um, another good one for unit tests within app is, uh, if you're doing structured output anywhere in your application, which is like a really common thing people like, confirm the schema is correct.

203. 33:45

     That's another good use case for unit tests within your application. Though those also, both of those things could also be done independently, like in CI, outside of your application.

204. 33:55

     So we are going to have more integration support for CI with LangSmith soon. Uh, I will check with the team on that. Um, but I think the interesting idea for unit tests with LLM applications is this idea of inline within your app itself.

205. 34:10

     Because LLMs are so good at self-correcting, if you run unit testing in your application, they can often catch the error and then correct themself. And unit tests are fast and cheap to run, so it's actually a really nice kinda like piece of alpha that...

206. 34:22

     In fact, that's exactly what Karpathy was mentioning here, that, you know, running unit tests in line with your application is, is actually really quite nice, um, and produce significant improvement in performance in AlphaCodiums.

207. 34:37

     Cool. Yep.

208. 34:39

     Is it, um, got to say that you can use a smaller LLM for unit test-

209. 34:45

     Okay

210. 34:45

     ... the ability to fine-tune can be fast.

211. 34:48

     Yeah, yeah. So what the question was, if I want to do some of this in-app error correction stuff, so let's take this example, the Corrective RAG thing. If I actually want this to run in my application, it obviously needs to be super fast.

212. 35:02

     So that's actually what we've seen. The tricks we've seen here are basically use very fast, smaller LLMs. So you mentioned, for example, even the ability to fine-tune. That's actually a good idea.

213. 35:12

     Uh, if you have a judging task that is very consistent, it's a very good use case for fine-tuning, actually. Fine-tune a small, low capacity, extremely fast, and effectively very cheap to deploy model.

214. 35:24

     That's a very good idea. We've seen people do that. Um, also use very simple grading criteria. Don't have some kind of arbitrary scale from zero to five with high cognitive load the LLM has to think about.

215. 35:35

     Yes/no. Very simple binary grading. E- even for some of the stuff, you can even be old school and fine-tune a classifier. But basically, really simple, lightweight, fast LLM as judge style classifier wouldn't be an LLM necessarily, but, but basically very simple, fast tests for in-app.

216. 35:54

     Anything like kind of in runtime you will need or want. Another cool use case for this or, or another like interesting option for this is Groq is very, very fast, kind of with their LPU stuff.

217. 36:05

     Um, and they actually would be a very interesting option. Uh, and we've done some work on that with Groq. Basically, for any of these kind of in-app LLM as judge error correction things, using something like Groq, which is extremely fast.

218. 36:17

     But it's a very good insight. Fine-tuning your own model is actually a really good i-- We've seen people do that for these types of any- anything with LangGraph in in-app error correction.

219. 36:28

     Cool. Yep.

220. 36:29

     Does it support multi-agent, um, and the communication-

221. 36:35

     Yeah. We have some good, uh, cookbooks talking about multi-agent, um, which I, again, will need to find a way to share with you. Um,

222. 36:46

     uh, yep. So we have this. If you go to LangGraph, LangGraph GitHub, examples, multi-agent, there's a few different notebooks here, uh, that are worth checking out.

223. 37:02

     Yep.

224. 37:04

     Uh, does LangGraph allow you to define, like, cyclical graphs? And if so, how do you prevent yourself from, like, getting stuck in a loop?

225. 37:12

     Yeah. Yeah. So actually, LangGraph is specifically designed for cycles. Um, so some of the exam- Like, what we're showing today is only a branch, so it's a simpler graph.

226. 37:21

     Um, but for example, the, uh, ReAct agent we'll show today is a cycle, so it's basically gonna f- gonna continue in a loop just like this. And what you do is you set a recursion limit in your LangGraph config.

227. 37:34

     So you basically tell it to only proceed for some number of cycles, and this is default set for you. I believe it's, like, 20 or something like that. Uh, but that's what you're gonna, that's what you're going to want to do.

228. 37:50

     Yep.

229. 37:50

     What about, like, timing of the responses? Is there, like, any way to control that?

230. 37:56

     Um, so the question was about timing of the responses. So do you mean, like, if you're implementing some kind of self-correction, how long that takes? Well, that's, that's kind of a-- That gets back to the question, uh, that, that this g- gentleman asked.

231. 38:11

     Um, it depends a lot on the LLM you choose to use for your judging and its latency. So that's kind of where

232. 38:19

     if you're... So maybe there's two s- sides of this. One side of it is choosing an LLM that's very, very fast, and that's very important to do. Could be something like Groq, could be a fine-tuned, like, deployment that you do yourself, could be a GPT-3.5.

233. 38:33

     So that's, like, one side of it. The other side of it is, um, how do you actually kinda monitor and measure that? And so, again, LangSmith actually does have very good support for tracing and observability, and we do have timings all.

234. 38:47

     In fact, I can go ahead and show you very quickly if you wanna see. Um, so this is my LangSmith dashboard. I'll zoom in a little bit. These are my experiments.

235. 38:56

     Um, now, if I, if I zoom in here, I can open up one. Um, the Wi-Fi's a little bit slow. These are my replicates. I can open up one of my traces, and what I can see here is over here, I get the timing.

236. 39:09

     So this is the timing of the entire graph, and I can go through my steps. So this is, like, the re- retrieval's really fast. That's good. You know, less than a second.

237. 39:17

     Okay, now here's what's interesting. My grading in this particular case is, like, four seconds, so that's, you know, not acceptable in a production setting most likely. Um, but again, this is just, like, a, this is a test case.

238. 39:26

     In fact, I'm using... What am I using to grade here? Uh, f- GPT-4.0. So it is, um, you know, there's ways you could speed this up by using different models or different prompts or grading everything in bulk, so there's a lot of ideas.

239. 39:39

     I actually grade each document independently. Oh, actually, you know what? This is using Chat Fireworks. Uh, so anyway, but you can look at your timings in lengths, but that's a really nice way I like to do it to kind of see, um, to kind of monitor the timing of my applications.

240. 39:55

     Um, yep.

241. 39:58

     Uh, so, uh, the case you showed, you showed it, like, passing it from scratch, like, when the agent is running, uh, from the beginning and trying to accomplish the task.

242. 40:10

     But in reality, usually we will have some context or history that's being passed to the agent. Do you have any suggestion in terms of how to pass that through?

243. 40:21

     Because, say, they don't have to do a function call because that's already in the history versus, like, doing it-

244. 40:29

     Um, let me make sure I got the question. So I-- The question was with agents you typically pass them a message history. That is absolutely true. Um, and in fact, the ReAct agent that we implement here, like, I can even open up one of the traces.

245. 40:46

     We can look at it together. So, um,

246. 40:51

     so here's a ReAct agent with, uh, GPT-4.0.

247. 40:56

     Here's one of my traces. Let's open it up and actually see what's going on here. So what happens is, um, first my assistant is right here, so this is OpenAI, will...

248. 41:08

     You know, here's the system prompt, right? So you're a helpful assistant. You're answering questions. Um, here's the human question, right? So again, this is the start of our message history, okay?

249. 41:17

     So what the, what the... Uh, and also these are the tools that the LLM has, and this is pretty cool. We can see this is the one it called.

250. 41:26

     So what happened is our LLM, it looked at here's our system prompt, here's the human instruction, and it says, "Okay, I'm gonna retrieve documents." Great. So then it goes, and this is a tool call.

251. 41:37

     It goes and retrieves the documents. Now that goes... So you look here, we can actually open up the retriever. What do we get? Here's our documents. Cool. Now I go-- it goes back to the assistant, so back to...

252. 41:47

     This is a looping thing. It started with our assistant. It made the tool call. The tool ran. We got documents. Now we get them back. Now let's go back to our LLM.

253. 41:55

     So now our LLM, this is pretty cool, right? Here's the message history like you were saying. Instructions, question, document retrieval. The documents that are retrieved are right here.

254. 42:08

     And then now the LLM says, "Okay, I wanna grade them." It calls the grader tool, and this is its reasoning, and this is its grade. So anyway, you are right that as this goes through, you basically accumulate a message history, and the LLM will use the message history and the most recent instruction to reason about what tool

255. 42:26

     to call next. That's exactly how it works. Um, I think I answered the question. Is, is there anything that isn't clear about that? So it is true. Like, let's look at an example, right?

256. 42:38

     So in this particular case, right- The LLM sees the retrieved documents from the tool, and then it makes decisions as, "Okay, I have this tool response, it's retrieved documents.

257. 42:49

     What should I do next?" And it says, "Okay, well, why don't I go ahead and grade them?" And it calls the grade tool. So it looks at the message history and reasons about what tool it'll call next.

258. 42:56

     It's exactly how these ReAct-style agents work. The, and the whole issue is that's kind of a noisy process. Like, it can look at that whole trajectory, it can get confused, it can call the wrong tool, then it's on the wrong track, and that's exactly why these more open-ended tool-calling agents fail.

259. 43:11

     Um-

260. 43:12

     Let's say like in the second follow-up question, uh, to the agent-

261. 43:16

     Yep

262. 43:17

     ... that, that technically should follow similar pattern to tool calling, right? Because sometimes the data retrieved from the first,

263. 43:29

     um, tool, from the first batch of tool calling. Is there a different way to, like, kind of structure your, um, tasking so that you can get the same, uh, value as before, but multiple iterations of question, not just like a single question and then get an answer?

264. 43:47

     Okay. Right. So I think the, the question is, well, let's say there's a multi-turn conversation where I can-- the user can go ahead and ask a second question, of course, and that whole kind of message history will be, will be propagated.

265. 43:59

     Um, yes, that is, that is a common pattern, um, and that... Let's see. I mean, what's the question on that, though? Like, it,

266. 44:14

     it could use context from its initial trajectory to answer the second question, for sure. Um, it'll probably look at that jointly when it's deciding what tool to call. So for example, if it receives a question and in its message history it sees the context needed to answer the question, the agent could probably decide, "Okay, I don't need

267. 44:33

     to retrieve documents again. I have the documents needed to answer the question. I'll answer that question directly." So it is true that in a multi-turn conversation, the agent can look at its message history to inform what to do next.

268. 44:44

     That's definitely true. Here, I don't consider evaluation of multi-turn conversations, um, but that is-- it's a good topic, actually. Um, I don't quite have a tutorial for that yet, but

269. 44:59

     I could think about putting that together. Yeah, that's a good point. Uh, I'll make a note of that, actually.

270. 45:08

     Yeah. So multi-turn is a good one. Cool.

271. 45:14

     Um, okay. Yep. Yep.

272. 45:17

     Uh, so, uh, test units are cool because they reduce the hallucinations. You can actually run within the app and so on and so forth. But have you experimented with this?

273. 45:26

     Do you have a model like C4 though in writing tests, and then you're gonna do that for an open-ended large language model, uh, tasks?

274. 45:33

     Um, so the question is, I believe, um,

275. 45:40

     have I tested, like, the ability to kind of online auto-generate unit tests?

276. 45:48

     Yeah.

277. 45:49

     Yeah. Okay. So that's a big topic. So basically, the AlphaCodium paper, um,

278. 45:56

     that Karpathy references here does that. I have not tested that because it does ramp up the complexity because then you're relying on an, an... And I don't think-- I mean, that would be aggressive for a production setting because basically you'd be relying on an LLM to auto-generate you unit tests, testing against things that are auto-generated.

279. 46:18

     There's a lot of opportunity for error there. I think it's interesting, particularly in terms of, like, offline challenges like this, but in terms of, like, a production application, that feels pretty hard and risky, but it's an interesting theme.

280. 46:30

     The thing I've tested more on and I found to be very effective is super simple, crisp, lightweight, free, effectively unit tests. Like, again, the good use case I found was our, um...

281. 46:43

     So LangChain, we have an internal a- RAG application called ChatLangChain, and it indexes our documents and provides QA. It occasionally hallucinates the imports, and that's a really bad experience, right?

282. 46:56

     If you take a code block, copy it from this app, and then it, this import doesn't exist, it's like, "What the hell?" You know? That's really annoying. So I incorporate a really simple check where I have a unit test that just, it does a function call where it extracts the imports from the code block from the answer,

283. 47:12

     and it tests the imports in isolation. If they don't exist, there's an error. I feed it back to LLM and say, "Look, this isn't a real import. Try again."

284. 47:19

     And I can do-- you can do other tricks like then context stuff, relevant docum- documents or something like that. Anyway, you can handle that differently. But that, just that little alpha significantly improved our performance.

285. 47:31

     So I kind of like simple, lightweight, free unit tests. Um, the idea of online generated unit tests is interesting, but, like, opens up a lot more surface area for errors.

286. 47:42

     The idea is that, you know, if you think about how unit tests look like, they oftentimes can be easier to write than the code and, uh, you got a better structure.

287. 47:51

     So, um, we use that for, like, not reviewing the outputs of some things, but have the unit tests done and then reducing the hallucination. And when we write the code further and write the code before we actually implement it, we get to a better implementation.

288. 48:07

     Yeah. Um, so I, the follow-up there was, um,

289. 48:12

     try, let's see. Trying to write the test before the app is implemented. Um, I see. Yes. So they also do that. So basically, for each question, they do have, they-- So the AlphaCodium work references, uh, existing unit tests for a given question as well as auto-generates.

290. 48:32

     So you are right. That'd be interesting to have a bunch of pre-generated unit tests that you know are good for certain questions and to run them. Absolutely. Hard to do in a production setting with an open-ended input, but potentially very useful in...

291. 48:47

     Well, okay, even in a production setting, you could maybe have some battery of unit tests and based upon the question type, pull related unit tests that you know are going to be relevant is a good idea.

292. 48:59

     Yeah, it's a good idea for sure. So basically using some battery of dynamically chosen pre-existing unit tests based upon the question type or the documentation, whatever documentation they're asking about, that's a good idea.

293. 49:12

     And for like larger projects, if you want to test for regressions, it's still very difficult to do otherwise.

294. 49:19

     Yeah. He was saying, so for larger projects, you can also use that to test for regressions. Yes, that is, that's definitely a good idea, and this paper does incorporate that idea as well as this auto-generated unit test thing, which is a little bit more aggressive.

295. 49:34

     Cool. Yep.

296. 49:36

     So how, like, some of the self-RAG, for instance, um, like LLM checking, LLM results and things like that. There are some, some frameworks that try to do this with different agents, agents helping with that, right?

297. 49:52

     Uh, Ora is one. Um, Microsoft have one too, but it's added complexity-

298. 49:58

     Yeah

299. 49:59

     ... added latency. I was wondering if, um, you've seen those

300. 50:06

     frameworks give, uh, more, um, better, better results than would justify the latency and complexity, or whether in the use cases you've seen kind of this one checker, one reflection node is, uh-

301. 50:22

     Yep. So the question was, in some of these self-reflective applications, like let's say this one, the self-reflective RAG, right? We're doing a few different checks here. We're checking documents, we're checking hallucinations, we're checking answer quality.

302. 50:35

     Instead of having some hard-coded single prompt, uh, to do that, can you have another agent, like kind of a, a checker or a grader agent that's a bit more sophisticated?

303. 50:46

     Um, I have-- and he mentioned a few frameworks. I have not played with them. I think it's interesting. I think it's, it's one of these things where it's really good for kind of academic papers.

304. 50:56

     It's very interesting. It's really good for-- It could be good for offline testing. In an online or production setting or, you know, it's-- the latency and complexity is probably a bit much.

305. 51:07

     I think in, in a production setting, it goes back to what I think this guy was referencing. You probably would want something that's extremely fast, lightweight, um, and I would not think about like a multi-agent system in a production setting doing any kind of grading.

306. 51:22

     This whole idea of LLM as LLM graders is a kind of a new idea. So I think this idea of like more complex agen-agent graders is interesting, but we're kind of in the-- we're taking baby steps at this point, especially in a production setting.

307. 51:37

     So I'd probably shy away from that for now, particularly if you're thinking about production. But for something offline or experimentation, it's probably interesting.

308. 51:46

     Yeah.

309. 51:47

     What's the best practice that you've seen so far for performing tests or evaluations on those cycles? Because obviously you can end up with many, many different, uh, sort of types of input and that sort of thing.

310. 52:01

     And obviously, LangGraph like sugge- like provides some tracing there. But what about, hey, here's an interesting test case. What are kind of best practices you've seen for actually performing those routine evaluations?

311. 52:12

     Yeah, exactly. So actually, the notebooks shared here go into it a little bit. So the way I like to do it is, um, and I can even-- Why don't I just show you the one of the notebooks?

312. 52:22

     So basically, um, so at the bottom I have kind of all the different evaluations. So this goes back to a question that someone mentioned down here as well.

313. 52:34

     Sorry, which one is this?

314. 52:35

     So this is, uh, so there's a Colab, and then there's a notebook, and they are both the same. So they have the same, uh, evaluation section. Uh, this is, this is rag-agent-testing.ipynb.

315. 52:50

     Uh, and there's also a Colab, which is, uh, it's the same notebook basically. It's, it's in the slides. Just make sure you got those links. But, but to your question, so the way I did it, and you can be very flexible with this, is basically I define the expected trajectory through the nodes in my graph.

316. 53:08

     And in this particular simple case, the trajectories that I expect are basically retrieval, grading, web search, generate, or retrieval, grade, generate. Those are the two expected trajectories that I want it to take.

317. 53:20

     Now, in this case, I don't do cycles. If you did cycles, you could just incorporate more steps here to say, "Okay, here's kind of the expected number of steps I want or expect to see through that cycle."

318. 53:31

     And I think someone mentioned here, if you have a really open-ended challenging task, then it may be hard to enumerate exactly. But for a lot of like more production setting applications where these are not extremely long-running, you can enumerate, here's the steps I expect to take through the cycle.

319. 53:45

     And, and the way I do the evaluator, it's as simple as this. There's not much code. It's all in the notebooks. But basically, I compare the tool calls that the thing did to these trajectories.

320. 53:55

     That's it. Super simple. So that's how I would think about it. I keep it really simple. I would basically try to enumerate, here's the steps through the difference-- the cycle I want it to take.

321. 54:06

     And yeah, go ahead.

322. 54:08

     Just to make sure I understand.

323. 54:09

     Right.

324. 54:09

     Basically, you're using, so in the LangGraph concept-

325. 54:13

     Yep

326. 54:14

     ... basically using the nodes and saying, "Hey, I expect it to be calling these tools along the way."

327. 54:18

     That's it.

328. 54:19

     And that's how I'm performing my evaluation.

329. 54:20

     That's it. That's it. So it, it's actually really simple. In, in-- This is actually a good point. In the LangGraph case, for this custom agent you'll see all I do is for every node, and we could get into the code if we want, maybe people have already explored.

330. 54:38

     So, well, I'll just answer the question directly, then we can back up and go through all the code if we want. But basically-

331. 54:46

     Each node in my graph, I just append this step name, right? So like retrieve, I just say retrieve documents. This is my generate node. I just, you know, append this thing, generate answer, and I return that in my state as steps.

332. 55:01

     So then I have this record of the steps I took in my graph. That's it. And I can go ahead and fetch that at eval time and compare it to what I expect.

333. 55:09

     That's all you're doing. So that's how it would work with, like, the custom LangGraph thing. And now with the ReACT agent, actually, it's a little easier because the ReACT agent uses a message history.

334. 55:20

     So I can just go to my message history, and that's exactly what I show here. You can strip out, um, I, I guess I do it up above. But basically, I have this function in the notebook.

335. 55:29

     Um, let's see. Where is it? Yeah. It's find tool calls ReACT. So it's this little function that will, um, basically look at my message history and strip out all the tool calls.

336. 55:42

     So, um, yeah, it's a nice little, nice little thing. With a ReACT agent, it's really easy to get the tool calls out. With LangGraph, I just log them at every node, and then at, at eval time, I just can extract them.

337. 56:01

     Yep.

338. 56:02

     So can you elaborate on the multi-turn scenario with this context switching?

339. 56:07

     Where does it actually, the cognitive ability that say, "I don't have the answer" in a multi... It's called retrieve. Where is that cognitive ability? Is it in the, in the query router point in the graph?

340. 56:25

     Yeah. So the question was, with an agent in a multi-turn conversation setting, how does it know whether or not it has the answer to a, to a given question?

341. 56:37

     How does it know where to go?

342. 56:39

     Where to go, exactly.

343. 56:41

     It, it may be multiple places.

344. 56:42

     Yeah. That's right. So there's a couple different ways to kinda break this down. So with these agents,

345. 56:51

     there's a few levels of instruction you give it. First, you give it an overall agent prompt. So if I look at the notebook here, uh, we can go look at the ReACT agent as an example of this.

346. 57:03

     So the ReACT agent is defined, uh, right here. This is kinda like the planning step. Here's my, like, naive prompt, okay? So you're a helpful assistant that's tasked with answering tasks.

347. 57:15

     Use the provided vector store to retrieve documents, grade them, and go on. Now, let's take the case that's more complicated. Let's say I had two vector stores. So one thing I can do is I can explicitly put in the agent prompt, you have two vector stores, A, B.

348. 57:30

     A has this, B has this. And then you're implicitly having the LLM, uh, giving it the ability to kinda reason out which one to use. Now, this is where the second piece comes in.

349. 57:42

     You have to als- you also bind it to a set of tools. This is really where the decision-making comes in. When you create this tool... So here's retrieved documents, right?

350. 57:51

     This tool description is captured by the agent, so the agent knows what's in this tool. And this is really where that decision to use this retriever tool versus another one would be made.

351. 58:03

     It'd be a combination of the prompt you give to the agent and/or the tool description. So if you had two vector stores, you could basically say, like, retrieve documents one.

352. 58:12

     This vector store contains information about X, another one contains information about Y. Then the agent is deciding what tool to call based on that description and maybe based on its overall prompt.

353. 58:24

     But to your point, it's not easy. So actually, with custom agent, this is with the ReACT style agent. With the LangGraph custom agent, you can do it a little bit differently, where I actually don't have it in this notebook, but I have other cases where you c- actually can build a router node.

354. 58:41

     And I mean, I'll show you, actually. So this particular notebook, um, so this one, this self-RAG, it's in the slides. If you open this up, we did this with, uh, LlamaFolks.

355. 58:53

     So this is actually a trick I really like. If you go to, um, if you go to LangGraph RAG Agent Local here, um...

356. 59:03

     Let's see. This Wi-Fi's a little slow. What I define here is a very specific router at the start of my agent that decides where to send the query. And this is something I really like to do because, like we saw with the ReACT thing, it has to kinda decide the right tool to use, which can be kind

357. 59:20

     of noisy, versus right here. So here's my router, right?

358. 59:29

     This is reliable enough to run locally. And what I do here is I run this at the start of my graph, and I have the agent, or, and yeah, the agent explicitly take the question and decide what to use.

359. 59:41

     And based on what decision it makes, I can send it to either web search, or in this case, a vector store. So to answer your question, if I pull all the way back, I personally like to do explicit routing as a node at the start of my graph because it's, it's pretty clean.

360. 59:56

     And you can see in the flow of this overall graph, this router runs first, and it looks at my question, and it sends it to one of two places, and this can be more complex.

361. 1:00:04

     You can send it to one of N places. But I just do one of two here. This is with, like, a custom LangGraph agent. This is what I like to do.

362. 1:00:10

     If you're using a ReACT agent, it is then using a combination of the, the tool definitions and the overall agent prompt to decide what tool to call. But you can see it's more squishy because it has to call the right tool, and as opposed to giving it a router and saying, "Always run this router first," it has

363. 1:00:30

     to kind of make the right decision as to what tool to call based on the context question, which is harder. And that gets back to our h- get, that gets back to our overall story about these kind of LangGraph explicitly defined agents are more reliable because you can, like, lay out this routing step right up front and

364. 1:00:45

     have it always execute that step before going forward.

365. 1:00:56

     Here's like, hey, I actually have the answer based on previous... Uh, so this seems to be very deterministic. Everything's in a route and go receive document. But I actually in the graph would say, "Do I have a history of the answer?

366. 1:01:12

     Can I answer?" That's the thing that we just can't do is that you have not only look at the tree, but you also have to look at the cognitive history.

367. 1:01:20

     Can I answer the question? I'm just curious-

368. 1:01:22

     Yep

369. 1:01:22

     ... how do you orchestrate that flow?

370. 1:01:24

     Okay, got it. So the question was, how do I incorporate the idea of routing with history? So here's what you would do. It's actually, you know, kind of... It, it should be pretty straightforward.

371. 1:01:37

     You can define this router node in your graph, and that router node... Oh, I should go down to it. So basically, here's my graph, and here's the, the, uh, route question node, right?

372. 1:01:50

     Um, yeah, actually, in this particular case, it's, it's an edge. Uh, don't worry about those details. Basically, what you could do is you could have a node that takes in the state.

373. 1:02:04

     Now, that state could include your history. So what you could do is in that router prompt, you could really easily, um, here include another placeholder variable for, like, your, your message history or something.

374. 1:02:21

     And you-- what you could say then is make a decision about where to go next based upon the question and based upon something in our history. And so you actually would plumb in your message history here and use it to jointly decide what to do next.

375. 1:02:34

     So that's actually really easily handled in LangGraph using a node, and you can reference the history that can be passed-- you can pass into that node as state.

376. 1:02:46

     Uh, cool. Yep. [background chatter] The notion of state. Yeah, let's, let's talk about state in a little bit more detail. So let's actually go to the notebooks that we're working with here that I've shared.

377. 1:03:03

     Um, so here's this RAG Agent testing notebook. So if you go down to the custom LangGraph agent,

378. 1:03:14

     the way you do it is, um... Let me find the state. Yeah, so here is what I call graph state. So the graph state is basically something that lives across the lifetime of my graph.

379. 1:03:28

     I typically like to do something as simple as just a dictionary. So basically, this is a RAG graph. And here I'm basically gonna define a number of attributes in my state that are relevant to what I wanna do with RAG.

380. 1:03:42

     A question, my answer generated, whether or not to run search, some documents, my step list. And basically, the idea here is that I define my state up front

381. 1:03:53

     and then at every node I br- I basically accept state as the input, and then I operate on it in some way and write back out to state. So basically what's happening is I define state generally up front as like a dictionary or something like that.

382. 1:04:09

     It's a placeholder for things I wanna modify throughout my graph,

383. 1:04:13

     throughout my agent. And every node just takes in state, does something, and writes back out to state. That's really it.

384. 1:04:22

     So basically, it's a way... You can think of it as a really simple mechanism to persist information across the lifetime of my agent. And for this RAG agent, it's things that are really intuitive for RAG.

385. 1:04:33

     It's like question, it's documents. And so let's take an example. Like, um, okay, here's a fun one. So my grade documents node, right? What I'm doing here is I'm taking in state, and from my state, it's just a dictionary, so I'm extracting my question, I'm extracting my documents, um, and I'm appending my-- I'm, I'm appending a new

386. 1:04:52

     step 'cause I'm notifying, "Hey, here's my new step." And basically I'm doing some operation. I'm iterating through my documents. I'm grading each one. Uh, if the grade's yes, I, I, I keep it.

387. 1:05:04

     If, um, if the grade is no... So yes/no means like, is it relevant or not, basically. So if yes, it's relevant, I keep it, I put it in the filtered docs list.

388. 1:05:14

     If it's not, I set the search flag to yes, which means I'm gonna run web search because I wanna supplement, I have some docs that are irrelevant, and I write back to state at the end.

389. 1:05:23

     My filtered docs, my question, search, and steps. That's it. So state's a really convenient way to just pass information across my agent.

390. 1:05:34

     And I like using a dictionary, just like nice and clean to manage, uh, as opposed to a message history, which is like a little more confusing. Like in any node, if you use a message history, it's like a stack of messages.

391. 1:05:45

     So if you want the question, you have to like... It's like usually the first message, you need to just index it. Just kind of ugly. I like using a dict, where it's just like I just getting the question out as a, as a key in my dictionary.

392. 1:05:57

     Um, cool. Let's see. We're about, uh, an hour in. I can also, um, you know, let people just hack and walk around, talk to people,

393. 1:06:12

     stay up here, whatever's best. Uh, and you can keep asking questions if you want to. I think people are just working, doing their own thing now anyway. Um,

394. 1:06:22

     so I might ask one question just for fun. Is anyone interested in local agents? We didn't talk about that too much. Uh, it's a big theme. Yeah. So I shared a notebook for that.

395. 1:06:33

     Um, and by default I am using Llama 3.

396. 1:06:39

     Uh, you can absolutely test other models. Um, so this is set up to test just Llama 3 with Ollama. Um, try other things. I have a M2 32 gig, so 8- 8B runs fine for me.

397. 1:06:56

     If you something bigger, um, you could actually bump that up a little bit. So that can be a nice idea.

398. 1:07:04

     I used 70B on my M1 and I don't recommend it.

399. 1:07:07

     Yeah, exactly. 70B is, uh, it's unfortunate that... Yeah, I've, um, I actually want a bigger machine so I can run that. 'Cause it-- I found for tool calling,

400. 1:07:20

     8B is really at the edge of, of reliability. Um,

401. 1:07:26

     so that's actually why you really can't run the ReAct agent locally with an eight billion model reliably. You can run the LangGraph agent very reliably 'cause it doesn't actually need tool calling.

402. 1:07:37

     It only needs structured outputs. You'll see in the notebook. So that's actually a really nice thing. Um,

403. 1:07:44

     but ReAct agent won't, won't run locally, reliably at least.

404. 1:07:50

     Yep.

405. 1:07:51

     Uh, regarding RAG, what is, is there a typical way to

406. 1:07:57

     chunk documents?

407. 1:07:57

     Did you say chunk documents?

408. 1:07:59

     Yeah.

409. 1:08:00

     Okay. Yeah. So this, this question always comes up. With RAG, what is a typical chunk size? Yeah, if you ask 10 people, you'll get 10 answers. It's, uh, notoriously ad hoc.

410. 1:08:13

     You know what? To be honest, I did, um,

411. 1:08:16

     I did kind of a talk on the future of RAG with long context models. I'm kind of a fan of, um, trying to keep chunk size as large as possible.

412. 1:08:25

     Actually, I think-- This is a whole tangent, but I actually think one of the nicest tricks, uh, let me see if I have a good visual for it. Um,

413. 1:08:36

     basically... Uh, let me try to find something here.

414. 1:08:41

     Um, this is a whole separate talk. But basically, um, I think, uh... Yeah, this one, RAG and long context. So this is a whole different thing. But, um,

415. 1:08:56

     yeah, this idea. So I think for RAG, one of the ideas I like the most is

416. 1:09:04

     decoupling-- I'll explain this. I'll just say it and then I'll explain it. Decoupling what you actually index for retrieval from what you pass to the LLM, 'cause you have this weird tension, right?

417. 1:09:14

     Smaller semantic-related chunks are good for retrieval relative to a question, right? But LLMs can process huge amounts of context at this point, you know, up to, say, a million tokens.

418. 1:09:26

     So historically, what we would do is you would chunk really small, like very, very-- try to get it as tight as possible, all these tricks, semantic chunking, a lot of things to really compress down to just compress and group semantic-related chunks of contexts, right?

419. 1:09:42

     But the problem is then you're passing to the LLM very narrow chunks of information, which has problems in recall. So basically, it's more likely that you'll miss context necessary.

420. 1:09:53

     So it's good for retrieval, but bad for answer generation. So a nice trick is for retrieval, use some chunking strategy, whatever you want, like make it, you know, small.

421. 1:10:05

     But you can actually use like a doc store to store the full document. And what you can do is retrieve based on small chunks, but then reference the full document and pass the full document to the LLM for actual generation time.

422. 1:10:17

     So you decouple the problem of, like, are you sure you're passing sufficient context to the LLM itself? Now, I also understand if you have massive documents, it can be wasteful in terms of tokens to pass full documents through.

423. 1:10:29

     But, um, there's some Pareto optimum here where I think

424. 1:10:34

     in that being too strict with your indexing approach doesn't make sense anymore given that you can process very large context in your LLM. So you wanna avoid being overly restrictive with your chunk size.

425. 1:10:46

     So this is a nice way to get around it. Basically, to, to summarize, you can choose a chunking strategy, whatever one you want. But I like this idea of referencing full documents and then basically passing full documents to the LLM for the answer itself.

426. 1:10:59

     It gets around the problem of an overly aggressive chunking strategy that misses context needed to actually answer your question. And I think with long context LLMs getting cheaper and cheaper, this is starting to make more sense.

427. 1:11:11

     Um, I'm trying to think. I had a whole kinda like slide on, uh... Yeah, th-this one is kinda like balancing system complexity and latency. So it's kinda like on the left is like maybe the historical view.

428. 1:11:22

     You need the exact relevant chunk. You can get really complex chunking schemes, a lot of over-engineering, lower recall. Like if you're passing 100 token chunks to your LLM for the final answer, you might miss the exact n- you know, part of the document that's necessary.

429. 1:11:37

     Um, very sensitive chunk size, K, all these weird parameters, right? On the other extreme, just throw in everything, throw everything into context. Google actually I think this week will probably announce some interesting stuff with context caching.

430. 1:11:51

     Seems really cool. Maybe that actually could be a really good option for this. But higher latency, higher token usage, can't audit retrieval, security and authentication, like if you're passing 10 million tokens of context in for your answer generation.

431. 1:12:05

     So something in the middle is what I'm advocating for, and I think, uh, this kind of document-level, uh, decoupling, where basically you index and reference full documents and pass full documents through your LLM is like a nice trick.

432. 1:12:22

     Um, we've seen a lot of people use this pretty effectively. So... Yeah.

433. 1:12:27

     Yeah, kind of, uh, I think backing off of that, um, is there any way or any guarantees or tricks that you have to make sure that you're not, um, actually missing any of the relevance of the context throughout the entire window?

434. 1:12:38

     Like how do you measure that you're actually using all of the context as opposed to just beginning or end or anything like that?

435. 1:12:45

     Okay, that's interesting. So the question was, how can you evaluate the amount of context you're using? Okay.

436. 1:12:56

     Yeah, like does it definitely use the whole state? I don't wanna miss any-

437. 1:13:01

     Uh, okay. But I guess, so in a RAG context, you have a question, you have an answer. So you have a question, you have an answer, and you have some retrieved documents.

438. 1:13:10

     So you can evaluate- The question relative to your document, that's one way to get at this. Like, how much of the document is relevant to your question? So that's maybe one approach.

439. 1:13:24

     And actually, the notebooks show a few prompts to kinda get at that. Um, and actually, a good way to think about that is you can, you can think about document precision and document recall, and this is a little confusing maybe, so I should explain it.

440. 1:13:39

     Basically, document recall is, does a document contain the answer to my question anywhere? So let's say you have a 100-page document. On page 55 is my answer. Recall's one.

441. 1:13:50

     It's in there. Precision's the other f- side of that coin, which is, does it contain any information not relevant to my question? In that particular case, huge amount of in- irrelevant information, so recall would be one, precision would be very low.

442. 1:14:01

     So that's one thing you can do. You can actually look at your retrieved docs, measure precision and recall. That's, like, one thing I think I would probably like to do there, and that's probably the best way to get at this question of, like, how much of my documents am I actually using?

443. 1:14:14

     Now, with this approach, your recall will be high, your precision will be kind of low, and you would say, "I don't care. It's fine. If I have a model, it's super cheap to use large number of tokens, I'm okay."

444. 1:14:27

     Maybe I'll frame it another way. I care more about recall than precision. I wanna make sure I always-- I answer the question. If I pass a little bit more context than necessary, I'm okay with that.

445. 1:14:36

     Versus if you're a precision-gated system, then you would say, "Okay, I'm gonna miss the answer sometimes, but I'm okay with that 'cause I never want to pass more context than necessary."

446. 1:14:45

     I think a lot of people are moving towards higher recall because these LLMs are getting cheaper, and they can process more context. So that's kind of how I might think about it, and I think this approach is a nice idea of, you know, indexing, indexing full documents.

447. 1:15:00

     Or, or indexing chunks, but then referencing full documents, passing full documents to your LLM to actually generate answers.

448. 1:15:07

     So, yep.

449. 1:15:09

     I was wondering, it seems like there's a couple options for where exactly you inject the context into the conversation history. Like, just for one example, you could stick something into the system prompt and say, "Hey, this is your knowledge that you know."

450. 1:15:25

     And then the user just asks a question, and there's no prompting from the user. The other thing is you could completely modify what the u- what the quote unquote user is, and then say, "Answer this question with this context," and it goes in there.

451. 1:15:40

     And then, like, a, a further idea on top of that is you ask a question, then you ask a follow-up question. Maybe if you say, "Please elaborate," there's no need to actually retrieve anything, but you're just having a chat, a, a complete chat where you ask something else that needs retrieval.

452. 1:15:59

     Do you have any, like, observations as far as things that work well or haven't worked well for continually appending things into the system prompt or, like, adding another user?

453. 1:16:09

     Like, where do you stick that context? How much of it do you keep? Like history-

454. 1:16:14

     Yep. Yeah, okay. So this is a great question. The question was related to kind of like agentic RAG and where to actually put documents. So let's walk through the cases.

455. 1:16:28

     So case one is I have a fixed context for every question user's gonna ask. Let's say I have a, like a RAG bot against, like, one particular document, and that document is always gonna be referenced.

456. 1:16:42

     So that-- You make a very good point. In that case, what I would do is, let's go back to our, like, agent example. You can put that in the system prompt itself, like you said.

457. 1:16:52

     So, and actually, I'll, I'll mention something else about this. It'd be like, uh, right here. So here's your system prompt for your agent. You just plumb that whole document in, and you say, "Every question, you're going to reference this document."

458. 1:17:05

     You don't need retriever system. You're done. That's a very nice case. No retrieval complexity. You just context stuff your whole document.

459. 1:17:13

     The thing that Google is going to announce this week, I believe, this context caching, seems really interesting for this 'cause basically, what they're saying is, if I get it right, I think Logan will speak to this, but...

460. 1:17:26

     And I think they have a context win a, a million to ten million tokens. It's huge, right? So basically, you can take a large set of documents and stuff them into this model effectively, and they house them for you somehow.

461. 1:17:36

     I think you have some minor data storage fee. But then for every inference call, they don't charge you for all those tokens that are cached, which is pretty nice.

462. 1:17:47

     So basically, here's the use case to your point exactly. I have some set of documentation. It's ten million tokens. That's a lot of pages, hundreds of pages. I have it cached with the model, and every time I user ask a question, I don't get charged, you know, ten millions of tokens to process the answer.

463. 1:18:02

     They just cache it for me. Really nice idea. So that's your point of, like, the first thing. That's, like, not quite in the system prompt. It's in, like, the cache, but that's the same idea.

464. 1:18:11

     So you have cached or system prompt fixed context. That's like case one. So case two is you have-- You want a dynamic retrieve, so you don't, you can't stuff your context.

465. 1:18:21

     Maybe you have a few different vector stores like we were talking about here with routing. In that case, yeah, you have to use a, you have to use an index of some sort, maybe a router, to choose which index to retrieve from.

466. 1:18:32

     So that's kinda case two. Um, yeah, I'm trying to remember, um, the... Oh, okay, yeah. So, like, in that particular case, for follow-up questions,

467. 1:18:44

     how do I kind of control whether or not I re-retrieve or not? So that's the nice thing about either one of these agents. It has some state. So the state lives across the lifetime of the agent.

468. 1:18:55

     So basically, the agent, and this actually gets exactly to what this, the, the other question was on.

469. 1:19:01

     Let's say I built my agent with, uh, I'll show you right here. So

470. 1:19:06

     let's say I have a router node at the start of my agent, okay? And that, that router has access to state. What I can do is then, given a question, this could be-- Let's say it's a multi-turn thing.

471. 1:19:17

     It's the second question in my, in my conversation. I have an appended state for the rest of my discussion here. Um, the agent knows it returned an answer. So, um, basically when a new question comes in, you could pass like the entire state back to that router and the router could know, okay, here's the docs I've already

472. 1:19:35

     retrieved and it can basically then decide to answer directly because I already have the answer to the question. So that's a long way of saying you can use state, either message history or explicitly defined in, in your LangGraph agent to preserve docs that you've retrieved already and then to just use them to answer a question without re-retrieving.

473. 1:19:56

     So that's kinda what these graph-- these, these RAG agents can be really good at. That was kind of like storing that in short-term memory and reasoning about, hey, do I need to retrieve or not?

474. 1:20:07

     So that's exactly the intuition behind why these RAG agents can be pretty nice. Yeah.

475. 1:20:12

     Yeah. Just a follow-up to that. Um, it seems like, you know, there's this idea like RAG is a hack, like, right, like we just kind of took some model that's designed to do chat and then we found some stuff.

476. 1:20:25

     Yeah.

477. 1:20:25

     It, it seems like there's a little bit of a back and forth going on where, um, the model is designed in this general purpose way- [clears throat] ... where coders hack something together and now, you know, you go back to people training the models and they say, "Oh, well we can actually fine-tune the model to specifically be good at

478. 1:20:44

     this." And it-- I'm, I'm curious, it almost seems like you're, you're gonna like propose all these ideas and then we're just gonna confirm whatever the people fine-tuning those models are doing because that's what's gonna work, right?

479. 1:21:00

     Like-

480. 1:21:00

     Yeah

481. 1:21:00

     ... do we see it-- like do you see the industry moving in a direction of like stuffing things into the user-

482. 1:21:06

     Yeah

483. 1:21:06

     ... context versus the system prompt and, and where are like the indications of which one's just gonna work better simply because that's what it ends up requiring?

484. 1:21:15

     Yeah. Yeah, exactly. Okay. This is a really good-- This is a really good discussion and whole debate. So the highest level framing of this is how do you want to, um, how do you want your model to learn?

485. 1:21:29

     So one option is you can modify the weights of the model itself with something like fine-tuning. Another is you can use what we call like in-context learning through your prompt.

486. 1:21:37

     So RAG is like a form of in-context learning. I'm basically giving it some documents, it's reasoning about those documents, producing answers, not touching the weights of my model. Fine-tune would be taking the knowledge I want to run RAG on, fine-tuning your model and updating the weights so that it has that knowledge.

487. 1:21:52

     There's a lot of debates on this and the w- and actually I think Hamel has a whole course on fine-tuning. Oh yeah, do you have a f-

488. 1:22:00

     I, I just wanna clarify. I don't mean fine-tuning information in. I just mean-

489. 1:22:04

     Okay

490. 1:22:05

     ... fine-tuning so that you have the idea like if you-- you could fine-tune it so that it knows, oh, I should always retrieve this information from my system prompt and use that when answering questions.

491. 1:22:17

     Or you could-

492. 1:22:18

     Okay

493. 1:22:19

     ... fine-tune the model to really use whatever extra context the, the user is providing. So, so just as a simple example of that, if you have the system prompt at the very beginning of the chat, you would be training the model to focus more on the information at the beginning of the chat.

494. 1:22:35

     Whereas if you were fine-tuning the model to be good at using the context from the u-- that the user provides, then it's focusing more of its attention on that end of the chat.

495. 1:22:46

     So it's not-

496. 1:22:46

     Yeah

497. 1:22:47

     ... putting the information in, it's just how do I use the information-

498. 1:22:51

     Yep

499. 1:22:51

     ... to hack some cool-

500. 1:22:52

     I got it. Okay. I'll, I'll repeat that. So the clarification was thinking about using fine-tuning more to kind of to, uh, govern the behavior of the agent rather than to kind of encode facts, which is a good-- it's a very good clarification 'cause I was gonna say using fine-tuning to encode facts, I think a lot of kind

501. 1:23:13

     of literature has pointed to that being a bad idea for a lot of reasons. It's costly. You have to continually fine-tune as facts change. So let's dispatch that. I think that's a kind of a not a great idea.

502. 1:23:23

     But you make a very interesting point about fine-tuning to govern behavior. Now, there's a paper called RAFT that came out kind of recently and actually as far as my understanding is, I haven't played with it live, it's fine-tuning to kind of do what our notebooks show today of this kinda like

503. 1:23:41

     look at the documents that are retrieved, reason if they're relevant and then don't use them if they're automatically filter. It's doing exactly what we're doing in this LangGraph thing, but it's kind of achieving that same outcome through a process of fine-tuning.

504. 1:23:53

     So that's a very good insight. You're right. It seems promising to have these like kind of fine-tuned RAG agents so to speak or, or it wouldn't be an agent, it'd be an LLM fine-tuned for RAG that incorporates this kind of logical reasoning or what you're saying like maybe some kind of reasoning about if you have a multi-turn

505. 1:24:12

     conversation like avoid recency bias, whatever it is. That seems like a very good trend and interesting. Um, the challenge is a little bit if you're fine-tuning yourself, fine-tuning is hard and somewhat advanced and all that.

506. 1:24:28

     Alternatively, if it's like a very niche use case like, like the RAFT system would basically it fine-tunes to kind of do this as models get changed or update all the time, you kinda need to keep your fine-tune model up to date if you see what I'm saying.

507. 1:24:42

     So I think I'm still a little bit queasy about using fine-tuning even in that context because of the challenge of keeping it kind of up to date with the state of the art.

508. 1:24:51

     But it's interesting. I think the RAFT paper is a good reference in this direction and does exactly what we do in this workshop, but I believe it fine-tunes this into the model or attempts to which is a very intuitive thing to think about.

509. 1:25:03

     Basically let the model reflect automatically on retrieved documents and like automatically filter them for you. Seems like it should be able to do that. Um, seems like a good idea so...

510. 1:25:14

     But my hesitation would still be like, well, if I wanna switch my models I need to like re-fine-tune. I wanna use Llama 3, I have to fine-tune Llama 3 on this task.

511. 1:25:21

     I can't use propri- maybe I can fine-tune, you know, GPT-4o might have fine-tuning now. I'm not even sure. I could fine-tune GB- you know, so again if you have fine-tune myself that's hard.

512. 1:25:31

     It still feels a little bit like I'd rather just set up a simple, like, orchestrated agent that does it rather than re-rely on fine-tuning is my sense. Yeah.

513. 1:25:40

     I, I guess I'm s- I'm not really imagining the person doing or implementing RAG doing that fine-tuning, 'cause I think it-

514. 1:25:48

     Yeah, the provider

515. 1:25:49

     ... an analogy, you know, like, like, uh, Tesla's data engine that Andrej Karpathy-

516. 1:25:55

     Right

517. 1:25:56

     ... years ago, you just keep adding into that long tail.

518. 1:25:59

     Right.

519. 1:25:59

     Right, like, like OpenAI is gonna have some kind of data engine.

520. 1:26:03

     Right.

521. 1:26:03

     They started with, you know, answer these questions usefully.

522. 1:26:08

     Yep.

523. 1:26:08

     But it's gonna start to become that data engine. There's sort of like a long tail to follow that analogy that's going to be answer this with this context or this with this tool.

524. 1:26:18

     Right.

525. 1:26:18

     And as, you know, we just tack things together-

526. 1:26:22

     Right

527. 1:26:22

     ... whoever's training the model with a separate entity, which is fine-tunes to that. But there, there's gotta be some kind of, like, back and forth.

528. 1:26:31

     Right.

529. 1:26:32

     Definitely there isn't quote-unquote, like, this is the right way to do it, but if that's the way that it's fine-tuned, then it has become the right way.

530. 1:26:39

     Yeah, okay, so that's a very good point. I think this is, this is also a very big debate. So OpenAI just did an acquisition this week on a, a retrieval company.

531. 1:26:47

     I forget the name. Roc- Roxett, I believe. So I think they are moving more in the direction of retrieval. I could absolutely see them offering, you know, an API that potentially does retrieval for you and incorporates some of these ideas for you.

532. 1:27:01

     So how much of this does get pushed behind APIs, and they take care of whatever's necessary behind the scenes for you, that could absolutely be the case. I would not be surprised if at all they move in that direction.

533. 1:27:12

     And I think there's always, you know, it's an interesting trade-off, like how much are you willing to kind of, um, you know, abstract behind an API versus not. I think there's always a lot of companies, developers that wanna kind of control everything themselves and build it themselves, have full transparency, and others that don't.

534. 1:27:27

     And so I, you know, it's an interesting question. But of course, for certain functionalities, multimodality, very few people are gonna stand that up themself. You are-- you kind of let that live behind an API.

535. 1:27:37

     So what do you allow to live behind an API or not? My only concern is, I think for some of these kind of things, they could be very domain specific.

536. 1:27:46

     Like what you consider relevant or not could be very relevant to you in your application. You kind of wanna be able to control that. So the only thing I can imagine,

537. 1:27:54

     it could be kind of hard to abstract that all behind an API, which I think is maybe why OpenAI hasn't done too much in retrieval yet. It's just a hard beast.

538. 1:28:01

     I know they've been trying for a while. Um, I don't know. It's a great debate though. Yeah, we, we can discuss more if, uh... Yeah, it's a good topic for sure.

539. 1:28:13

     Yep.

540. 1:28:14

     Uh, you mentioned about like long context windows elements, right?

541. 1:28:18

     Oh, yeah.

542. 1:28:18

     So there's a problem of like lost in the middle. For example, the precision, like if, uh, like for example, in the context, let's say 95% is relevant information, 5% is somewhere in the middle, and somehow LLM kind of makes a mistake at that point.

543. 1:28:32

     Like, do you have any directions over there?

544. 1:28:34

     Yeah. Yeah, so the question's on lost in the middle of long context. So I actually did a whole study on this with Greg Kamerand.

545. 1:28:41

     Yeah, it's a, a really interesting topic. Um, so the insight was basically that long context LLMs tend to have lower recall or like, you know, factual recall for things in the middle of the context, okay?

546. 1:28:56

     So that was one observation. At least that's what their paper reported. So I actually looked at this with Greg, and we did something a little bit even harder. We actually tested for multiple fact retrieval.

547. 1:29:09

     So we tested can you retrieve one, three, or ten different facts from the context, and this was using GPT-4, uh, GPT-4 Turbo, single turn. Um, and on the X-axis, you can see the fraction of, of the needles that it basically can get, and then on the Y is the number of needles.

548. 1:29:28

     So basically one needle, um, three needles, ten needles. Green versus red is basically just retrieving versus retrieving and reasoning. So it's like reasoning's a little bit harder than, than, than just retrieving.

549. 1:29:42

     Um, these needles were actually pizza ingredients. So basically the background was, you know, this was 120,000 tokens of Paul Graham essays and three secret pizza ingredients or however many, one, three, or ten, but I injected in that context.

550. 1:29:57

     And I basically asked the LLM, "What's the ingredients needed to build the secret pizza?" So it'd have to find them in there. And basically, as you ramp up the number of needles, go from one to ten, um, it gets worse.

551. 1:30:08

     So with ten, it's actually, retrieval itself was only like 60%.

552. 1:30:14

     So then I looked at, okay, well, where is it failing? And that's what I look at here in this heat map. So basically, this is telling you, like, how long the context is, so 1,000 tokens all the way up to 12-- uh, 120,000.

553. 1:30:26

     And then here's like the needle placement, so one to ten. So this red means you couldn't retrieve it, and what I found is actually it doesn't get them towards the start of the document.

554. 1:30:36

     So the retrieval gets worse if the needle's at the front. So it's like this. If r- I read a book, I ask you a question about the first chapter, you forgot 'cause like, "Oh, I read that a month ago," or something.

555. 1:30:44

     Same idea, and actually I put this on Twitter, and then someone said, "Oh yeah, it's probably recency bias." And that's a good point that basically the most informative tokens in next token prediction are often the more recent ones.

556. 1:30:55

     So, you know, basically these LLMs learn a bias to attend to recent tokens, and that's not good for RAG. So that is all to say I'm a little wary about long context retrieval.

557. 1:31:08

     I wouldn't quite trust basically high-quality RAG across a million tokens of context. You can see, look, if it's 1,000 tokens, no problem. If it's 120,000 tokens of context,

558. 1:31:20

     you know, it depends a lot on where those facts are. If they're towards the start, you actually can have much lower recall, and so that's a real risk, which is kinda why it goes back to this whole thing of, uh,

559. 1:31:31

     like I don't really buy just stuffing everything into context, that far right side. I think there's too many issues with bad recall, recency bias, like you said, um- And so I think until we have very-- And by the way, I also don't really trust...

560. 1:31:46

     You know when they show those needle in the haystack charts and it's, like, perfect? I don't trust any of that. I, I did my own study. I found there's, like, a lot of errors, and I think it depends a lot on, um, a couple different things.

561. 1:31:57

     One, how many needles? So in this case, you see with one, it's okay. With ten, it's really bad, right? So how many needles? And then I saw an interesting study saying that, like, the, the difference in the needles relative to your context makes it easier.

562. 1:32:09

     So, like, in these studies, it's like pizza ingredients and Paul Graham essay is really different. But if it's just, like, related slightly, it's actually harder still. So that is to say I don't really trust the needle in the haystack studies.

563. 1:32:21

     I don't particularly trust stuffing, you know, passing a million tokens of context and counting that, and counting on that to just work effectively. I'd be very wary about that.

564. 1:32:33

     That's kinda my thing. But, you know, in these studies, look, a thousand tokens of context, already, you know, that's, that is, uh, if you're stuffing a thousand tokens, eh, I mean, that's actually still pretty small.

565. 1:32:42

     So yeah. I, I'd just be wary about very, retrieval from very large contexts. Yeah.

566. 1:32:49

     Um, what about, uh, the number of tools? Is it better to have small unit tools versus-

567. 1:32:54

     Yeah

568. 1:32:54

     ... complex tools?

569. 1:32:56

     Okay. Yeah, that's a great question. With agents, the number of tools.

570. 1:33:00

     This is a really big issue I hear mentioned a lot. Um, so if you recall, if you go back to the agent stuff, um,

571. 1:33:10

     so you're basically binding some set of tools to your LLM, right? And that's what we show here, right?

572. 1:33:17

     Uh, I've seen a lot of issues with a large number of tools. So I don't know exactly know what the, the ki- exact cutoff is, but this is one of the big problems with open-ended tool calling agents, is if I am basically selecting from twenty different tools, I, I actually, I-- maybe the Berkeley leaderboard has r- data

573. 1:33:40

     on this. If someone knows, feel free to mention it. But, uh,

574. 1:33:44

     reliability of tool calling, even with a small number of tools, like on the order of five, is already challenging. If you're talking about hundreds of tools or dozens of tools, I don't think there's really, uh, yeah, I think it's quite challenging.

575. 1:33:57

     And which is why I've seen more success in, A, not using these open-ended style tool calling agents,

576. 1:34:06

     laying it out more explicitly as a LangGraph where, uh, the tool calls live inside nodes, and you're not relying on your agent to pick from, like, twenty different tools.

577. 1:34:16

     So you can lay out more of, like, a control flow where, um, you route to different tool nodes based upon the logic. So, so that's kinda one thing I've seen.

578. 1:34:24

     Another thing I've seen is maybe multi-agent type things where you have w- different agents with subtasks with each having, like, maybe a small number of tools. But basically, what I've seen is it seems to be that--

579. 1:34:36

     Okay, maybe it's two things. Selection from a large number of tools is definitely challenging. One of the most interesting things I saw is something like, um, you can use something like RAG, where basically you can take a description of your tools, create a natural language description, embed it, and then use basic, like, semantic similarity search, your query

580. 1:34:54

     versus the embedded summaries to select using semantics. That's actually not a bad idea. I would actually use that more than I would trust an LLM to just, like, do the tool selection from a list of a hundred.

581. 1:35:03

     That's not gonna work. So actually, I think that, like, RAG for tool selection's a cool idea. I was gonna do a little, like, test that out and do a little tutorial.

582. 1:35:11

     So actually, maybe I'll just make a note of that. Uh, that's a, that's a great, a great question. Uh, to do RAG for many tools.

583. 1:35:23

     That brings me to the follow-up question because I think the similar approaches you usually often use for data querying-

584. 1:35:31

     Right

585. 1:35:32

     ... like semantic similarity. Was it, yeah, were you thinking about something like that for the tool involved, like-

586. 1:35:41

     Yeah. Well, I, I think, I think using semantic similarity for tool selection is a, is a good idea. Definitely.

587. 1:35:52

     What about the data querying?

588. 1:35:52

     What do you mean by data querying, though?

589. 1:35:53

     Um, as well as the language. If, if you plug in, um, some tables as, as one of your tools or source of information.

590. 1:36:05

     Right. Um, let me make sure I understand. I think the way I would think about it is, so you know how in your, in the notebook, like in the code for all of our tools, right, you have this little tool description,

591. 1:36:20

     right? Like retrieve documents, grade them, run web search. I would actually write verbose descriptions for all my tools and then index those descriptions, right? Or embed them. And then I would do-- And I would probably create, like, very, very verbose, high-quality summaries of what the tool actually does, and then do semantic similarity search against those summaries.

592. 1:36:43

     I think that could actually-- I haven't done that yet, but I think that could work really well, um, because it's a very tall task to ask an LLM to differentiate between, like, twenty different tools.

593. 1:36:53

     Whereas if you could do something like semantic similarity, that would actually probably be very effective.

594. 1:36:59

     Yep.

595. 1:37:01

     Um, I just wanna go back to the multi-agent kind of problem where, like, it seems like it's, you know, pretty cool to want to orchestrate,

596. 1:37:11

     like, you can have, like, an agent provide tools, and one of the tools can call other tools.

597. 1:37:16

     Right.

598. 1:37:16

     Like, have you done any, like, experiments with that? Like, have you, or know any of the researchers who are doing, like, that problem?

599. 1:37:25

     Yeah.

600. 1:37:28

     You talked earlier about agents and, like, it might not be a good idea to, like, have tools-

601. 1:37:31

     Yeah, yeah. So the question was about, um, multi-agent context when you want to orchestrate a large number of tools, having sub-agents with specializations that manage some small set of tools each, and how do you kind of move between them.

602. 1:37:47

     So if you look at our LangGraph repo, we do have a subdirectory. It's under, um- It's under, uh, LangGraph examples multi-agent. We have a few different notebooks that have multi-agent style kind of layouts, which I would encourage you to look at.

603. 1:38:01

     I haven't personally done too much work on it. Um, it seems promising, but I haven't played with it. Um,

604. 1:38:10

     multi-agent in general-- A-and these all reference papers. You can also look at the papers. But multi-agent in general for production setting feels quite aggressive. Uh, although that said, as far as I understand, I remember looking at the code a while ago, Devin and some of the software agents do use, like, multi-agent style setups.

605. 1:38:27

     Um, so maybe have a look at the Devin repo or there's Open Devin. Uh, have a look at these notebooks. Those could all be useful if you wanna learn more about multi-agent.

606. 1:38:39

     Yep.

607. 1:38:41

     Um, so, um, I'm wondering, like, and I stepped out for this so-

608. 1:38:46

     Yeah, sure

609. 1:38:46

     ... I'm gonna ask this. But I'm wondering, like, you've been talking a lot about kind of the variability and unpredictable nature of having the LLM decide which tool to call.

610. 1:38:57

     Right.

611. 1:38:57

     So given that, I'm wondering how do you think about when it makes sense to wrap RAG inside of an agent versus just making a chain on its own?

612. 1:39:10

     That's a classic question. Yeah. So the question was, when do I use a chain versus an agent? So that's very good. So we kind of touched on it a little bit, uh, kinda here.

613. 1:39:21

     So I think that the intuition behind why, where and why an agent can make sense is simply that sometimes you want your application control flow to be variable.

614. 1:39:34

     And if you want some flexibility within your application, an agent is a nice idea. And so all this self-corrective type stuff we're talking about, the corrective RAG thing, those are all kind of agentic flows where the control flow depends upon the grading of the documents.

615. 1:39:49

     And so, um, you know, historically, people have largely been building chains, and chains are very reliable and they're easy to ship and all that. I think with n- things like LangGraph and of course, I work at LangChain, so I'll speak my book about LangGraph, but I've really used it quite a bit, and I found it to be

616. 1:40:09

     reliable. And I-- we are seeing a lot of people starting to deploy with it because you can actually ship and deploy a reliable agent with LangGraph. And so I think a blocker to the ability to kinda ship agents has been reliability.

617. 1:40:19

     And I think we're-- I would actually encourage you to play with the notebooks and look at LangGraph because, um, it does allow you to have that kind of reliability, whether it be necessary to, to, to ship something in production.

618. 1:40:29

     And we do have customers that have LangGraph in production. Whereas a ReACT agent production is n-not recommended.

619. 1:40:35

     But let's imagine my options aren't-

620. 1:40:38

     Yeah

621. 1:40:38

     ... LangGraph agent or ReACT agent-

622. 1:40:40

     Yep

623. 1:40:40

     ... but rather just traditional RAG-

624. 1:40:42

     Sure

625. 1:40:42

     ... versus LangGraph agent.

626. 1:40:43

     Hundred percent.

627. 1:40:44

     In that case, would it only make sense to go with a LangGraph if I wanted something like do retries or ranking of documents or something like that into it?

628. 1:40:53

     So, okay. So I, I think, yeah. So the different-- why would you ever want kind of like an agent, be it LangGraph, ReACT, or otherwise versus not? And I get-- again, I think it goes back to, do you want your application to have any kind of adaptability?

629. 1:41:06

     So okay, here's one we can talk about, routing. I have three different vector stores. I wanna be able to route between them. That is kind of a quote-unquote agentic use case because the control flow depends on the question.

630. 1:41:16

     So that's one. You might want routing. You may want self-correction. So that's kinda what we talked about here a whole bunch with the corrective RAG stuff. So you want routing, you want self-correction.

631. 1:41:25

     Um, I mean, those are two obvious ones in the context of RAG itself. Um, I mean, that's one thing I've often found problem with the RAG systems is, is the routing thing is a real issue.

632. 1:41:37

     Like, you want your system to be flexible enough to deal with questions that are out of domain for your vector store, and, uh, you need some, you need some kind of dynamism in your application to handle that.

633. 1:41:46

     So looking at the question saying, "Okay, just answer this directly. Don't, don't use the vector store."

634. 1:41:52

     So those are, like, the most popular ones, self-correction or routing.

635. 1:41:56

     Great. Thank you.

636. 1:41:57

     Yeah. Yep.

637. 1:41:58

     Uh, I'm wondering if there's anything to be said about building evaluation datasets. Like, question-answer pairs are so domain specific.

638. 1:42:04

     Yeah.

639. 1:42:04

     I'm wondering if there are, like, general best practices, mental models, like things to think about when sitting down to build an evaluation dataset.

640. 1:42:11

     Yeah. Yeah. Okay. So the question was about kind of building eval datasets. Oh, yeah, that's a great question. It's often ver- a very, very challenging part of app development.

641. 1:42:20

     Um, so if you're, if you have a RAG application that's domain specific, then oftentimes you have some set of canonical question-answer pairs you care about.

642. 1:42:28

     Uh, you know, it's hard to sign-- to find like, you know, very, very general rules for that. Y-I think it depends on your application. Um, I think there's kind of this, this hurdle,

643. 1:42:39

     any evaluation is better than no evaluation, so small scale eval sets that you can use and just work are already better than not doing any evaluation. So I mean, for this particular case, I just looked at the document.

644. 1:42:53

     Now, maybe, maybe I'll back up and, and answer. So, so one thing I've seen, and I've done this a little bit, is you can use LLM-assisted QA generation. So here's one thing you can do, and I've done this a little bit with LangChain docs.

645. 1:43:09

     I can build a prompt that says, "Given this document, produce three high quality question-answer pairs from it," right? And I can just basically load my documents and pass them into that LLM.

646. 1:43:19

     I use a high capacity model like Sonnet or four O and have it generate QA pairs for me and then audit them. That's a nice trick. I've used that.

647. 1:43:26

     It actually kinda works. Now, you have to be careful with it. You only would pass it, like, usually one document at a time to keep it really, like, you know, restricted.

648. 1:43:33

     Um, and you audit them, but actually that's a nice way to bootstrap your eval sets. That's, like, idea one. And this-- that gets into the whole idea of synthetic datasets.

649. 1:43:41

     But if you're building, you know, domain-specific synthetic QA pair datasets, that's a nice trick. So basically use an LLM to help bootstrap. I think that's one idea that can help a lot.

650. 1:43:49

     Um- Yeah. And otherwise, I think that basically trying to stand up a small evaluation set, for example, for RAG is, even this case, five questions, but you can already see I can get some nice insights.

651. 1:44:03

     It's very simple to set these up. I have my little experiments all over here, and again, it's only five questions, but it gives me some immediate insights about the reliability of ReAct versus, versus LangGraph agent.

652. 1:44:15

     So keep it small, potentially use synthetic data, start with something, and then, like, kind of build it out over time. Now, a whole other thing here is we didn't talk about this too much, but if you have an app in production, then the way this whole thing kind of comes together is

653. 1:44:31

     you can actually have different types of evaluators that run on your app online. We call those online evaluators, okay? So this is with our internal app, and this gets back to the question I think he mentioned of you can have a bunch of evaluators for RAG that don't require a reference.

654. 1:44:45

     So like, um, oh, I don't show it here, but, but basically I can look at, like, document retrieval quality. I can look at my answer relevance or hallucinations. I can run that online.

655. 1:44:54

     I can flag cases where things did not work well, and I can actually roll those back into my eval set. So if I do that, uh, then I actually have this self-perpetuating loop like Karpathy talked about with the data flywheel, where actually I'm running my app in production, I'm collecting cases of bad behavior that I'm tagging with,

656. 1:45:14

     like, online evaluation, and I'm rolling those back into my offline eval set. So what you do there is look at the case that the app is doing poorly in production, audit them, correct them, so build, like, a canonical question-answer pair from that and put that back into your test set, and that's a good way to build, bootstrap

657. 1:45:30

     and build it up. So I'd start cold start, synthetic data, small scale examples, online evaluation or some system to check online where it's failing, loop those back in and build it up that way.

658. 1:45:41

     That's like your data flywheel. Yeah. And actually, I even had a slide on this in one of my older talks. I used to work in self-driving for many years, and I actually was a big fan of Karpathy's stuff at Tesla, and actually I've had his, his thing here of this is like the, the data engine thing of

659. 1:45:57

     like, you know, you ship your model, you do some kind of online evaluation. Where is it failing? Capture those failures, curate them, put them back in your test set, run that as a loop.

660. 1:46:06

     That's like-- He called it Operation Vacation 'cause you can go on vacation and the model keeps getting better. And this was more in the context of like training models 'cause basically all those, those failed examples, once they're labeled, they become part of your, your training set.

661. 1:46:19

     But the same thing applies here with LLM apps.

662. 1:46:25

     Cool. Yeah.

663. 1:46:27

     Yeah. So, uh, I, I just wanted to ask, uh, you, you just mentioned text-to-SQL, uh, generation, right?

664. 1:46:34

     Oh, yeah.

665. 1:46:35

     It creates the tables and everything. So, uh, I, I just had a question where like, for example, all that, uh, all that happens and we send tables schema, and it generates its SQL query, and then we run it on, uh, the database. [clears throat]

666. 1:46:50

     And if the result set is too, too large or something, so, uh, we cannot send it for another LLM to generate a user-friendly way or something, like user-friendly answer or something like...

667. 1:47:01

     How do we handle that? Like what you suggest on, like, the query, uh, result set is so big that we cannot fit it into the context. And we gen-- A-and I want to generate a user-friendly answer for that.

668. 1:47:14

     Yeah. So-

669. 1:47:14

     With the data. [clears throat]

670. 1:47:16

     Yeah, the question was on text-to-SQL. Um, we actually have a pretty nice text-to-SQL agent, uh, example here. Um, so it's in LangGraph examples. Um, I think it's in... Is it SQL?

671. 1:47:31

     Uh, where is it? Um, I'll find it here. Uh,

672. 1:47:41

     where is it? Um, uh, tutorials. Oh yeah, it's in tutorials. Um, so SQL agent here. Um, I think a lot's in the prompting. So basically in this particular case, I believe Ankush from our team set this up.

673. 1:48:01

     Um, you prompt your-- You can do a couple things. So you can prompt your SQL agent to, um... Where is it? It's somewhere where he tells it to... Yeah, it's based on all these instructions.

674. 1:48:16

     Um, be very careful about, uh... Let me just find the... But ba- in short, you basically instruct the SQL agent when it's writing its query to, um, ensure not to extract an excessive amount of context, and I can't remember exactly where it does that.

675. 1:48:36

     Uh, okay. Yeah, it's right here. Um, limit your-- Always use a limit statement on your query to restrict it to like whatever it is, five results, and that's the hard-coded thing here.

676. 1:48:48

     And then also in your SQL agent, you can incorporate a query check node, um, to actually look at the query before you execute it to sanity check for things like this.

677. 1:48:58

     So basically I would have a look at the, um, LangGraph examples tutorial SQL agent notebook to have... I've, I've actually, uh, I ran this and evaluated it, and I found it did work pretty well.

678. 1:49:11

     So that's one thing that I would look at.

679. 1:49:15

     Uh, yeah. One more question on text-to-SQL. So a lot of times, like with text-to-SQL, um, they do well on like group by kind of queries. But how do you handle like filtering questions like where clause like when you have a high cardinality column?

680. 1:49:32

     Like, like I have a question, right? Like if it's converted to SQL query and it needs to use a where, where clause in SQL, where is actually translates to like any column value, like any value from the column, right?

681. 1:49:43

     Like especially if you have some table with very high cardinality, like maybe five hundred values like that. Like how do you handle that? Like that's what I-

682. 1:49:51

     Hmm Yeah. So the question was related to, like, how do you handle high cardinality columns? And I guess that is related to restricting the output size. Um, I mean, I'm actually not really a SQL expert, so I'm probably not the right person to ask about very gory details of text-to-SQL.

683. 1:50:11

     Um, but in general, I would kind of consider can you prompt the LLM effectively? Like, okay, two different things. One, the general ideas here were basically upfront prompting of your LLM, uh, to kind of follow some general kind of query formulation criteria.

684. 1:50:30

     And then two, an actual query check explicitly to review and confirm it doesn't have some of the issues like ex-extracting excessive context. But for anything more detailed than that, um, I'm probably not the person for those insights.

685. 1:50:46

     But there might be a specific text-to-SQL deep dive at some other point in this conference, and you should definitely seek that out. But I haven't done that much with text-to-SQL.

686. 1:50:57

     Yeah.

687. 1:50:58

     Um, you've seen quite a lot of, you know, question answer RAG patterns, but one thing I'm quite interested in is more long-form document generation, so, you know, writing a report or something like that.

688. 1:51:07

     Yeah.

689. 1:51:07

     Um, have you seen any interesting patterns or, like, what you think about some smart different solutions, whether it be LangGraph or other types of things to achieve that type of output?

690. 1:51:16

     Yeah. Okay. So the question was related to kind of document generation. Uh, that is a really good theme. So we actually have kind of-- There was an interesting paper.

691. 1:51:27

     I did this a while ago. I need to find it. Um, where is it? So we have a notebook. If you look in LangGraph examples, um, Storm. So this was actually for wiki article generation.

692. 1:51:41

     Um, here's kind of the diagram for it. We actually have a video on this too. Um, I'm actually trying to refresh myself. I did this, like, three or four months ago.

693. 1:51:49

     Um, but it was basically a multi-agent style setup in LangGraph, uh, where if I recall correctly-- I'm just looking at the flow here myself. But basically, what it did was you give it a topic, and it'll kind of do initially this kind of, like, generation of related topics, um, and actually uses the multi-agent thing of, like, editors

694. 1:52:11

     and per-- and, um, experts. The experts go and do, like, web research. Don't worry too much about the details. So the point is it was an interesting paper in flow for wiki article generation using LangGraph in a multi-step process.

695. 1:52:24

     And actually, the wikis are, like, pretty good. I think at the bottom of the notebook, I have an example wiki, so you can see all the code here. Uh, that's the graph.

696. 1:52:34

     Yeah, and then here's the final wiki that you get out from this, this type of thing. So it's pretty good. Um, have a look at that notebook. Um, I think, yeah, Jason Liu also had a post on this recently, this idea of, like, report generation's a theme that, like, we're gonna see more and more of.

697. 1:52:50

     This was a-- this was one idea that was pretty sophisticated, though. You can probably simplify it a lot. I've done a lot of work just on, like, a simple kind of distillation prompt, like perform RAG and then have some generation prompt give a bunch of instructions for how I want the output to be formatted.

698. 1:53:06

     That also is really effective. Yeah.

699. 1:53:10

     Um, just for the user-facing chat interface, what is a suggested way to implement a clarification tactic also in the interface? Because during the reasoning process stage, you might, you know, or I might get to know, like, uh, whatever RAG information that you provided, like say mode three or mode four, you might understand that there is some information

700. 1:53:31

     that is very specific to the domain and that we do not have. Uh, what is a way to implement a pass back to the reviewers so that they can provide information that may be used subsequently?

701. 1:53:42

     Yeah. So the idea-- The question was related to user feedback. Uh, that's a really good one. So we do have-- I think I mentioned previously, if you look at LangGraph, um, where is it?

702. 1:53:54

     We-- I believe we have some user feedback examples. Um, Thursday, we're definitely gonna be announcing something that has a lot of support for user feedback, and I would encourage you to keep an eye out for that.

703. 1:54:07

     So Terrason's gonna launch that here on Thursday. Um,

704. 1:54:12

     I will look for-- I know we have some user feedback examples in LangGraph, but I will need to find them. Let me see. We probably...

705. 1:54:26

     Let's check Lang. We probably tweeted about it at some point. I haven't actually done anything with user feedback, though. Let's see.

706. 1:54:36

     Uh, yeah. Hmm. Yeah, maybe I'll-- I might have to get back to you on that. I thought we had some nice examples with LangGraph.

707. 1:55:03

     Hmm. Yeah, I'll have to get back to you on that one.

708. 1:55:10

     Hmm. Feedback. Customer support might be in here.

709. 1:55:31

     Hmm. Let's try something else. Look at LangGraph docs.

710. 1:55:52

     Uh Hmm. I'd poke around the LangGraph docs.

711. 1:56:08

     The-- We have a bunch of tutorials in the docs here. Um, just Google LangGraph documentation. Um, I'm just poking around here for user feedback.

712. 1:56:20

     Uh, ah, here we go. Look at this. So LangGraph how-tos human-in-the-loop. I would have a look at that. I have not played with that myself, but yeah. [background noise]

713. 1:56:34

     Apart from, uh, you asking for user feedback, what would be a good way to judge whether there is missing information and then ask for the user feedback? Basically, looking at the information that is there and looking at what the user had asked you.

714. 1:56:49

     So basically trying to judge whether we have all the information that is required to answer the question, else put the human in the loop and-

715. 1:56:57

     Yeah, yeah, yeah. Okay. Right. So for like, um, mid to longer term problem-solving tasks, how do you incorporate user feedback to ask for more information? So, um, I would have a look at this documentation 'cause I would imagine it will, uh, cover examples along those lines.

716. 1:57:16

     I haven't personally done that. Um, I also believe that

717. 1:57:24

     I would have a look at the customer support bot, um, because that's an example of, of a kind of multi-turn interaction between a user and a support agent. Um, so I would look at the customer support bot, which Will and my team did, as well as the documentation on human-in-the-loop.

718. 1:57:42

     So those are two things I would check out there. Um.

719. 1:57:52

     Nice. Yep.

720. 1:57:58

     Is there any research into model architectures for training models for agentic reasoning, optimized

721. 1:58:06

     for agentic reasoning?

722. 1:58:06

     For agentic reasoning?

723. 1:58:15

     Yeah.

724. 1:58:15

     Um, yeah. So that's, that's kind of an interesting question. So the q- the question was related to training models specifically for agentic reasoning. Um, if anyone-- So I mean, there's a lot of,

725. 1:58:27

     there's a lot of work on prompting approaches for different types of reasoning, for sure. Um,

726. 1:58:34

     I'm a little bit less familiar with like efforts to fine-tune a model specifically for particular like agentic architecture or use case, but you could imagine it. Um, most of the work that I've encountered, though, is just using generalist, like high-capacity generalist models with tool calling and specific prompting techniques.

727. 1:58:53

     So re- like, ReAct is kind of a particular orchestration flow and prompting technique rather than, you know, and you can interchange the LLM accordingly. I think the main thing typically or historically for, for agents has been the ability to perform high quality and accurate tool calling because agents, that's one of the central components of agents.

728. 1:59:18

     And so, um, that's kind of been the gating thing, and I think model providers have been focused a lot on just high-quality tool calling, which helps kind of like all agent architectures.

729. 1:59:27

     So I haven't seen as much on like fine-tuning for one particular architecture. I think it's like high-capacity generalist models with tool calling and then prompting. So it's more like in-context learning.

730. 1:59:37

     That's kinda the, the trend I've seen at least.

731. 1:59:42

     Yeah.

732. 1:59:43

     Um, can you talk a little bit more about, uh, LangGraph checkpoint? 'Cause I just saw that in the notebook.

733. 1:59:52

     And, uh, am I right in saying that it's more about sort of production-friendly feature and then sort of saving the state?

734. 2:00:00

     Yeah.

735. 2:00:00

     How much else does this do you think? Can you just talk a little bit?

736. 2:00:05

     Yeah. So the checkpointing stuff, actually, this Thursday, that's gonna be a lot more relevant because we're launching some stuff to support, uh, deployments for LangGraph. In which case, you, you can do a d-- bunch of different things, but you can have a single state that persists across many different sessions.

737. 2:00:22

     You can also have checkpoints, or you can return to state and revisit an agent from a particular point. Um, don't worry about that too much for now. I think there'll be a lot more documentation and n- and kind of context for that on Thursday when the stuff for deployment comes in.

738. 2:00:39

     But it's good to be somewhat aware of. I would poke around the documentation, uh, for a little bit more on checkpointing, but it really becomes relevant on the stuff we're announcing on Thursday.

739. 2:00:49

     Um, so I would have a look then.

740. 2:00:53

     Uh, let's see if there's-- if we've updated our docs.

741. 2:00:58

     Yeah. Yeah. So there are s-- there is some documentation on it now, but it'll become a lot more interesting and relevant come Thursday when we have a lot more support for deployment.

742. 2:01:09

     Yep.

743. 2:01:11

     Um, how do you suggest we, we deal with the issue when the agent, uh, is not, is not able to, um, to call the right tool or makes small mistakes in, uh-

744. 2:01:22

     Yeah

745. 2:01:22

     ... choosing the tool? For instance, instead of YouTube video download, it says YouTube video upload.

746. 2:01:28

     Yeah.

747. 2:01:28

     Do you stop the process or do you try to-

748. 2:01:31

     Okay. Yeah. So that, that's a, that's a really good question. And so

749. 2:01:38

     the way it works with the existing, um-- So it depends on the architecture. So using the ReAct architecture, let's see if I can find an example of it. Um, so here's with ReAct agent.

750. 2:01:51

     Uh, let's look at one of the traces. Let's see if I have an example. So basically, the tool call itself will return like an error, and the LLM then is expected to self-correct from that error.

751. 2:02:04

     It has to kinda self-correct. So that, that's kind of one approach That at least we do with the React agent. Um, so actually you can see it in the notebook, um, if you go to--

752. 2:02:22

     And if I can find some traces that have that example, I will pull them up. Um, but,

753. 2:02:29

     uh, I think it's in utilities somewhere. Yeah, so basically this tool node with fallbacks. Basically what happens is in this tool error, so this, if there's an error in the tool call itself, it'll return that error, and usually the agent will then, or the LLM assistant will look at that and, like, self-correct its tool call.

754. 2:02:50

     So that's, that's typically how it's done. And this actually is reasonably effective. But again, you know, the nice thing about the, the other implementation, the custom agent I called in the notebook, is you don't rely on tool calling in this way, and so you can get around this type of issue.

755. 2:03:09

     Um, but basically catching the errors in the tool call itself with this code is, is what's currently done. Let's see if I can actually find an example. Um,

756. 2:03:20

     yeah. I may have to-- I can look for one where it gets the answer wrong. Yeah, let's see. This one. Let's see if we can find a tool call failure.

757. 2:03:29

     Um, so here's the trace. Um, let's see.

758. 2:03:36

     Uh, okay. This didn't have a tool call error. Yeah. So basically you-- what you'll see in the message history is they'll-- like the tool itself will return this error message, and then the LLM will say, "Oh, okay, I need to re-retry," and then it'll retry and hopefully get it right.

759. 2:03:52

     Yeah. Yep.

760. 2:03:54

     So follow-up to that question is I use, uh, Python, Pydantic, and, uh, Jason Liu's Instructor-

761. 2:03:59

     Oh, yeah

762. 2:04:00

     ... uh, which is great for that exact problem right there.

763. 2:04:02

     Yep.

764. 2:04:02

     It does like the instant revalidation of like the output, especially for those simple little errors.

765. 2:04:07

     Yes.

766. 2:04:07

     Um, I'm wondering, well, first of all, that probably works, but does, um, LangChain, I haven't used it much, like could you use it with Instructor and Pydantic in a similar way?

767. 2:04:16

     Okay. So this is a very good point. So yeah, I'm a big fan of Instructor. Um, I haven't used it as much, but what you're saying is, is one particular type of tool call.

768. 2:04:27

     So basically that pertains, I believe, more to structured outputs, which is indeed a kind of tool call. And wh- when you're using something like a Pydantic schema, you're right, it's very easy to check and like correct errors.

769. 2:04:38

     So I've found catching errors like with schema validation, like using Instructor is, is really good and we have some other things you can use within LangChain to do the same thing.

770. 2:04:49

     So, so that's one type of error that's actually particularly easy to kind of detect and correct. What we show in this notebook here, and the code I showed is more for any general tool.

771. 2:05:01

     So, um, so this code here will operate on any tool you call regardless. So it doesn't have to do with structured outputs or anything. And so it's just a more general, uh, check for tool call errors.

772. 2:05:16

     Now, in terms of Instructor with LangChain, now maybe I'll just back up a little bit. So LangGraph does not require LangChain at all. So that's kind of point one, and neither does LangSmith.

773. 2:05:28

     So actually, everything we're doing here does not need to use LangChain. So actually,

774. 2:05:36

     that could be a pretty interesting thing to try for like kind of the choose your own adventure thing. But basically in the custom agent part, um, I use with structured outputs to do the grading.

775. 2:05:50

     So if you go to, um, yeah, if you look at the, um,

776. 2:05:56

     the retrieval grader here, so this is using LLM with structured output and here's my grade schema. Try that one in Instructor. That should work great. You don't need LangChain at all for this.

777. 2:06:06

     Um, and that'll fit right into LangGraph. So actually, I think it'd be great to use Instructor with LangGraph for this particular use case. And I do agree that Lang-- that Instructor is really nice for those kind of like schema validation error correction.

778. 2:06:20

     I th-

779. 2:06:20

     Plug and play.

780. 2:06:22

     Plug and play. That, that-- I'm gonna make a note of that. That's a really good kind of choose your own adventure case. Um, where should I put that?

781. 2:06:33

     Uh, try Instructor with LangGraph for grading. Yeah, I like that a lot. [keyboard clacking]

782. 2:06:43

     Yep. Yep.

783. 2:06:48

     Um, kind of more of a meta question, just, uh, it really aligns with what you guys are trying to tackle. Um, what's the-- what are the path forward, uh, to continue to make this technology better?

784. 2:06:59

     And specifically, where is the non-- uh, where is the RAG pipeline family right now? Like, what are we still not good at? And you guys have been close to do so, like down and long term.

785. 2:07:10

     Yeah. So question was related to just RAG in general, and where is RAG?

786. 2:07:15

     Like, uh, RAG agents.

787. 2:07:17

     RAG agents. Yeah, sure. Well, to be honest, a lot of the problems with RAG, I think about our own internal application chat LangChain.

788. 2:07:28

     A lot of the problems with RAG actually are retrieval problems. Retrieval is just hard. I'll give a good example. Like LangChain, we have, um, I'm trying to remember, five million tokens of context across all our docs, something like that.

789. 2:07:44

     We have all sorts of d-- We have a very long tail of integration docs. You want very high quality coverage and questions across all of that. There's a lot in how you index all that stuff to ensure that you

790. 2:07:55

     boost retrievals from more canonical how-to guides that are much better documented, but still having coverage over long tail content for like, you know, long tail questions. For example, if you're using raw semantic similarity search You could have relevance to, you know, say, your how-to guide, which is really well-developed, and three random long-tail documents that are not well-developed,

791. 2:08:18

     and they'll all get returned. And so how do you overlay different systems, it could be re-ranking, uh, to basically promote content, uh, that you believe to be more accurate or better based on some criteria?

792. 2:08:33

     So that is all to say, I think with RAG, the challenge is actually just domain-specific retrieval for your application. That's just a hard problem, and there's been a lot of work on this.

793. 2:08:41

     It's been around for a long time. I think that's really the limiter. And actually, there's kind of no, no silver bullet. Like in our case, we're having, having to look at the structure of our documents very carefully, design our retrieval strategy based on that doc structure.

794. 2:08:56

     Like in particular, we're thinking about applying certain re-- um, post-retrieval ranking to docs of certain types based upon their importance. We're thinking about retrieving a large, like a large initial number of docs and then boiling them down with, with kind of re-ranking based upon importance.

795. 2:09:13

     So I still think retrieval is very hard. It's very domain-specific. It depends on the structure of your documentation, and there's kind of no free lunch. And I think the things that are good about RAG is context windows are getting much larger for LLMs.

796. 2:09:25

     And so back to that point I was making before, I think we're seeing, and we're considering this ourselves,

797. 2:09:33

     less worry about the exact right chunk size. You can think more about chunking in, you know, different ways, um, and then passing full documents to your final model. So I think that part of it's really good.

798. 2:09:46

     Um, but still, even like, even in this particular case, you probably still need some, uh, re-ranking to promote the most important documents. Um, so I think retrieval is still quite hard.

799. 2:10:03

     In particular, um, like even looking at the LangChain docs, in particular, the overlay of document importance on top of raw semantic similarity search, right? Take a case of like I have a question, semantically it's similar to ten different documents.

800. 2:10:22

     Those documents, they'll vary widely in their quality and their relevant, like, uh, more like higher level relevance. Like maybe that passage is related, but like it might be a general question about how to build an agent, and then some random integration doc talks about building an agent for integration X.

801. 2:10:38

     And I wanna make sure that the more canonical, well-developed agent, you know, overview doc gets promoted and passed back in the end answer, stuff like that. Sorry, it's a long answer, but basically RAG is...

802. 2:10:50

     It's hard. I mean, I think retrieval is really the hard part.

803. 2:10:54

     The generation part is getting better and better as long contexts grow.

804. 2:10:58

     Uh, yep.

805. 2:11:00

     Lance, so, um, this re-ranking approach for your documents.

806. 2:11:04

     Yeah.

807. 2:11:04

     Um, what's, what's the metadata? Do you have a relevancy to a particular topic as well as numerical ranking?

808. 2:11:11

     Yeah. Okay, that's a great question. So the question was, when we talk about this re-ranking, how do you assign this relevance to your documents of what is that? So I'll just give you what we've been thinking about.

809. 2:11:23

     Uh, I actually think it is, for us, going to be a hand-tuned kind of relevance score based upon our doc structure. So if you look at the LangChain docs, like, um,

810. 2:11:37

     go to LangChain, uh, documents. Um, yeah, so LangChain documentation. We have these sections up here, tutorials, how-to guides, conceptual guides, which are like really well-developed, more recent, well-curated.

811. 2:11:54

     These, you can imagine, have some kind of relevance or importance ranking of one or highest. So these are documents that contain very high quality, well-curated answers that we wanna promote and serve to users in the generation phase.

812. 2:12:06

     However, let's say someone asks a question about one particular integration, right? If you go to Integrations, we have all these pages, right? Components, go to Retrievers, look at the, you know, Zepp Cloud retriever.

813. 2:12:19

     This is some stuff related to Zepp Cloud specifically. If someone asks about Zepp Cloud, you do wanna be able to retrieve that doc, right? And so, um, some ability to differentiate between questions that need, you know, general answers, in which case you would promote your more canonical how-to guides, conceptual docs, versus questions that require retrieval from very

814. 2:12:40

     specific integration docs, in which case you would still promote this information. That's kind of the crux of it. And I think we'll probably use kind of manual or heuristic scoring to up, up weight or up rank, um, our core like how-to guides and conceptual guides, um, over longer tail integration docs.

815. 2:13:00

     And we might have a router that will indicate whether the question is general or specific. So those are the two things that I'd probably do. So routing on the question side, and then some kind of heuristic, uh, relevance or importance grading or quality grading on the document side.

816. 2:13:17

     And that can be packed in the metadata that you pack along with your, your index chunks.

817. 2:13:23

     Yep. Uh, maybe not. Yeah. Oh, yeah.

818. 2:13:30

     So, uh, so let's say, uh, a typical RAG application where there's a question and answer pair, but we kind of maintain the multi-turn. Like, for example, we maintain the conversation history of the user to kind of create a conversation.

819. 2:13:41

     So the problem is, like, let's say, for example, a question is asked, and then the, the retrieved chunks are like, let's say, five-

820. 2:13:48

     Right

821. 2:13:48

     ... right? And then a subsequent question is asked. So let's say, uh, no, but it's related to the first question. But still, somehow in the first node, you transform the query, and then the retrieved chunks are still the same.

822. 2:14:00

     So like, I would just get an answer which is like more of the first answer. So like, how do you, uh... My question is like, how do you make sure that, let's say- He wanted to deep dive into the document, like, uh, into the more context.

823. 2:14:17

     How do you make that happen in like different-

824. 2:14:20

     Yeah. So the question I guess was like in a multi-turn RAG context, let's say you have a case where, um, a user asks like an initial question

825. 2:14:30

     and you retrieve some documents, you produce an answer, and they ask a follow-up that says, "Give me more information about this." Now, do you wanna re-retrieve or do you want to re-reference those same docs?

826. 2:14:44

     No. So, uh, what happened in my case is like-

827. 2:14:47

     Yeah

828. 2:14:47

     ... I kind of, I try to rewrite that question.

829. 2:14:50

     Okay. You do a rewriting. So you rewrite the question, okay.

830. 2:14:53

     And then go search the documents. So most often the document would be the same as the first-

831. 2:14:58

     The same as before.

832. 2:14:59

     Yeah.

833. 2:15:00

     Okay.

834. 2:15:00

     So the answers would be like mostly the same.

835. 2:15:03

     Okay. Okay, interesting. So the problem there is more of a retrieval problem. You're doing a rewrite. You're still retrieving the same set of documents, though. Now, what do you want to have happen?

836. 2:15:15

     You want to-- Do you actually want to retrieve different documents, or do you want to-

837. 2:15:23

     It's kind of like deep dive. Like for example-

838. 2:15:26

     Yeah. But that's the question. What do you mean by deep dive?

839. 2:15:29

     So, uh-

840. 2:15:29

     Like, you're retrieving, let's say it's a chapter of a book. You're retrieving only the first page and you wanna retrieve the whole chapter.

841. 2:15:36

     Yeah.

842. 2:15:37

     Okay.

843. 2:15:37

     It is, it gives you the next chapter.

844. 2:15:40

     Okay. Then I think actually a question rewrite would probably not sufficient. What I would think about more is for that second pass,

845. 2:15:48

     um, you could actually do something like metadata filtering on your chunks. If you have your data or your documents partitioned by like chapters or s- or some sections, I would just do a bulk retrieval of the whole section or something like that.

846. 2:16:03

     So it's more like a trick on the f- on the retrieval side rather than a w- rewrite of the query. 'Cause I hear, I see what you're saying. You rewrite the query, you might get the same docs back.

847. 2:16:12

     If you wanna guarantee that you actually get like a deeper dive in your docs, then maybe it's something in your retriever itself. You could increase K, so retrieve more docs.

848. 2:16:19

     You could use metadata filtering to, like, ensure you get all the docs in a given chapter. So I think it's more a retrieval thing.

849. 2:16:28

     But that's kind of an interesting point though, yeah.

850. 2:16:36

     Cool. Well, I know it's been two and a half hours almost. So there we go. It was good. Um-

851. 2:16:44

     Bye.

852. 2:16:44

     Yeah, yeah. Yeah. I'm, I'm, I'm hanging out for another, till noon, so. Okay. So I did the, um, local tutorial. Oh, cool. And, um, it's working fine. I was just wondering about the sports questions.

853. 2:16:57

     Is that like control group questions? Yeah, yeah. Okay, okay. This is good. So, um, see, the question was he's doing the, the local, the local, um, agent tutorial, and the question's on the eval set.

854. 2:17:11

     So actually, that's a fun one. Modify them any way you want. The key point was I wanted some questions that are definitely outside the vector store. So I asked something about like two things about sports 'cause I know it's not in my vector store about agents.

855. 2:17:24

     So I think I indexed three blog posts about like agents and prompting and adversarial examples. I just wanted some orthogonal questions that'll force web search. So that's the only thing there.

856. 2:17:34

     But you actually play with those, and you can modify them and all that. Um, yeah. But that's cool. It's work-- Are, are you using Llama 3? Yeah. Cool. Yeah.

857. 2:17:44

     I was doing the 70B. Oh, you've, you're, you have a laptop big enough for 70B. That's- Not really. Okay, okay. You're at the edge of- There's been one that it's taking like a minute.

858. 2:17:53

     Okay, okay. But then I switched to the, um, the 8B. Yeah, 8B, 8B's- Much faster. Yeah, exactly.

859. 2:18:00

     I mean, it's actually kinda nice you can even run the 70B, to be honest. I'm not sure I can even run it, but yeah, that's cool.

860. 2:18:08

     Uh, nice. Let's see. Yeah. Well, I can just hang out and, um... Oh, yeah.

861. 2:18:21

     I have a question.

862. 2:18:21

     Sure.

863. 2:18:22

     Um, what would be the best way to incorporate follow-up questions? Uh, is it prompting or...

864. 2:18:30

     Yeah. Well, if you have a chat application, so the question was related to how do you incorporate multi-turn.

865. 2:18:41

     So if you look at the React agent, it uses a chat history as its state. In that case, follow-up questions will be captured just in the message history as part of chat.

866. 2:18:55

     The, I think the current layout of the custom LangGraph agent, though, is a little bit more single sh- single turn, so it'd be kinda question answer. Um,

867. 2:19:06

     that's... Oh, yeah.

868. 2:19:08

     I, I meant like that the agent actually has to come back with a question to-

869. 2:19:14

     Oh, okay

870. 2:19:15

     ... about details which were-

871. 2:19:17

     Okay, got it. So the... Okay, got it. So the question was how do you modify the agent so that it, it will actually return, like if it needs more clarification from the user?

872. 2:19:29

     Um, yeah. These particular agent examples, uh, don't do that. Um, but again, I think

873. 2:19:40

     that's what may be a good takeaway for me. I should add that to these tutorials, um, incorporate a simple example of multi-turn. Um,

874. 2:19:52

     I will, I will do that and, uh, get my contact and I will send that to you. Um-

875. 2:20:01

     For now, I-

876. 2:20:02

     Yes

877. 2:20:02

     ... I have like a simple eval, uh, or fo- follow-up questions.

878. 2:20:06

     Yes.

879. 2:20:07

     I think that should be a bit smarter than-

880. 2:20:10

     Yes, exactly. So you, you want... So I mentioned previously, if you look at LangGraph, um...

881. 2:20:19

     Let me find it. It's, it's one of our notebooks.

882. 2:20:24

     Um, the customer support agent, uh, this, so LangGraph examples customer support is an example of an agent that has like multi-turn dialogue, but it's complicated. So I'd like to maybe augment these tutorials with a simpler example.

883. 2:20:44

     Um, I will... Yeah, I will, I'll follow up on that if you give me, give me contact info, and I'll, I'll send you something. [sighs]

884. 2:21:22

     Cool. Well, I'll just, I'll sit up here. Anyone can just come and grab me. Uh, thanks for everything. Hopefully, the cookbooks are working. Um, yeah, it was good. [audience applauding] Made it two and a half hours, so.

885. 2:21:35

     Good. Thanks. [outro music]