Playground in Prod - Optimising Agents in Production Environments

Talk transcript

1. 0:00

   [upbeat music] I'm Samuel.

2. 0:15

   I'm probably best known as the creator of Pydantic, the, the open source library. Now I run Pydantic, the company. We do a bunch of stuff. Um, Pydantic, we still maintain Pydantic Validation, of course.

3. 0:26

   We have Pydantic AI, the agent framework, and then we have Pydantic Logfire, our observability platform. How many people coming into this know, have heard the word Pydantic? Just, just to...

4. 0:35

   Okay, that's reassuring. Um, and how many people have at some point played with Pydantic AI? Yeah, Pydantic AI, and Logfire? Okay. So most people. Um, Logfire is fundamentally under the hood, a general observability platform, OpenTelemetry, logs, metrics, traces.

5. 0:52

   I don't really believe in AI observability. I think it's a feature, not a category, and it will get eaten by either observability or AI at some point. But today we sell as AI observability because that is understandably the thing that people want.

6. 1:05

   Um, one of... So we go beyond, like, the standard observability of logs, metrics, traces. We do stuff like evals, which I'll talk about a bit today, and managed variables is one of the, one of the newer features.

7. 1:17

   Um, and then the step beyond managed fe- managed variables, which we're working on at the moment, is then optimizing your agent, uh, autonomously, effectively from the platform. That's not what I'm gonna demonstrate today.

8. 1:28

   I'm gonna talk about GEPA and how you can use it with managed variables, kind of in a somewhat Heath Robinson way until we have the full end-to-end all singing, all dancing, you give us, uh, your money, and we magically optimize your agent.

9. 1:41

   Probably also more interesting than the, like, magically we make your agent better argument anyway. Um, so yeah, there are two, there are two, um, core subjects I'm gonna talk about, and then I'm gonna try and put them together at the end.

10. 1:55

    So GEPA, um, is a library. How many people here have heard of, have heard of GEPA and, and kind of feel like they understand what it does? Okay. Well, that, that, that probably includes me to an extent.

11. 2:05

    Um, so it, it... The name comes from, uh, genetic Pareto. So it's a genetic algorithm in the sense that it, uh, looks for the best, um, value of some kind.

12. 2:17

    Um, and then once it's found a good value, it kind of mixes it with some other values that appear to be good and produces a new, um, candidate a-and tries to, uh, optimize that.

13. 2:27

    So it's ultimately an optimization library. It optimizes a string. Now, that string can be a simple text prompt, or it can be some JSON data, which ultimately contains whatever you want.

14. 2:37

    Um, and the Pareto bit comes from, it basically takes candidate from the kind of Pareto frontier of the best, uh, examples it has. So it is effectively, if you imagine you're breeding racehorses, you take the best racehorses, and you breed them each time.

15. 2:51

    You don't, for the most part, go and take some really slow horse and, like, add it into the mix to see what happens. You take all of the, like, best racehorses and breed them and hope to get to, to better, better racehorses, and it does a similar thing.

16. 3:02

    And then the second is managed variables. So

17. 3:06

    lots of platforms, including ours, have prompt management. So basically the idea that you can have, uh, your text prompt i-in the platform, and you can edit it from there.

18. 3:14

    We take, uh, managed variables one step further. Uh, sorry. We take pro-, um, uh, prompt management one step further, and we have managed variables. So they don't have to be just text.

19. 3:23

    They can be effectively any object that you can define with a Pydantic model can be, uh, managed inside, inside Logfire. And so we're gonna, we're gonna kind of try and com-combine those two things today.

20. 3:35

    I will say a-a-at the beginning of this, due to some family situations, I wrote most of this talk overnight. So, uh, maybe in keeping with the rest of this conference, it may be slightly, uh, chaotic.

21. 3:47

    If I... If what I'm saying doesn't make sense or I go to sleep at any point in this talk, please throw something at me, and I will endeavor to make more sense.

22. 3:55

    Um, so I'm gonna talk quickly about the subject that we're gonna go and optimize. It has to do with politics. Don't worry, it's not particularly controversial politics relative to what's going on at the moment.

23. 4:08

    But, um, how many people... Actually, I wo- I won't do the how many people. There's a, there's a, um, podcast called, um, The Rest is Politics, which I'm a big, big fan of and a big listener to.

24. 4:17

    They were having a conversation back in April last year, almost exactly a year ago, in fact, about how many politicians basically come from, like, political, um, dynasties, families of, of politicians.

25. 4:29

    And there was a lot of, like, bluster about what the answer might be, and I used Pydantic AI to go and analyze the Wikipedia article for each MP to basically look for references to, um, their relations who were also politicians, and therefore was able to come up with some percentage.

26. 4:44

    I think it might have been twenty-four percent of MPs have, uh, some kind of, uh, ancestor who is a, who is a politician. But at the time, I basically just ran it with whatever model I could get, got an answer, submitted my question to The Rest is Politics.

27. 4:57

    They read it out. That was very nice. But, like, I didn't actually go and check how well it had done. And so here we're gonna take, um, that challenge and try to optimize it to improve the prompt that we use.

28. 5:09

    Uh, the, the subtlety... Let me, let me go into some code here and, and talk through the task. So hopefully this is a scale you can read.

29. 5:20

    Um, there's a bunch of, uh, fluff here. In particular, there's a, there's a, um, a archive file here that will be automatically downloaded if you run the script, so you don't have to go and do all the scraping from Wikipedia, which just contains the, the basically the raw HTML of the Wikipedia pages.

30. 5:36

    Um, we then have some schema for, for the, for the MP's data, and then we have schema for their relations. So in particular, so this is obviously the name, the role they might have had, in particular their relations.

31. 5:48

    So whether they're a, like, sibling or a spouse or a grandparent. And in particular, with regard to the question that we're trying to answer about, uh, kind of, um, political dynasties, it isn't really relevant if someone has a child or a spouse or a sibling who is also a politician.

32. 6:06

    It's to do with their, like, parents' generation or parents, parents... basically ancestors. And this is something that I found really hard to get models to, um, to respect. Once you said relations, it just couldn't help but add spouse or children or, uh, sister-in-law into the, into the mix.

33. 6:25

    And so the way I actually went and solved this last year was I basically said, "Include the relation, whatever it is," and then afterwards I went through and scanned and removed the ones which were obviously, um, the same generation or beneath, as it were.

34. 6:37

    So here, one of the things we're trying to optimize is finding a prompt which will get the agent to efficiently discount, uh, yeah, non- non-ancestors.

35. 6:49

    Um, so this is a schema we have. Um, we've got a task input which is, which is relevant. We have some, some very basic, um, initial instructions, uh, for the agent.

36. 6:58

    We have some slightly more advanced instructions, so we can kind of see how the two perform, and then ultimately we're gonna try to optimize to find an even better prompt.

37. 7:10

    So, uh, the, the core of this is this, this Pydantic AI agent. Uh, it's actually just using a hard-coded model. It's a variable because in theory you can substitute it with the command line, but I don't think we need to today.

38. 7:23

    Um, the most interesting bit is that the, the way that we're doing this is effectively a structured output question. So this could be... This is equivalent to any other structured output.

39. 7:32

    Find the address in this email, um, find the invoice lines in this PDF, whatever it might be. But we're using the same idea of structured outputs to go and find this, this political relation.

40. 7:44

    So we're just, we're just substituting in, uh... Yeah, so we're just, we're just setting the output type to be list of, uh, political relations. Um,

41. 7:54

    we set the instructions from above, um, and then ultimately when we go and run the agent, all we're doing is taking the HTML, doing a little bit of processing with Beautiful Soup to strip-- to get out the text, so we don't have to take all of the, like, other crap going on in the HTML page, and then

42. 8:09

    we're passing it to the agent. And this works surprisingly, surprisingly well. If you go and, if you go and read through the, the cases, you will see that it is pretty damn accurate at finding all political relations.

43. 8:19

    Even the relatively dumb models do a pretty good job. The main place where they get confused is, are their relations political versus, like, otherwise, uh, like public figures and this idea of a relation being an ancestor seem to really confuse the models.

44. 8:37

    Um, you will see if you look in the rest of, uh, the files here, I'm sorry if the, if the file explorer is small, but, um, once you run it once, you'll have this MPs directory which contains, uh, a JSON file with, um, the MPs and the URLs and the, the re- the raw data, um, and most

45. 8:58

    importantly of all the pages, which is the, the HTML pages, as I say. That will just be downloaded and put into that directory the first time you run it.

46. 9:05

    Um, yeah. And then there's also this golden relations. So this is supposedly the exactly correct answer for each politician. So, um, uh, Stephen Kinnock has, uh, I mean, for anyone who's [REDACTED:origin] will know a, like, reasonably famous political name.

47. 9:24

    He has a bunch of political relations. Um, uh, you see here that again, it's included his wife, who was the Prime Minister of Denmark, even though, um... I, I'm seeing some question, questions there.

48. 9:37

    I believe that's true. Um, even though we, we said don't include relations. That's why I've used this technique of basically including those relations and then stripping them out based on what the relation is.

49. 9:48

    Um, et cetera, et cetera. And this, um, I believe this is pretty accurate. Truth is, we ran it, we just ran a similar script with the, with the, like, with Opus four point six to get that data, but I've, I've checked it quite a lot, and it appears to be pretty much correct.

50. 10:03

    And so in terms of the, the first step here is to run some evals against this and, um,

51. 10:11

    uh, demonstrate that it's the, the, like, performance relative to the golden dataset. But before we do that, I'm just gonna run it, um,

52. 10:21

    run a single te- case in the terminal and show what's going on, just so that you have some feeling for what we're doing here. So I think if I look in task here,

53. 10:33

    uh, yeah, if I go, um, UV run task, uh, two is the ID of the, the first politician for some reason, Steven Kinnock. We'll go and run the model and successfully find the two relations.

54. 10:49

    And in this case, we've prompted it and said, uh, "Don't include... Only include ancestors," so it's done that correctly, and it's excluded his wife. Um,

55. 10:58

    d- I mean, I guess people want to ki- w-would prefer to, to, like, run along with this and, and try this themselves. Is that, is that true? Okay. So you will-- To do that, you will need...

56. 11:09

    And, um, you-- Ideally, you'd have a Logfire account. That would allow you to do the managed variable stuff. You can use a free Logfire account. The, the free tier is extremely generous, so everything you're doing there is free.

57. 11:21

    The other thing you need is obviously an API key to connect to the models. You have two choices there. Either in the code, we will basically be using,

58. 11:30

    uh, Pydantic AI Gateway as a simple way of being able to access all of the models. So you just see gateway prefix in front of the model name. Uh, if you have your own OpenAI or Anthropic key and you wanna use that, just remove the gateway prefix and it will work.

59. 11:43

    Um, if you want to use Pydantic AI Gateway so that everything just works, I'm just gonna generate an API key in Logfire and share it with you all, hence that channel, and put a, put a limit on it and hope you don't-- no one goes and uses it all up before we finish the lesson, finish the, finish

60. 11:58

    the workshop. So, um, I'm in, in Logfire here. I'm gonna go into gateway. Um, if I had set this up more formally, we could basically invite you all, and you could all go and generate your own keys, and you'd all have your own limits.

61. 12:13

    I have not prepped that, and so I'm just gonna re- generate one key. Uh, with a thousand dollar limit and hope we-- that does the, does the job today.

62. 12:20

    Uh, and then delete it after, after the workshop. Um,

63. 12:32

    so I'm just gonna copy that into this channel here.

64. 12:39

    Um, so you should just be able to s- uh, export that environment variable, um, and then when you run your code, for example, this, this task here, you've rerun task two, um, you should get, get an output there.

65. 12:53

    Um, shall I wait a couple of minutes for everyone to... The other thing you'll need to do is set up Logfire for when you run, um, the evals and the, the simplest way of doing that is once you've run UV sync in this di- So you'll be in this directory, you'll run UV sync.

66. 13:11

    So note we're in the, we're in the sub-directory for this, this talk. We'll run UV sync,

67. 13:17

    um, and then to connect it to your, your project, you'll see here my project is called, uh, demo. So I'm gonna run UV, uh, run Logfire project use demo.

68. 13:32

    Demo being the name of the project that I wanna connect it to, and that will go and connect Logfire in this directory to the demo. So when I go and run stuff, it will output, uh, w- record, uh, traces to Logfire and use the managed variables from there.

69. 13:45

    Um, I'll wait a couple of minutes 'cause I'm sure people wanna get a bit set up and get that kind of, like, very simple hello world thing, thing working, um, and then we can move on.

70. 13:55

    Anyone got any, any questions while we're just getting that set up?

71. 13:59

    Yeah. What, what does gateway mean in the model?

72. 14:03

    Uh, so AI gateways, we have one, but we're by no means the only ones. Uh, the idea is, uh, you can basically make requests to most models through, through our gateway with one API key.

73. 14:15

    And in particular, we, we have observability. So if you're, if you're using the, the production version, you can see requests that your team are making or that your agents are making.

74. 14:24

    Um, we have things like caching and fallback, so if one model fails, we can fall back to another one, uh, et cetera, et cetera. But in this context, it just means you can have one API key and connect to Anthropic or OpenAI or Groq with a Q or, uh, Gemini models.

75. 14:41

    So gateway is your own product or something?

76. 14:43

    Yes. But as I say, you don't have to use it here, but it might just make things easier to, to get all of the models working.

77. 14:57

    I'll wait a couple of minutes, but any, any questions or should I go on?

78. 15:01

    What was the directory name? What was the directory name?

79. 15:03

    Sorry.

80. 15:04

    Directory name.

81. 15:06

    Uh, so it's, it's, uh, the, the repo itself, uh,

82. 15:11

    is, um, uh, GitHub Pydantic slash talks, and then the directory is twenty twenty-six oh four AI engineer.

83. 15:34

    It wants me to log-- to authenticate through, through a browser even though-

84. 15:38

    Have you logged in with Logfire before?

85. 15:40

    No.

86. 15:41

    Okay. So you'll need to do that so that you have the, um, machine key.

87. 15:49

    So I think it's UV run Logfire auth. Sorry, I, I've done that.

88. 16:51

    So I'm gonna keep going just to, to keep things moving, but like, unless anyone has any, any questions. So we now have the, the very basic case working. What we wanna do is run an eval to see how, um, uh, our model with this-- with our prompt is per-per-performing relative to our golden dataset.

89. 17:11

    So you will see if we go into evals here, we have, um,

90. 17:17

    uh... Where's the dataset defined? Sorry. Um, you'll see at the top, um, we have the load dataset, which is gonna basically create our, um, our dataset, which is the, the object that we can then call eval on.

91. 17:34

    Um, this is gonna load, um, the, the, um, the cases which, um-- and, and basically create one case for each of the, each of the MPs that we're, that we're gonna run.

92. 17:49

    Um, and then it's gonna create, yeah, g-go through and, uh, generate these cases and ultimately return this dataset, dataset object. And we're gonna register one custom evaluator here, which, um, in turn actually generates a whole bunch of metrics or assertions for, like, how accurate this, um, uh, this run is.

93. 18:10

    Um, I don't think... I mean, you can go and read through the logic. I'm not gonna go through each individual bit. I'm not claiming here that these are the perfect set of evals in terms of the actual evaluators.

94. 18:20

    But they're, they're a useful start. Um, the point is, if you go, if... I can-- you can look through the docs, but we have a bunch of pre-built evaluators like LLM as a Judge.

95. 18:30

    Um, and then we allow you to j- uh, define your own. And generally, defining your own is far better than LLM as a Judge, 'cause the LLM as a Judge is effectively the kind of, you know...

96. 18:39

    I don't know if it's politically acceptable, but the kind of lunatics running the asylum, and it can lead to, lead to slight issues. So, if you can have a like deterministic eval like this where we're comparing what the result is versus a golden dataset, that's much better.

97. 18:52

    Um, uh, and so if you look in main where we're gonna go and run this, ultimately main is really just a CLI. So we set up a bunch of CLI cases, and then in the case of eval, we run the run evaluation function.

98. 19:08

    If you look in here, ultimately what we're doing, print out a bit of stuff, and then we're just gonna call this evaluate function. We're using Pydantic AI's override functionality to override, uh, the prompt that is set by default with, um, with the custom prompt that we're using for that particular that we're evaluating, and a model that we're

99. 19:26

    evaluating. We're not actually gonna change model in this case. Um,

100. 19:32

     and so we're... Yeah, we're gonna... And we'll then go and call... So ultimately, we're calling dataset evaluate, which takes a function, which is the actual function we're gonna go and, um, perform the evaluation on.

101. 19:44

     And that will in parallel with a, uh, max concurrency of five, go and run, uh, all of our cases, um, which we will, you will see in a minute.

102. 19:55

     Um, and then we're gonna give it some name, um, so that we can look at it in Logfire.

103. 20:01

     Now, w- you will also see here we've set up Logfire at the, at the top here. If token present just means it would run without causing an error if you don't have the, the token set.

104. 20:10

     Um, we set an environment service name. We're not gonna print to the console because, uh, in these complex cases, it's actually very quite tricky to see what's going on in the terminal.

105. 20:19

     It doesn't really help to have Logfire kind of printing to the terminal. And then we've switched off scrubbing because occasionally something like the word password or auth or something will appear in an output, and we- we're happy to-- for that to go to Logfire.

106. 20:31

     And then we're gonna instrument Pydantic AI. We don't actually need it now, uh, so I won't instrument prints. But when we get to j- using GEPA instrument prints, just as a nice way of getting the output from GEPA into Logfire as well.

107. 20:43

     Um, and so if I look the... And the, the... As you may have realized, the help file has... Help.md has some useful examples of what we're gonna run. So we're gonna go and run, um, this script

108. 20:56

     here. So we're gonna UV run, uh, main.py. We're gonna call with the arguments eval. We'll split. So we're, we're splitting it based on just the test k- k- set that we're running, and then the prompt is the, the initial one, so the very simple prompt, um,

109. 21:12

     kind of one-liner description of what the agent, uh, should do rather than the more complex prompt.

110. 21:19

     And if we go and run this eval, you won't see that much in the terminal because we, we, uh, uh, switched off terminal, but you will see it running through the cases.

111. 21:29

     And if we go and look in Logfire, you can see, um, those cases running and their, their costs, et cetera. We can dive into what an individual case looks like.

112. 21:38

     So if you look here, we're running, um,

113. 21:42

     uh, the agent, and I guess that's probably quite small, but I will try and zoom in to let you see what's happening here. We have the system input. Again, that's the very...

114. 21:51

     the, um, simple instructions that we talked about. And then we have the input, which is the, the contents of the Wikipedia page as text. And then we have the final output, and in this particular case, it found no, no relations.

115. 22:03

     Um, that has now finished. And so as well as this view of the individual cases here, we can go into evals in Logfire, and you will see, um, this case here, and you will...

116. 22:19

     Sorry, this dataset, and then within the dataset, you will see that most recent case from one minute ago. And if we go in and look at that,

117. 22:28

     we should get a view of what's going on. So again, this is pretty small on the big screen, I guess, but it'll look much clearer on your screen. We can see, um, where, um, the cases of...

118. 22:39

     The mo- most, most relevant metric is, um, accuracy, where you'll see there are ones where it's correct and zeros where it's incorrect, and you'll see a f- a few values where it's lower, where the output from the golden dataset is slightly different to the output from, um, the, from the model that we're running here.

119. 23:00

     So you see here, this [REDACTED:gender] has a father who is a, um, something to do with UKIP, and it's, it's got it correctly his name, but it, there's a slight difference in the description.

120. 23:10

     Hence, it's like got a point nine score.

121. 23:14

     And you can see the overall performance of eighty-five percent. So eighty-five percent of the time, it got, it got all of the stuff, uh, right. That's not particularly good, but it's also not particularly interesting in its, um...

122. 23:27

     until we go and compare it to running with a, with a better prompt. So, um, how many people are up to kind of being able to run that, run that eval?

123. 23:36

     A few people. Okay. I'll wait a couple of minutes 'cause I think it's probably, uh, slightly more entertaining for us to... We, we... Yeah, we're twenty minutes in. We've got quite a lot of time.

124. 23:44

     I'm gonna wait a couple of minutes.

125. 23:46

     Can you show the command again, please?

126. 23:48

     Yeah. So it's, it's this command. It's, if you look in, um, help.md, we're basically running the, the eval command.

127. 23:57

     Can you move, please? Yeah. It's, it's taking some time.

128. 24:03

     Yeah.

129. 24:04

     How long does it take this eval? About seven minutes.

130. 24:07

     Running eval?

131. 24:08

     Yeah.

132. 24:09

     It took-

133. 24:10

     I think it's calculating relations nine percent, uh, seven minutes ago.

134. 24:15

     It w- ran really quickly for me. Um-

135. 24:17

     Me too. Yeah, it took thirty seconds.

136. 24:20

     I think it should take about thirty seconds, so I don't know whether

137. 24:24

     you're on a different internet or- Changed, uh, changed the model. Um-

138. 24:33

     Do you think

139. 24:34

     Ah, yeah, that will be much lower than I think we're using... Well, A, Anthropic was down this morning when I was trying to use it a lot, so it's being really, really slow.

140. 24:43

     And-

141. 24:44

     Yeah

142. 24:45

     ... GPT 4.1 is flying because not many people are using it, and it's doing a reasonable job. That's why I'm using one of those models. It's partly to, like, show the eval performance, but also bluntly to get the answers more quickly.

143. 24:55

     Yeah.

144. 25:16

     While people are, are getting there, I'm just gonna run the s- the, the next one, which is compare. If you go and look at what this is doing, it's basically just running the eval twice with the, with the, like, bad prompt and the good prompt.

145. 25:26

     And then we'll be able to, uh, compare those outputs in, in Logfire and kind of see where the differences are coming from. Um, and then we can go on to the full GEPA optimization step.

146. 25:38

     I'll just run the compare, which should be... take a little bit more time, but still, still hopefully relatively quick here with these models.

147. 25:47

     You see, we're just running the sixty-five, uh, test cases here rather than the full six hundred and fifty MPs as we'll use when we get to the full, um, optimization.

148. 26:03

     If you, if you accidentally run the same prompt twice, you'll just create a new

149. 26:08

     You'll create a new... Obviously, you'll have more spans in Logfire, and you'll get another, um, run in the Logfire Evals view. You can archive them when you've, when you've got confused by how many different ones you've got.

150. 26:19

     So if you come over here to Evals, come into that most recent case, you'll see I've now got a bunch. Uh, I could go and archive,

151. 26:27

     uh, so yeah, archive the older ones. Well, the other way around, just to keep things, uh, clear.

152. 26:35

     But if I take the, those... So I've now got two new, new runs here, uh, two new experiments, and I'm gonna select them and, um,

153. 26:46

     compare. And now we have a bunch of different, uh, metrics, um, uh, in particular. And you will see that, like, ultimately accuracy being kind of the most important one here, slightly higher, ninety-two percent versus eighty-seven percent for the, like, expert prompt versus the initial prompt.

154. 27:06

     And we have the, like, possibly slightly useless graph of different metrics and how they've performed. Um, with, with fewer, um, metrics or assertions, this can be more useful. Um, and again, what- what's probably most interesting, though, is to look into individual cases where the performance differs.

155. 27:24

     So you see here, um, with this MP,

156. 27:29

     uh, Joe White, that the initial prompt, um, found one match, um,

157. 27:37

     relation spouse, which obviously it should not have found because we're not looking for spouses, whereas the expert prompt correctly ignored that one. Um, and then if you look in this case, Laura Kyle, um, you will see that the, the expected output is no matches at all.

158. 27:54

     Both, um, of the, these most recent runs, they found a match, um, because they had some four-times great-grandfather who was the governor of Hudson Bay, which is not a politician, but is a, like, public figure.

159. 28:08

     So you can kind of start to see where, where the differences are coming from and the kinds of things you might want to do in the prompt to, to improve the performance.

160. 28:16

     That's ultimately the kind of data that GEPA is gonna use to, like, um, inform it, uh, generating a next, the next, like, Pareto frontier prompt for it to go and try.

161. 28:36

     I didn't manage to see the graphs

162. 28:39

     You, you will see the graphs only if you're comparing two cases. So if you open a single case, it doesn't show the graphs.

163. 28:46

     Two cases.

164. 28:48

     So I've run compare. Once I had run eval, I then ran compare with this c- uh, command here.

165. 28:53

     Yeah.

166. 28:55

     And then, um, here I've taken two cases.

167. 29:00

     Oh, that you selected there. Sorry. Okay. Perfect.

168. 29:03

     And then you hit compare, and then you see, uh, see the graphs, um, comparing the performance.

169. 29:09

     Oh, cool. It wasn't quite clear what you compared it to. Is that the expert one?

170. 29:23

     So there were... If you look in the, um, task.py, there are two prompts, uh, as a kind of starting point. So one is a very simple one-liner, and then the second is a bit more detailed description of the kinds of things you should be looking for.

171. 29:38

     Still not kind of optimized, but, like, what you might get if you, as a human, wrote out a decent prompt to ask the model, like, "Write me a better prompt."

172. 29:45

     So you can see we've, like, improved the performance from eighty-seven to, like, ninety-two percent with this model through doing that approximately. And we should, with GEPA, be able to get better.

173. 29:55

     We-- I think I got ninety-six percent when I was running it earlier. Um, by no means perfect, but this is, you know. But you... I, you know. There were examples of using GEPA to, like...

174. 30:05

     I suppose what, what people generally do is they either hold one variable constant, like we need the following quality, and now we wanna reduce time or reduce cost, or we need to, like, drive up the quality.

175. 30:14

     So there was an example from Shopify using GEPA. They were using-- They were basically looking at, um, Shopify sites and looking for things like whether they were fraudulent or wh- which tax category they fell into.

176. 30:27

     And they switched from basically just giving the entire website to GPT 5 and saying, "What is this?" To using an agent and using a Qwen model and GEPA to optimize the prompt.

177. 30:39

     And they got the price down from five million dollars a year to sixty, sixty, seventy-three thousand dollars a year. Um, and improve performance over time. So that's obviously also...

178. 30:50

     That's not just optimizing the prompt. That's like some mixture of optimizing the prompt and moving to agentic and yada yada.

179. 30:58

     But in this case, we're just trying to kind of improve performance whilst using the same relatively fast model for the sake of, for the sake of, uh, the demo.

180. 31:08

     Um, so now we've done the comparison, I guess it's time to look into... Well, anyone want me to wait a bit longer, or should I keep going?

181. 31:16

     I'll take the no one asking me to wait as a signal to keep going. So we now get to the kind of GEPA, GEPA phase of this. So I'm gonna re-enable that while I remember.

182. 31:27

     And then ultimately, if you look here where we are gonna run the, uh, optimize, um, case, you'll see we have this runOptimization function. runOptimization here.

183. 31:40

     It's gonna load in the, the, the, the training and the validation datasets, um, error if it's not there. We're then gonna go and create this GEPA adapter. So the GEPA adapter is, is effectively like...

184. 31:54

     GEPA is not... I- it's an amazing... It's a kind of, I guess, the state of the art now in agent optimization. Lakshya, who created it, is a first-year PhD student at Berkeley.

185. 32:04

     He is not perhaps the most experienced kind of engineering Python developer, and so GEPA doesn't do async that nicely. It's not as type safe as I would like it to be.

186. 32:13

     I'm, like, itching to go and either fix it for him or fork it bluntly, but I have so far, uh, not done so. But, um,

187. 32:22

     yeah. So, so GEPA has this concept of adapters, which are basically how you define the agent that is going to propose the next case. And so we have a, we have this function here to create the adapter, um, which in turn generates one of th- this adapter here, which, which we've defined, which subclasses, um, their adapter type.

188. 32:46

     It's a data, data class, so it takes a bunch of arguments. But ultimately, the bit that matters is build_proposer_agent, um, which is obviously their, their method name. And what we're gonna, what we're gonna do here is return a Pydantic AI agent, uh, in turn.

189. 33:02

     So we're using a, we're using a Pydantic AI agent to propose new cases to optimize a Pydantic AI agent. Um, and we're gonna, uh, use again GPT 4.1 for the sake of speed in this case.

190. 33:16

     There is a slightly annoying characteristic where GEPA is sync, not async. And so if you try and run, each time it runs the proposer agent, it runs it in a new, um,

191. 33:29

     async context. And so we have to go and make sure we create a new HTTP connection, HTTPX connection. Otherwise, we get a bunch of errors, hence this, like, slightly weird formulation here.

192. 33:38

     We have, again, a prompt. Obviously, you can get a bit, uh, going round in circles on optimizing the prompt of optimizing the prompt, but we're basically saying, uh, "You're an expert prompt engineer.

193. 33:48

     Improve the, um, system prompt of this agent," yada, yada, yada. "Here are the things to consider." Um, and then we're gonna, um, call the ultimate evaluate function, which is going to, um, generate this bunch of cases, generate a Pydantic AI eval, run that eval, and that is what we're gonna use to basically test the performance of our

194. 34:10

     new proposed prompt. This will probably make some more sense when I actually go and run it. Um, yeah. And then we,

195. 34:20

     uh, a-apply skills for the, for the failures, and then we, um, return this evaluation batch, which kind of is a summary of how it's performed, which is what GEPA will then use as to whether or not this is a new Pareto frontier or whether to, to propose a new, a new prompt.

196. 34:35

     Um, uh, and so I think the simplest thing to do is just go and run this. It will take a little bit longer, um, and it will cost a little bit more, but that's fine.

197. 34:45

     I'm gonna run it. If you run it at fifty, it basically dies before it gets anywhere. But if I, um,

198. 34:53

     run this with four hundred calls, um, you will see, um, GEPA starts to... has a s- has a similar kind of progress bar. And I've-- as I said, I've switched off, um, Logfire printing so that you don't see what's going on in the background here because it basically just becomes impossible to view.

199. 35:15

     But if you look inside Logfire, you will see already this optimization is going on. So you can see that we, um, evaluate, um, the, the agent, and then we call the proposer agent here.

200. 35:28

     So the proposer agent, if you look at this step here, this is what we just had, the system prompt I set for the proposer agent.

201. 35:35

     Um, and then the input, which is basically GEPA's description of the context for the proposer agent to enable it to come up with a new prompt. And then from there, the proposer agent is gonna propose a new system prompt.

202. 35:51

     Um, and we're gonna then evaluate that, see how it performs, and iterate towards, towards finding a better solution. So that's what's going on here. You see why I did the instrument print, um, is so that, um, we can get the, the print output from GEPA, um, nicely viewable inside Logfire.

203. 36:13

     There may be a neater way of doing this, but this worked for me this morning. Um, and it's gonna work through. You can see it's a little bit more expensive.

204. 36:21

     We've already spent two dollars. Um, uh, so I guess if everyone runs it, we will spend a few hundred dollars. Uh, OpenAI will no doubt appreciate it. Um, uh, and that's gonna work through.

205. 36:34

     But you see we're already, like, um, fifty percent of the way through the optimization, and it is proposing these new prompts and then, um, uh,

206. 36:44

     evaluating them and, and using the, the feedback from the evaluation to decide where to go next. And I would encourage you, if you have some time- To go and have a read through how the, these traces because they're quite interesting, they're quite illuminating in terms of how the, how the optimization is working.

207. 37:01

     I think one of the things to note is people love to say that anything to do with AI is incredibly sophisticated and, and complicated and advanced. This optimization technique, whilst the state-of-the-art, is not actually comp-- that groundbreaking.

208. 37:13

     Relative to the complexity of the model itself, it's a, like, relatively crude sense of, like, ask an agent to generate a new prompt. If it does better, take bits of that, put it into a new prompt, keep doing that until you either run out of time or get to some prescribed score.

209. 37:29

     Yeah.

210. 37:30

     A couple of questions. How big the batches are, and do you have a agent skill for doing that?

211. 37:37

     Uh-

212. 37:38

     Thanks

213. 37:38

     ... how big the batches are. So we have-

214. 37:41

     Does, does the agent process, um, the prompt that is in user prompt of the agent, how, how many evals does it run? How many prompts?

215. 37:53

     So I think you can see it here where it's running the evals. So it's running... I think it is running all of the cases in the test dataset.

216. 38:00

     All of them.

217. 38:01

     Um, but I th- I think I've seen it run fewer. So I think GEPA is clever enough to, like, alter the number of cases based on how they performed, but I honestly not, not, not that sure.

218. 38:11

     Not one different.

219. 38:12

     Yeah. It-- So i-in terms of skill, what was the, what was the question?

220. 38:16

     Just, I was thinking how to reproduce that to optimize some prompts in my agent workflows. I was thinking whether you had a synthesized premade skill that it, you know, just-

221. 38:28

     I mean, I, I think we w- we will make this much easier with open source. We will also make it something where you just plug in Logfire and it just happens in the background.

222. 38:37

     Um, but we also wanna make it super easy to do it locally or open source as well as, like, we don't just wanna say, "Oh, you must plug it into Logfire to get optimization."

223. 38:47

     Thank you.

224. 38:49

     Yeah.

225. 38:50

     Um, now you just set the max calls, right? Which is, I think, max calls of AI or-

226. 38:54

     Yeah.

227. 38:55

     Uh, is there a smarter way to do this? Like, if you don't improve for ten iterations, just stop or-

228. 39:00

     I'm sure there is. I, I haven't lo-looked into it. Like I say, I did this in a bit of a hurry. I, I, totally, and all, like, presumably, for the most part, you're saying reach this threshold or, as you say, like, run out of optimization or whatever it might be.

229. 39:13

     I, I don't quite understand how you see the, the prompt optimization.

230. 39:19

     So if you look here in the proposer agent run, uh, you will see

231. 39:26

     effectively what, um, what input-- This is the, this is the input to the agent that is, that is, um,

232. 39:35

     uh, proposing a new prompt, and it is being told, based on this information, propose a new prompt. And then if you, if you look, the, the, the principle of the, like, Pareto frontier is this con-contains components of previous prompts that have done well.

233. 39:49

     Uh, and effect-- You know, it's a bit like the-

234. 39:51

     It's not like a div. It's a whole new-

235. 39:54

     Yes. It is prope- proposing a whole new, new div.

236. 39:59

     Thank you.

237. 40:01

     So, so I think if you have multiple... So, so I think GEPA operates on the idea of a kind of object or dict of, of known keys. And so here we only have basically one-- a dict with one key value pair, eg, like system prompt value.

238. 40:15

     If you have multiple different keys, then you can optimize by basically combining the best values from multiple different keys. So you could have, like, you could imagine e- you could have, imagine model and system prompt or model and some set of tools or even, like, different lines from a system prompt that you're basically, uh, allowing it to...

239. 40:37

     The, the proposal would just be choosing different values from your set of different lines. That's the kind of standard way of doing this in, in DSPY, is like basically take all of my different examples and choose which ones to include in the prompt.

240. 40:50

     What is DSPY?

241. 40:52

     So DSPY is another agent framework, uh, somewhat similar to Pydantic AI. Um, it comes from a very, um, machine learning background. I find it hideous to use because it's not type safe, but the caliber of people using it is generally very high.

242. 41:07

     Um, and it, it has, it has had this concept of optimization for a long time. For a long time, it was basically the only agent framework that had this idea.

243. 41:14

     Now GEPA has come along and is, like, available within DSPY, but is not only available within DSPY.

244. 41:22

     Yeah. Uh, what's your personal take on GEPA and this type of algorithm? Like, uh, from your personal experience, have, have they worked well in practice? Have you whole, like, used more home builds, uh, optimizers?

245. 41:37

     Yeah, curious on, on this informal take.

246. 41:40

     So I think that in the case where you're trying to basically get a dumber model or a faster model or a cheaper model to be able to do some task, they make a lot of sense.

247. 41:51

     If you go and take the state-of-the-art models are like Opus Four point six and you ask it most questions, it will just... If it has all the information it needs, it will just go and get it right for the most part, and so the optimization is slightly less relevant.

248. 42:04

     I think one of the subtleties here is that the statistic is that ninety-eight percent of data is private. I don't know if that number is, is correct or not, but, like, let's say even if it's off by miles and it's fifty-fifty data.

249. 42:15

     When you have a private set of data where the models have not been trained on it, you have some massive internal spec for how you're supposed to operate as a bank, let's say.

250. 42:24

     Adding the right bits of context into the system prompt or into the instructions is incredibly valuable, and the optimization really matters. The problem we have when we're trying to give demos like this is by definition, there isn't private data that we can use.

251. 42:37

     Because if someone lets us use their private data, they're not like, "Oh, yeah, and can we go and talk about it at AI Engineering?" So I've got this, like, slightly weird example of MPs, but actually you'll see in cases, stuff I'm gonna show you in the middle in a, in a minute, the model actually just goes and,

252. 42:51

     like, figures out who the MP for some, some borough is, even if it hasn't looked at the data, because actually this is all public data that it knows. So I think that one of the subtleties is that optimization matters way more where you have large amounts of private data.

253. 43:04

     And honestly, we're still trying to figure this out as a company. One option for us is to use stuff that is public but new enough that it's not in the training data.

254. 43:11

     So one option would be like news, but then that could get slightly politically dicey if we were using current news as our, like, source of stuff to optimize on.

255. 43:21

     So another option is to use, like, what's going on in sports because whether you find it interesting or not, at least it's not controversial, and we would, you know, ask it which footballer has done well in the Premier League season over the last five matches, and that might be public data, but equivalent to private in the sense

256. 43:37

     that it does not exist within the model, but it's not a really good example. It's something we're trying to figure out.

257. 43:43

     Uh, yes. I have a question regarding, uh, GEPA. Does it,

258. 43:48

     um...

259. 43:51

     I think it's not working, your mic. But c- if you just speak up, I think we can keep going.

260. 43:55

     Yeah. I, I'm just trying to understand if there is a, um, a way where it prioritizes kind of optimizing and adjusting the system prompt rather than kind of append into it because I feel like that's a recurrent issue.

261. 44:07

     Yeah.

262. 44:07

     You find new edge cases, and you're, like, appending and adding more information.

263. 44:11

     Yep.

264. 44:11

     So does it strike that balance of, okay, this we append and this we adjust?

265. 44:15

     So I was using GPT 5 Mini this morning, and one of the problem... A, it was slower in general, but also it was just producing this enormous system prompt, which then meant that the agent was slower.

266. 44:25

     I don't know if it did better or not. I think it, it did roughly similarly well. Um, so that definitely is a problem. I think the solution to that is the thing I was talking about earlier where ultimately GEPA wants a key value store, key va- a, a dict of keys and, and values, and then instead of

267. 44:40

     basically saying, the, instead of the proposer agent being like, "Go just generate me a new system prompt," you're basically saying, "Choose a new subset of our list of different inputs available for like..."

268. 44:50

     You would go and basically split your system prompt up into like 200 sentences and say which is the best 20 sentences to use, and now by definition, if it can only choose 20 sentences, it's not gonna get verbose.

269. 45:01

     In this case, we're using a Pydantic AI agent to basically summon up a whole new system prompt, which is probably better at finding the edge cases but, but does result in a verbose prompt.

270. 45:15

     Um, so in fact, on that very point, it has now finished. It has given us this optimal prompt. It has achieved a performance score of 96.7%, so significantly above the 92 we got from our, like, best case before, um, and at the expense of a relatively verbose prompt.

271. 45:34

     Um, uh, so you can see where it's gone through in relative detail and a- in particular, it's banged on about, like, what relation to use, when to, when to include a relation, what sort of things count as a relation.

272. 45:47

     Um, uh, yeah. Um, okay. So, so that, that, that is, is GEPA working. Uh, the other bit I was gonna talk about is the managed variables. So I will now switch onto the managed variables, but before we go onto that, any other questions on this bit of, of the, the talk or on GEPA?

273. 46:10

     Yep.

274. 46:11

     This is just probably me hard of hearing, but like, you, you got the validation scores compared to like a golden set, right?

275. 46:17

     Do you want to take that?

276. 46:19

     The, the validation scores comparing to like a golden set, right?

277. 46:22

     Yep.

278. 46:23

     Um, the golden set in this repository. It is, it is here, right?

279. 46:29

     Yep.

280. 46:29

     Oh, okay. Okay. So-

281. 46:30

     Yeah. So if you look in, um, if you look in the, uh, cases directory, you will see golden relations, and there's a ya... there's a JSON file with the, the supposedly golden set.

282. 46:42

     Now, I'm sure it's not a hundred percent perfect. As I say, we generated it with Opus 4.6 using a similar, similar script. But like, um...

283. 46:51

     I just had a silly thought. Like, if you didn't have that golden set, then things get quite difficult, doesn't it?

284. 46:58

     Yeah. Yeah. So evals are much easier if you have some, like, golden reference against what it's supposed to be doing. Uh,

285. 47:04

     in general, what people end up doing is they have some subset of data that's been, like, human annotated that works. But you can also have cases where, for example, if you have an agent that's writing code, you go run the code and check whether or not the code fails, for example.

286. 47:19

     Um, or if you can have a, like, feedback... If you can have a, a full loop of like, uh, basically returning that data to the original source and seeing how accurate it is, something like that.

287. 47:30

     Working out what your, what your judge is, is always, is the hard bit of evals.

288. 47:34

     But like, just for the, your example where the, where if you're running the code, that would be like a, like, like it either runs or doesn't, right? It's not like a...

289. 47:45

     How do you do a percentage then? I, I'm a little bit confused there.

290. 47:48

     Uh, well, you could check, for example, whether or not it was gonna generate invalid code or use libraries that weren't available. That would be a, like, first step of, like, how performance is going.

291. 47:58

     But yes. I mean, this is the, what this is the hard bit of evals is like what is, what is right? And, and as models get more intelligent, figuring out what right looks like is harder and harder.

292. 48:09

     If you... I mean, the ultimate case, right, is like you have an agent whose job it is to persuade people to stop smoking. The ultimate eval is wait forty years and see when they died.

293. 48:17

     But obviously you can't have an eval where you wait forty years to see if they died. So your eval is probably like does it include the, like, the correct words?

294. 48:24

     Does it not include some bad words that you definitely don't want it to suggest? Um, like, you know, take up a cigar instead, right? So text does not include cigar would be one of your evals perhaps.

295. 48:34

     And not... Like, that sounds dumb, but that is what an awful lot of evals end up being.

296. 48:44

     Can I ask a question?

297. 48:44

     Go ahead.

298. 48:44

     Can I... Yeah, sorry. Um, two, two, two questions. Uh, one is a very short one. Um, with Pydantic AI, you, you kind of break up the structure a lot of the time in the prompt.

299. 48:53

     Does this just jam everything together and you just get one prompt at the back at the end?

300. 48:58

     So you always have one, one set of instructions.

301. 49:01

     Yeah.

302. 49:01

     Um, so it is building one set of instructions.

303. 49:04

     Right. Okay. That's fine. Um, so the, the, the real question was, uh, you know, variance is huge with this. I mean, you showed your results. I ran it.

304. 49:13

     Yep.

305. 49:13

     I got different results. Is there a way of like having this, having the variants be part of the actual optimization?

306. 49:20

     Uh, I wouldn't say that. So, so there's a... When you run evals, I'm just looking for where we call run, you can basically set the number of times you run each case, and the best way of reducing variance is to run them lots and lots of times.

307. 49:32

     W- I know there's a bunch of hedge funds who are spending about $20,000 a night running their whole set of evals, uh, to see if they can improve over time, and if you have your own GPUs, you're like, "Well, it's sort of free to go and run them, so we should run them more and more times.

308. 49:45

     We should run everything a hun- hundred times and look in the morning." It's definitely one of the challenges.

309. 49:49

     And then do you like create a cost function or do you have like you can point it at like, "I want you to optimize like both accuracy and minimize variance."

310. 49:59

     Like, do you write out your own cost function then within-

311. 50:01

     Yeah. So, so if you look in the, in the adapter code here, you will see... I mean, have a look through it, but like where am I looking? Um, evaluate is returning this like, um, evaluation batch type, which where you can basically include whatever score, uh, scores is, uh, a list of floats, right?

312. 50:22

     So ultimately, you can return whatever floats you, you know, you want in there.

313. 50:27

     Okay.

314. 50:29

     And just, just in case people are, are wondering, trajectories is effectively similar to traces. It is the like sequence of different steps that agents went through to get to a particular point.

315. 50:40

     Um, so, um, how do you handle systematic errors? Like I ran the optimization and it got a high value of accuracy, 96%, but it got wrong aunts and uncles.

316. 50:54

     So the final prompt excludes explicitly aunts and uncles, even though they are in the golden relation. So do you suggest running... And like Claude Code spotted this immediately.

317. 51:07

     Yeah.

318. 51:08

     So maybe one could have a harness where there is another check at the end.

319. 51:12

     I, I think what it's doing is it's using a small set of test cases and then a bigger set of evaluation cases and so the, the subset of cases that

320. 51:21

     the optimizer agent has ex- gets exposed to maybe doesn't include aun- uncle and aunt, and so it just made it simpler to exclude it. So one of the other things you end up needing is like a kind of, uh, at least two x the, the amount of data that covers all of the different space so that you

321. 51:37

     can have one full set of data that you train on and one set of full data that you evaluate on. This is like classic problems of machine learning of like how do we go about like we need an awful lot of data to build something that's really reliably, you know, but, yeah, really reliable because ultimately, you know,

322. 51:55

     you're already getting to the point with the agent where you're kind of encouraging it to overfit, and I suspect the uncle and aunt thing is it overfitting.

323. 52:01

     Okay, thanks.

324. 52:04

     There's a question behind you.

325. 52:06

     This is probably a very silly question, but, but surely the, the prompts are... Can I assume that the prompt only works with the particular model that you tr- you would-

326. 52:17

     That is, that is people's take is that you would need to run this all over again if you changed your model.

327. 52:22

     That's gonna happen all the time, isn't it?

328. 52:24

     Yep, and that's one of the reasons that evals are hard, and that's one of the reasons that most people don't run evals or-- and don't run optimization. They write out a decent prompt.

329. 52:32

     They ask their coding agent of choice, "Does this prompt look good?" If it says yes, they kind of eyeball it, and then they put it into prod, and they worry about other things because probably the model that comes out in a month's time is gonna go and supersede whatever optimization you did in some cases.

330. 52:47

     Now, if you are a private equity firm who have, uh, two hundred million invoices from across your portfolio that you wanna go and analyze, it's worth going and optimizing for a particular version of Qwen 3.5 Light Mini to go and do that job instead of just throwing GPT 5 at it because the difference is like ten million

331. 53:06

     dollars and it costs you... You have your one analyst who earns half a million dollars, and you put them on it for three weeks, and you've solved it. So it like depends on the question that you're, that you're trying to solve.

332. 53:15

     But the ultimate example of this is the coding agents where they're like the trajectories are extremely sparse, as in there's an extremely wide range of things that they're doing, and for the most part, we are told that those companies don't have very many evals or don't have any evals at all.

333. 53:31

     If you go and ask Boris Cheney how does he work on, uh, Claude Code, he says, "Mostly vibes. Mostly we just like see what works and tweak it a bit."

334. 53:41

     So this is not a like-

335. 53:42

     Mm.

336. 53:42

     I'm not claiming this is a panacea in all cases, but there are definitely situations where being able to optimize your prompt or your agent choice is valuable. The other thing that we, we don't have a demo of here, but there's an awful lot more than just the prompt you can go and optimize.

337. 53:56

     So there's the model. There's things like the compaction strategy. There is how you register tools. There is things like including code mode in this, and the next step of what we want to allow is basically for you to go and optimize across the full range of things you can do within your agent to find the optimal choice,

338. 54:10

     where perhaps the particular model choice doesn't, doesn't matter so much.

339. 54:16

     A- a- and it just makes me think that w- we eval- you evaluate the prompt tip, but could we just switcheroo a bit and start evaluating different models or-

340. 54:26

     Yeah

341. 54:26

     ... that would be silly?

342. 54:27

     Yeah, so that would, that, that would be something you could totally go and do is try different models. I mean, given that there are... maybe you have ten models you wanna try, you probably just run all those ten models and see which one performs best and, and pick that one or which one optimizes for your particular subset

343. 54:40

     of like performance, price, time, et cetera. I think there's... There was another question behind you.

344. 54:45

     Yeah. Uh, thanks. I already got the mic. Um, so what you've shown is a task which was relatively narrow in what should be achieved, and I'm wondering, uh, how you would approach it if the task is more open-ended.

345. 54:58

     Uh, I think that my suspicion is that

346. 55:02

     that is a case where this optimization technique is harder to do because ba- ba-- like I say, the coding agent cha- case, you could imagine you end up with- Five thousand distinct tests for a coding agent, and you go and optimize as much as possible in those five thousand cases, and it turns out that they are like

347. 55:21

     drop in the ocean of different things people do, and now your agent is over-optimized for your thousand cases or five thousand cases. I think the bigger the, the breadth of the task your agent is performing, the harder it is to do this and the more you end up just having a, like,

348. 55:36

     clever agent.

349. 55:38

     Thanks.

350. 55:40

     Uh, the other thing to say is that my point earlier about where you have large amounts of private data, and so you have to put masses of context into the system prompt for like, u-understanding our specific domain, then choosing the right examples from that to put in, which basically cover the full data set is more and more

351. 55:58

     re-relevant. Did you have another question?

352. 56:05

     Thank you. So if I understand correctly, um, if you're a company and y-you would use this GEPA tech optimization technique, and if you had more money and maybe more expertise, you could decide to fine-tune a model, so they are kind of competing-

353. 56:24

     Yeah

354. 56:24

     ... uh, strategies, correct?

355. 56:27

     Yeah. And in answer to your pre-- the, the previous question about, oh, but isn't it, isn't it made obsolete by the next model? The main reason that the big model labs say don't bother fine-tuning is that there you really are spending tens of thousands of dollars to go and fine-tune a model, and actually, for the most part,

356. 56:42

     improve the harness, wait for the next model, like show a nicer loading icon to your users is probably better than fine-tuning. But that misses the cases, in particular in places like finance, where they have enormous numbers of runs, where they really do care about that optimization, and fine-tuning applies.

357. 56:58

     Thanks.

358. 57:01

     So I'm gonna go on to talk about, um, managed variables. Uh, as I say-- said earlier, this is a, this is a somewhat, um, uh, it's not gonna be a particularly sophisticated answer I've got here, but I, I'm gonna kind of show the example just, just to, to explain how it works.

359. 57:17

     So we have here a very simple FastAPI web server. Um, you will see that it has ultimately two endpoints. One returns some HTML, um, and the other one is a form submission, um, endpoint.

360. 57:32

     And if I go and run this, um, um...

361. 57:39

     Where have I had it running before? Here we are. It was in fact running. So if I-- If you wanna run it, you'll probably wanna use this command here to run it.

362. 57:49

     I, I'll put that into Slack as well in case anyone wants to run it.

363. 57:56

     And if you run this web server, you will-- Oh, not-- You'll see Logfire, which is not so interesting yet, but you will also see the server running locally. And this is just a very simple, basically form where you can ask questions about the same bunch of MP data.

364. 58:12

     Um, and, um, I can go and ask a question like,

365. 58:18

     we'll see how well this perf-performs. [REDACTED:location], where I live.

366. 58:27

     It will run for quite a long time because what it, what it's got in the background is basically one tool which lets it grep through, through the mark-- the, the, the HTML pages.

367. 58:35

     And it has replied relatively quickly on this occasion. And, uh, if I load this here, you will see the HTTP request coming in in Logfire, and then you will see the agent run, and it, um, it ran a rather, the, the, you know, a good, good sequel-- sorry, good regex.

368. 58:55

     F-got back the, the data and was able to correctly identify Andy Slaughter as the MP for [REDACTED:location], where I live. Um, all very well. But if you look in how this code is defined,

369. 59:08

     we have here this Logfire variable. And so this allows us to basically go and change the behavior of this agent without redeploying. Um, now, obviously, redeploying in this particular case where I'm running locally is very, very simple, but where I've got some big production CI stack to go and run and deployment takes hours, being able to go

370. 59:28

     and change variables in production or in staging very quickly is extremely valuable. So the way that, that Logfire variables work is that we have, have a type, here a Pydantic model, um, which takes, uh, has in turn three strings, the instructions, the model, um, and, um, the max tokens, which is an integer.

371. 59:47

     And then we've given it a default value, um, here, um, some very simple prompt, and then we've given it, we've, we've chosen the model, and we've chosen the temperature.

372. 59:56

     And what I have run here is I've run UV run web, and then there's a, there's a, basically there's a, like, CLI command for this, which just pushes the variable.

373. 1:00:07

     Ultimately, we're just calling Logfire push variables in this case. When I run that, I'll get asked, do I wanna overwrite this variable, which I don't. But if you, if you're running it for the first time, you, you won't be asked, and you'll, you'll create a new variable.

374. 1:00:20

     Now, it's worth saying to do this, we need to make the, the, the experience of using these variables slightly easier in Logfire. At the moment, they're not configured with the Logfire, um, projects use command that you did before.

375. 1:00:33

     So you need to go and set, uh, a separate, um, API key to use variables. That will be solved soon, but if you wanna do it today, the way to do that is to go into Settings, uh, API keys within Logfire.

376. 1:00:46

     You see I've created, uh, I can create-- I could show you the form. I can create a new API key, and I just need to give it the, the three variables permissions.

377. 1:00:55

     Well, or, or whichever ones you want. And now you can go and, like, u-- and we're ultimately using that API key to push the variables, uh, from local development, uh, fro-from the code definition through into Logfire and then pulling them to, to update the values.

378. 1:01:10

     And so because I have pushed that, if I go into, uh, managed variables here, you will see I have two variables set up. Um, this MP search config is the one we were looking at.

379. 1:01:22

     You see we have those three fields that we've set, um, and we have, um, some history of some updates.

380. 1:01:31

     But we can... In particular, what we can do in targeting is we can basically define what percentage of calls are using which of our different values. So this is very much like, uh, A/B testing.

381. 1:01:42

     In fact, under the hood, our managed variables use the open feature, open standard for, for doing this. So in theory, you can connect to, to Logfire with anything that speaks open feature.

382. 1:01:54

     Um, but yeah, so at the moment we're using the code default. So you saw when I ran this now, uh, I got back this answer, answer of Andy Slaughter.

383. 1:02:03

     But now let's go and update the value here. Um, I wanna go and edit,

384. 1:02:10

     and I'm gonna leave those values the same, and I'm gonna say, uh, reply in French to make it very clear what's happened. Save that.

385. 1:02:21

     And I'm gonna go into targeting, and I'm gonna dial up what u- the use of latest to a hundred percent. So it should now use,

386. 1:02:31

     uh, that latest value. And now if I, um, come in here and I ask, for example, the same question again, uh, and I stop the server, which was silly.

387. 1:02:43

     Um, but you can imagine the server is still running, and it's gonna pull the new variable each time that, that, that function is called. And if the demo gods are with me, and everything is hooked up correctly,

388. 1:02:58

     uh, it has indeed gone and replied in French. So we've managed to... We've updated the server without redeploying. Obviously, I did in fact restart the server, but I could, I could change it to not.

389. 1:03:07

     So just to prove that that works, let me come back over here, edit this again, and say reply in German.

390. 1:03:17

     And oh, that's very annoying. Uh. And now if I ask the same question again,

391. 1:03:28

     that variable defines the system prompt. It should be updated.

392. 1:03:37

     It's taking its sweet time, but it is indeed replying in German. But the nice thing here is, coming back to your point about, like, is it just a system prompt, we can set up our variables to not just be simple text values but be, for example, a Pydantic model, which allows us to edit multiple different fields.

393. 1:03:52

     So I can switch here from Gateway Anthropic, uh, Sonnet four point five to OpenAI, uh, GPT

394. 1:04:01

     four point one. I'll get rid of that 'cause that's gonna confuse me.

395. 1:04:10

     Uh, save that. Four point one. And the point is that we, we can update multiple different values. So if I instead ask y- ask it, uh...

396. 1:04:23

     Hopefully, it won't bother going off and running. It said it's now ChatGPT rather than Anthropic. So we can use this to, to change variables and ultimately go and, like, experiment with what new prompts or what new models or temperature or whatever it might be, uh, how they behave in production.

397. 1:04:38

     Now, of course, the ultimate aim here is to be able to take this whole system, wire it up, and instead of us having to, to basically

398. 1:04:47

     tweak, uh, man- uh, the managed variables manually, that we basically get, like, self-driving for managed variables. So given some set of evals, it can basically go and perform, uh, optimize towards, like, a hill climb towards some peak without us having to do anything.

399. 1:05:01

     That's the feature that we're, that we're working on, but I can kind of prototype what that might look like here. So let me just reload this.

400. 1:05:10

     You will see in managed variables, I actually have two variables here, and the other one is, uh, MP relations instructions, which is instructions for the, the, the task agent we were running before.

401. 1:05:20

     Um, and you will see we don't have any values in here yet. But what we have in our code, uh, in the web server here, is we have registered...

402. 1:05:32

     So we registered with this main agent two different tools. One was the MP search, which it was using before to find out who the MP for [REDACTED:location] was. But it's also got this extract political relations, which is basically running the same extraction logic and then,

403. 1:05:49

     uh, should be... I'm just looking through it. Yeah. Ultimately, it's calling that same extract relations function that we were optimizing earlier. So we can now use this agent. This agent can be asked, "Who are the political relations of a politician?"

404. 1:06:02

     And with a bit of luck, it will go and use this tool to go and do the calculation. At the moment, it's gonna use the simple done prompt, but the point is we can go and alter that prompt based on the optimized value that we got, um, and thereby basically apply our new state-of-the-art prompt without having to

405. 1:06:17

     redeploy. So if I come over here, first thing I'm gonna do actually is dial that, that previous variable back down to zero 'cause we don't want... We want it to be told correctly what to do.

406. 1:06:31

     Um, so I'm gonna put that to zero.

407. 1:06:34

     Um, and just to prove that something here is working, I'm going to say, um, uh, who are the political relations of...

408. 1:06:51

     What was he called? Steven Kinnock, who... 'Cause I know he does have some. I'm gonna go run this case,

409. 1:06:59

     and we can go and look in Logfire in a minute and see what it's done. But hopefully, it has used that tool, uh, as the correct way of identifying who the, who the relations are.

410. 1:07:07

     So yeah. So it's found them correctly, but more importantly, if we look in, uh, sorry. If we look in Logfire here,

411. 1:07:15

     um, you'll see that most recent call here. Um, and you can see that the, that the search, uh, agent run Uh, called the extract political relations tool with the input Steven Kinnock, which is correct.

412. 1:07:33

     Um, and so it's then, uh, gone and correctly output the value. But if you look inside the tool call, we then ran the other agent, uh, nested. Um, so it ran, and it correctly extracted the right data.

413. 1:07:46

     And so we're, we're now using our agent, and we can now basically, uh, click the value and improve the prompt now. And so I can come over here, and to be fair, I'm not gonna have a particularly good way of showing you that it's working better because it seems to be working pretty well, unless I can find

414. 1:08:00

     an example where it, where it wasn't. So let me actually try... I've got a bit of time. Let me try and see if I can find a, an eval case where the dumb prompt was doing badly.

415. 1:08:11

     Oh, we've got loads of runs now. Um,

416. 1:08:18

     sorry, let me... Okay, I want this case here, and I want

417. 1:08:23

     to compare these two. And so if you look,

418. 1:08:33

     so for example, this person here, uh, Paul Maski, it was perha- like, getting it wrong. The, the dumb prompt was, um... No, wrong way around.

419. 1:08:47

     Um, I wanna find a case where it's done badly where I can get it to do better. Okay, let's use Andrew Gwynne, where the correct answer is no, um, political relations, but, but the other, but the two, uh, prompts both returned suggested that they did have relations, and I'm sure that's because if we look, it'll be because,

420. 1:09:08

     yeah, they're a sports commentator, their, their father, which the, the dumb prompt wrongly in- suggested was a political relation. And so if I go and run

421. 1:09:19

     this, now with a bit of luck, this is using the dumb or the, the poor prompt, and so it should, um, find those relations. And then conversely, if I update it to the correct prompt, it should find...

422. 1:09:31

     Yeah, so it's, it's i- identified the sports commentator as a political relation. And now without redeploying, I can come over to manage variables, update the prompt that we're using.

423. 1:09:42

     I think I may have a mistake here. Let me... You didn't see me just go and tweak the code.

424. 1:09:49

     Um, it's gonna tweak the pro... tweak the code to use this version of the task where we are using the instructions manage variable. Um,

425. 1:10:00

     let me restart the server. Um, and now without redeploying, I promise you, uh, I'm gonna go and create a new case here, and instead of the, the simple prompt, I'm gonna put in the best one we got back from, uh,

426. 1:10:17

     running GEPA. Uh, save that. Uh, um, and without further ado, without redeploying, I promise, we go and run that again, and it'll almost certainly go wrong 'cause that's what things do, but we can hope.

427. 1:10:42

     And it's gone wrong. It's again identified it, unfortunately. Um, but, uh, hopefully you get the, you get the principle here. Yes, there's a question.

428. 1:10:52

     This web interface where you, where you search and get a response from the, uh, the model, the... What do, what do you call this?

429. 1:11:00

     Yeah.

430. 1:11:00

     This, this thing that you submitted to. Is there a name for it?

431. 1:11:02

     Uh, well, the form and then the agent who, that I'm submitting it.

432. 1:11:04

     Yeah. Um, do, uh, do you have any thoughts about, about, like, collecting feedback from, from users themselves? Like, like NPS and, and like what's good or bad?

433. 1:11:13

     Yeah.

434. 1:11:14

     Like, what's-

435. 1:11:15

     And my number one piece of feedback is no one collects them, and so there's a, there's a, there's a... Well, maybe there's Mike, but...

436. 1:11:21

     You ready? Uh, sure.

437. 1:11:25

     Um, um, so-

438. 1:11:28

     Or so, or, or you could be like nefarious like myself and just click n- bad even though-

439. 1:11:33

     So, so I'm, uh-

440. 1:11:34

     I wouldn't do that

441. 1:11:35

     ... my, my experience is that if you're gonna do it, you do only thumbs up, thumbs down. Um, we have an annotation system that I'm, I'm not demoing today where you can basically record annotations against prompts.

442. 1:11:44

     That is another place to build your golden dataset from or at least a starting point for your golden dataset. Um, the, the best thing you can possibly do is something which is implicit in the user's exper-- the user's interaction.

443. 1:11:59

     So if you have something more advanced than this simple, um, text prompt, but you have a, you have a chat, the single best way of getting, o- of evaluating the performance of a given question-response pair is to look at what the user did next.

444. 1:12:13

     'Cause if the user says, "You idiot, try again," you can get pretty strong evidence that they've, the, the, the, that the agent behaved badly. And if the user was like, "Thanks, that's great," or goes away, you assume it's right.

445. 1:12:22

     And if they, they probably don't... It's probably not as extreme as, "You idiot, try again," but it's probably like, "No, I mean X." And so I mean, that's, that's how Google works, right?

446. 1:12:30

     Like, the way they identify how good a website is back in the old days when we used, uh, Google, was to see how long you spent on a site.

447. 1:12:37

     If you came straight back and searched for, clicked on another link, they assumed the page was bad.

448. 1:12:41

     And this form here, is it a chat, or is it... If you, if you-

449. 1:12:44

     This is just... I honestly just asked Claude, "Go build me a, go build me a single-

450. 1:12:48

     Okay

451. 1:12:48

     ..." page chat."

452. 1:12:48

     Like-

453. 1:12:49

     We have, um, we have, uh, an integration with Vercel AI pro- AI protocol so that you can basically build a full chat interface on a Pydantic AI agent in, in, like, a few lines of code, supposedly, that I would try and demo if I had a few bit more time.

454. 1:13:05

     But you, this is just me trying to have, like, the simplest example to run.

455. 1:13:10

     In fact, go on, I'll have one try at doing it now-

456. 1:13:12

     And, and, and-

457. 1:13:12

     ... since that one didn't work

458. 1:13:13

     ... just... And a, a chat would mean j- s- it, it, that it kept its context somewhat, right?

459. 1:13:18

     Yeah. Yeah, so if I do, if I, um, get rid of this single run, and instead I do, uh... What, what's the agent called?

460. 1:13:31

     Uh, relations agent dot, uh, to web. Um, and then I go and run, uh, let me kill that web server to avoid confusion. Um.

461. 1:13:50

     Why is that not running? Uh, maybe I do need to call .run.

462. 1:14:01

     Uh, maybe I'm forgetting exactly how to call it, but, like, the, the principle is that with basically one line of code here, if I remember what it is, you can basically turn that into a proper chat interface.

463. 1:14:09

     Um. Uh, let me just see if the docs will tell me what it is quickly.

464. 1:14:17

     Um. Uh. So I, I define the app, which gives you a Starlette app, and then I run, uh, uv run uv acorn task,

465. 1:14:42

     uh, app. And there's a typo. And that should give me a, like, nice interface now to go and ask questions where I can, where I can inter- where I can say, like, um...

466. 1:15:03

     Uh, maybe I... I don't know if that's gonna work, but we will try it.

467. 1:15:11

     But the point is now I would, um, uh, you would get back a final result of it, would, it would go and, like, you could use it like an agent, and then you could ask follow-up questions.

468. 1:15:19

     I mean, this agent is designed to always reply with structured data, so it's not very chatty, but you get the idea. Um, I'm gonna stop there. Um, thank you very much.

469. 1:15:28

     Happy to answer any questions afterwards. [audience applauding] Sorry, did you have a question?

470. 1:15:40

     Yeah.

471. 1:15:41

     Go on.

472. 1:15:42

     Thank you. Can you hear me? Yeah.

473. 1:15:44

     Yep.

474. 1:15:44

     Uh, do you have, um, core use cases internally where you use your stack, but, like, agents that you've built? Because I hear, like, a lot of, uh, "Look how many tokens are we burning," and, "Look my evals."

475. 1:15:56

     But have you seen some, like you mentioned the private equity firms earlier, like, these are classic old problems classifiers. Do you have use cases that internally you're really impressed by agents or what the team is using internally to ship faster or create more value for either you or your clients?

476. 1:16:16

     So we have the-- there's a, there's an agent inside Logfire to basically do your free text search, and it converts to SQL, and we optimize that agent a fair bit.

477. 1:16:25

     Um, we don't care particularly about token count in that case because it's not high enough volume. We do care about generating good SQL. The reason we can't use it is that it very often ends up with, like, private data within the SQL.

478. 1:16:36

     As in, someone will say, "Find me results for invoice one, two, three, four, five," and we don't wanna go and show people that. It's one of the reasons we can't use that example in, in demos.

479. 1:16:44

     Um, I think there was another question. Does that answer the question?

480. 1:16:47

     Yeah.

481. 1:16:51

     Right. So in Logfire, is it possible to kind of obfuscate or encrypt the input and the output and just evaluate against a golden set? Let's say if you have set up an application to work in a field with sensitive information like medical or...

482. 1:17:08

     So you can choose not to send the system prompt or anything to the agent and just basically record the... The output could be, for example, the, the performance, like good or bad, and then you have the metrics.

483. 1:17:19

     You can choose not to send the, the raw data. It's obviously less valuable. My somewhat biased answer would be you can obviously enterprise self-host Logfire, and then you can have it inside your VPC.

484. 1:17:29

     Um, that's probably what people do. But yes, you can do it without, without recording the, like, actual input and output.

485. 1:17:36

     Great.

486. 1:17:36

     What, what people do in those cases, so for example, the big, like, Lagora and Harvey, they're like legal tech companies where they are very strictly not allowed to exfiltrate the, the data because it is their clients', clients' very private data, is they record categorical performance.

487. 1:17:53

     So like good, bad, whatever. Like, you could have like fifty grades of how it performed, but, like, and you can exfiltrate, exfiltrate that data without actually exfiltrating, like, any, um, generated content where you might include private data.

488. 1:18:08

     Great.

489. 1:18:09

     Any other questions? Yeah, there's one here.

490. 1:18:21

     Thank you. This is, uh, a tiny bit of an internal question. I noticed that, uh, you're using data classes throughout.

491. 1:18:27

     Yep.

492. 1:18:28

     Uh, is that for speed reasons or any other reasons?

493. 1:18:31

     I think they're just the canonical thing in, in Python, and so they're like... If you're not trying to perform validation, I think that they're generally the kind of, yeah, the canonical choice over Pydantic models.

494. 1:18:41

     Uh, and it doesn't make any difference. The performance of any of, any of these cases, the Pydantic validation or the data class construction time will be shrinking, you know, point zero, zero, zero one percent of the performance.

495. 1:18:52

     Thank you.

496. 1:18:58

     Thanks for the presentation. I had a quick question. So with the, um, prompt optimization, obviously that's like a type of context engineering, right?

497. 1:19:06

     Yeah.

498. 1:19:07

     So I was wondering, have you done evals where you've combined that with, um, essentially summary? 'Cause obviously when you, you-- there's con-context degradation even though the context windows are so big.

499. 1:19:16

     Obviously after like five or so it'll still be less than, or after twenty, you know, calls it'll be less than very quickly. So combining prompt optimization with just spinning up new, um, uh, LLM calls and, uh, n-new context, uh, restarts as opposed to, say-

500. 1:19:32

     Yeah, the-

501. 1:19:32

     ... uh, compaction, which still reduces as well.

502. 1:19:34

     The compaction strategy is definitely one of the things you wanna go and optimize in a more complex case.

503. 1:19:38

     Yep.

504. 1:19:39

     I think here we optimize for something that everyone can grok in a, in a, in a session rather than the most complex use cases. But yes, definitely that's, that's relevant.

505. 1:19:47

     So for example, the, the Shopify example of like, uh, how they were analyzing these sites. GPT 5 you could just give it the entire content of the website and be like, "Hey, go figure out whether this is fraudulent."

506. 1:20:00

     Using a Qwen model, you couldn't do that. You needed to do something more agentic where it, like, performed a bunch of, like, queries to look for certain terms. And so that is a, like, variation on context engineering to basically allow a smaller model to perform the same task and be more deterministic.

507. 1:20:16

     Great. Thank you.

508. 1:20:19

     I have to stop. Apparently, I have to stop. Thank you everyone. [audience applauding] [upbeat music]