Human seeded Evals — Samuel Colvin, Pydantic

Talk transcript

1. 0:00

   [upbeat music] I'll assume, given the time we have, that you kind of get who I am and what Pydantic is to some extent.

2. 0:19

   So I will, I will move on. This is, this-- I'm using the talk I gave at PyCon, so, uh, it-it was building, uh, AI applications the Pydantic way, which is, uh, I guess, somewhat akin.

3. 0:30

   As I say, I'm not gonna be able to get to the eval stuff today, um, but I, I can talk about these two. So everything is changing really fast, as we all get told repeatedly in ever more hysterical terms.

4. 0:42

   Actually, some things are not changing. We still wanna build reliable, scalable applications, and that is still hard. Arguably, it's actually harder with GenAI than it was before, whether that is using GenAI to build it or using GenAI within your application.

5. 0:55

   Um, so what we're trying to talk about here is, is, uh, some techniques that you can use to build applications, uh, quickly but also somewhat more safely than, than you might, you might do if you, uh, otherwise.

6. 1:08

   Um, I'm a strong believer that type safety is one of the really important parts of that. Not just for in production avoiding bugs, but if you-- no one starts off building an AI application knowing what it's gonna look like, so you're gonna have to end up refactoring your application multiple times.

7. 1:23

   If you build your application in a type-safe way, if you use frameworks that allow it to be type-safe, you can refactor it with confidence much more quickly. If you're using a coding agent like Cursor, it can use type safety or running type, type checking to get, get-- basically mark its own homework and work out what it's doing

8. 1:38

   right in a way that you can't do if you use a framework like LangChain or LangGraph, who either through decision or inability decided not to build something that's type-safe.

9. 1:46

   Um, I'll talk a bit about MCP if I have a moment. Um, and I won't talk about how eval's fit in 'cause I don't have time. Um, so before-- Look, nothing I'm gonna say here on what an agent is, is controversial.

10. 1:59

    This is, um, reasonably well accepted now by, by most people as a definition of an agent. This, uh,

11. 2:10

    image here is from Barry Zhang's talk at AI Engineer in New York in February. This is his definition or the, the, the Anthropic definition of what an agent is now being copied by us, by OpenAI, by Google's ADK.

12. 2:23

    I think generally the accepted definition of an agent. This, although very neat, doesn't really make any sense to me. This, however, does make sense. So what, what they say is that an agent is effectively something that has, has an environment.

13. 2:37

    There are some tools which may have access to the environment. There is some system prompt that describes to it what it's supposed to do. And then you have a while loop where you call the LLM, get back some actions to run in the tool, run the tools, that updates the state, uh, and then you call the LLM

14. 2:52

    again. There is, however, even in his, whatever it is, six-line pseudocode, a bug, which is there is no exit from that loop. And sure enough, that points towards a real problem, which is that there isn't-- it is not clear when you should exit that loop, that, that loop.

15. 3:07

    And so there are, there are a number of different things you can do. You can say when the LLM returns plain text rather than calling a tool, that is the end.

16. 3:14

    Or you can have certain tools which are kind of, uh, what we call final result tools, which basically trigger the end, end of the run. Or if you have models like OpenAI or Google which have structured output types, you can use that to end your run.

17. 3:28

    But it i-it's not necessarily trivial to work out when the end is.

18. 3:32

    So enough pseudocode. Let me run a, a real minimal example of Pydantic AI. So this is, uh, a very simple, um, Pydantic-based model with three fields. Uh, and then we're gonna use Pydantic AI to extract structured data that fits that, that person, uh, schema from unstructured data, this sentence here.

19. 3:50

    Now, here, obviously, to fit this into on screen, uh, this is,

20. 3:56

    um, a very, very simple example, but this could be a PDF, uh, tens of megabytes. Well, probably not tens of megabytes necessarily in context, but like definitely an, you know, enormous document.

21. 4:06

    And, and this schema is very simple, but this could be an in-incredibly complex nested schema. Models are still able to do it. And sure enough, if we go and run this example, and the gods of the internet are with us, sure enough, we get the, the Pydantic model, uh, printed out.

22. 4:20

    So, but the-- Some of you will notice that this example is simple enough that we don't actually need an agent or this loop. We're doing one shot. We make one call to the LLM, returns the structured data.

23. 4:30

    We call under the hood. We call a final result tool. Pydantic AI performs validation, and we get back the data. But we don't have to change that example very much to start seeing the value of the agentic loop.

24. 4:42

    So here I'm being a little bit unfair to the, to the model. I've added a, a field validator to my person model, which says that date of birth needs to be before [REDACTED:dob].

25. 4:53

    And obviously, the, the actual definition here is abstract. Uh, uh, is, uh, doesn't define what year we're going to be, um... W-well, sorry, w-which century we're talking about. You would obviously-- The model will, for the most part, assume eighty-seven is [REDACTED:dob].

26. 5:09

    We'll then get a validation error when you do the validation, and that's where the agentic bit kicks in because we will take those validation errors and return them to the model basically as a definite and say, "Please try again," as I'll show you in a moment.

27. 5:21

    And the model is then able to use the information from the validation error to, to try again. Obviously, if you were trying to do this case in production, you would add a, uh, a doc string to the DOB field saying it must be in the 19th century.

28. 5:32

    But there are definitely cases where models, even the smartest models, don't, uh, pass validation. And being able to use this trick of returning validation errors, um, to the model is a, is a very effective way of fixing a lot of the simplest use cases.

29. 5:46

    So if we run this, you see we had two calls to Gemini here. And if I come and open... The other thing you'll see in this example is we instrumented, um, this code with, uh, with Logfire, our observability platform, so we can actually go in and see exactly what happened.

30. 6:04

    So you'll see our agent run. We had two- Uh, two calls to the model, in this case Gemini Flash. And if we go and look at the,

31. 6:13

    the exchange, you can see what's happened here. So

32. 6:17

    we... I'll just try and make it big enough that you can see it. We first of all had the user prompt, the description, it called the final result tool, as you might expect, the date of birth being [REDACTED:dob].

33. 6:27

    Uh, we then responded, the tool response was validation error incorrect, please try... And then we, we add on the end, please fix the error and try again. And sure enough, it was then able to return, uh, correctly call the final result tool with the right date of birth and succeed.

34. 6:44

    Cool. I've got five minutes. I feel like I'm in one of those, uh, see how fast I can go. Uh, I'm on the wrong window, am I?

35. 6:53

    I am. Here we are. Um, I think the other thing that's worth, worth saying here, even if I don't have that much time, is if you take a look at the exa- this example, I talked about type safety.

36. 7:06

    If you look, the way that we're doing this under the hood, agent, because of the output type, is generic in, in this case person. And so we can act- when we access, uh, result or output, both in typing terms, it's an instance of person, and, uh, runtime, we're guaranteed from the Pydantic validation that it will really be

37. 7:23

    an instance of person. So if I access here .name, all will be well. If I access first name, uh, we suddenly get a validation, we get a runtime, we get the, the nice error from typing saying this is a i- incorrect field.

38. 7:37

    So that's the kind of, the, the kind of very beginning of the value of, uh, static typing, uh, of, of our typing support. We go a lot further. You will have seen, or some of you might have noticed there's a second generic on agent, um, which is the deps type.

39. 7:51

    And so if you register tools with this agent, they, you, we can have type safe dependencies to tools, which I will show you in a moment.

40. 7:59

    Um, so what... The other thing you will, you will notice is missing from this example is any tools. So let's look at an example with tools. So if I open this example here, we have...

41. 8:11

    Well, this is, uh, an example of memory, long-term memory in particular, where we're using a tool to record memories and then, uh, another tool to be able to retrieve memories.

42. 8:19

    So you'll see we have these two tools here, record memory and retrieve memory. Tools are, are set up by registering them with the agent.tool decora- uh, decorator. But this is where the typing, as I say, gets more complex.

43. 8:32

    Now, you will see that we've set deps type when we've defined the agent, and so our agent is now generic in that deps type. The return type is string because that's the default.

44. 8:41

    And so we... When we call a tool decorator, we have to set the first argument to be this run context to parameterize with our deps type. And so when we access context.deps, that is an instance of our, of our deps data class that you see there.

45. 8:55

    And if we access one of its attributes, we get the actual type. And if we change this to be

46. 9:01

    int, let's say, suddenly we get an error saying we've used the wrong, the wrong type. So we get this guarantee that the type here matches the type here, matches the attributes you can access here.

47. 9:12

    And then when we come to run the agent, we need our deps to be an instance of that deps type. So again, if we put- gave it the wrong type, we would get a val- a, a typing error saying, "You're using the wrong type."

48. 9:22

    And as far as I know, we're what? The only agent framework that works this hard to be type safe. And, uh, it is quite a lot of work on our side.

49. 9:29

    I'll be honest, there's a little bit of work on your side as well as in it's not necessarily as trivial to set up, but it makes it incredibly easy to go and refactor your code.

50. 9:37

    Um, and yeah, you, we run this here, and we give it the, the... I'm pretty sure I don't have Postgres running.

51. 9:45

    Uh, do I have Docker running? I don't know if I have time to make that work.

52. 9:50

    I will... That's Docker running. I'll just try and run this very quickly. Uh, Docker run.

53. 9:58

    Hopefully, that is enough. If I now come and run this example,

54. 10:03

    what you will see is it successfully failed. Great. [laughing] Um, I will try one more time and see if I get lucky. I don't know quite what was going on there.

55. 10:19

    Ah, and I have no idea. Well, we can look in Logfire and see what happened, uh, to make it fail. I promise you I hadn't set that up to fail the first time to demonstrate the value of observability, but maybe it can help here.

56. 10:29

    So if you look, um, this first time, we, um, our first agent run, you'll see that we re- used the, uh, the tool call, uh, record memory. The user's name is Samuel.

57. 10:46

    Um, and then it, it returned finished. And then the second time, uh, you can see that the, when it did retrieve memory, where it called the, that tool, the parameter or the, the argument it gave was your name, um, which was not, does, is not contained within the, the query the previous time.

58. 11:05

    We're just doing a very simple I like here. So your name is not a substring of user's name is Samuel, and so that's why it, why it failed that time.

59. 11:15

    Um, so this has turned into a very useful example of where, where Logfire can help. And if we look at the, that second time,

60. 11:23

    you'll see user's name is Samuel, and then when it, when it ran the agent, it just asked for name. Name is obviously a substring of, of the user's name is Samuel, and so it was able, it got the response.

61. 11:35

    User's name is Samuel and therefore succeeded. The other thing we get here is like, obviously, we get this tracing information, so we can see how long each of those calls took.

62. 11:44

    Um, and we also get pricing on both aggregate across the whole of the trace and individual spans. Um, I am told that I am running out of time, so thank you very much. [outro music]