AI Engineer World's Fair 2025
Human seeded Evals — Samuel Colvin, Pydantic
Read the talk
Human seeded Evals — Samuel Colvin, Pydantic
Reliable AI applications need explicit contracts: typed outputs, validation that feeds corrective retries, typed tool dependencies, and traces that explain why a run failed.
From a talk by Samuel Colvin
Before you start: Familiarity with Python type annotations, Pydantic models, and basic LLM tool calling will help you follow the examples.
How do you refactor an application whose shape is still changing?
How do you build a reliable, scalable application when you do not yet know what it will become? Adding generative AI does not remove that problem. It can make it harder, whether a model is helping write the application or running inside it. Despite the recording’s title, Samuel Colvin explicitly leaves evals out of this shortened presentation, drawing instead on Building AI Applications the Pydantic Way. The practical focus is building and changing AI applications safely.
Type safety makes repeated refactoring less precarious. An AI application is likely to change shape several times as its requirements become clearer. A type checker can reveal the places where a change breaks an existing contract. A coding agent such as Cursor can also run those checks and use the diagnostics to correct its work.
Colvin contrasts this with what he considers inadequate type safety in LangChain and LangGraph. That is his assessment of their guarantees in the presentation, not a claim that they have no typing facilities: current LangGraph documentation includes typed state schemas. The distinction that matters for the coming examples is whether types remain connected across an agent’s output, its tools, and the dependencies supplied when it runs. MCP is mentioned as a possible topic, but the demonstrations concentrate on these contracts and observability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The agent loop needs an exit
Colvin starts with an agent diagram he attributes to Barry Zhang’s February AI Engineer talk in New York and Anthropic, describing the definition as one adopted by Pydantic, OpenAI, and Google’s ADK. The operational version is straightforward: an agent has an environment, tools that may access it, and a system prompt describing its job. It then repeats a cycle: call the model, receive tool actions, execute those tools, update state, and call the model again.
The displayed loop has a revealing omission: there is no exit condition. Repeating model and tool calls is easy to describe; deciding that the work is complete requires another contract. This criticism concerns the displayed code, rather than Anthropic’s broader guidance—Building effective agents also discusses explicit stopping conditions.
Three possible completion signals appear in the presentation:
| Completion signal | How the run ends |
|---|---|
| Plain text | The model responds with text instead of requesting a tool. |
| Final result tool | The model calls a designated tool that completes the run. |
| Structured output | The model supplies an output in the required structure. |
Colvin points to OpenAI and Google models as supporting the structured-output option. Choosing among these signals is part of defining the agent’s behavior, not an incidental detail of the loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract a Person, then make validation matter
The first Pydantic AI example extracts a three-field Person model from an unstructured sentence. Person inherits from Pydantic’s BaseModel, so the requested result has a concrete schema. The sentence and schema are small enough to fit on screen, but the same approach can target larger documents and nested models. Colvin suggests a large PDF, then qualifies the idea: document size still has to fit the model’s available context. Running the example prints a Pydantic model.
So far, this does not require repeated reasoning or tool use. The model makes a final result tool call, Pydantic AI validates the returned data, and the application receives the result. The successful extraction is a single-call case; the framework’s loop becomes useful when validation rejects an answer.
Colvin adds a field validator requiring the date of birth to precede 1900. The sentence gives the year as 87, leaving the century ambiguous. The model generally interprets that as 1987, which is plausible from the sentence but invalid under the new rule. A compact Python version of that constraint is:
python
from datetime import date
from pydantic import BaseModel, field_validator
class Person(BaseModel):
name: str
dob: date
@field_validator('dob')
@classmethod
def born_before_1900(cls, value: date) -> date:
if value >= date(1900, 1, 1):
raise ValueError('Date of birth must be before 1900')
return value
This reduced schema isolates the rule: a date can be well formed and still violate an application constraint.
The validation failure becomes input to the next model call. Pydantic AI returns the error to the model and asks it to try again, giving it information it did not use in the first answer. Colvin deliberately makes this example unfair by withholding the intended century from the field description. In a real application, he would document the nineteenth-century requirement on the date-of-birth field. Even with clear instructions, however, capable models can produce invalid output; returning the specific error gives them a way to repair it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect the retry, then follow the output type
The instrumented extraction run makes two calls to Gemini Flash, with the second call producing an output that passes validation. Logfire exposes the exchange inside the agent run, making the correction visible rather than leaving it hidden behind the final Person.
The trace shows the sequence:
- The user supplies the description.
- The model calls the final result tool with a birth year of 1987.
- The tool response returns the validation error and asks for a correction.
- The model calls the final result tool again with a date that satisfies the constraint.
The validator does more than protect the caller from an unacceptable value: its error supplies actionable feedback for another iteration.
The same output declaration also carries information into the editor. With output_type=Person, the agent’s output generic is Person, and result.output is statically typed accordingly. Pydantic validation separately ensures that a successfully returned output is a Person instance at runtime. These checks act at different boundaries: validation checks the model’s data; static typing checks how application code uses the resulting object.
Accessing result.output.name is valid. Accessing result.output.first_name produces an editor diagnostic because Person has no such field. That is a static typing error, not the date validator rejecting model output. The agent also has a dependency generic, which extends this connection into registered tools.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry dependency types through memory tools
The next example introduces long-term memory with two tools: one records memories and another retrieves them. Both are registered with @agent.tool. Unlike the extraction example, this agent uses the default string output, while deps_type specifies the application dependencies available to its tools.
The dependency contract connects the declaration to the tool and then to the call site:
| Location | Type relationship |
|---|---|
| Agent definition | deps_type=Deps declares the dependency type. |
| Tool’s first parameter | RunContext[Deps] carries that same type. |
| Inside the tool | context.deps exposes a Deps instance and typed attributes. |
| Agent invocation | deps must match the declared dependency type. |
Changing the tool’s context parameterization to int creates a typing error. Passing the wrong dependency type when running the agent does too. The benefit comes from keeping these declarations connected, so changing one part of the application reveals incompatible uses elsewhere.
Colvin says that, as far as he knows, no other agent framework works as hard at this level of type safety. He also acknowledges the cost: the framework has to do substantial work to preserve the relationships, and users have a little more setup to write. The intended payoff is easier refactoring as the application grows and changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A valid tool call can still search for the wrong thing
Running the memory example introduces a different class of problem. Colvin suspects the local Postgres instance is not running, checks Docker, and attempts to start the database. The example then fails, prompting another attempt. He turns to Logfire to understand what happened, explaining that the failure was not staged.
The trace shows that recording worked: the record-memory tool stored User's name is Samuel, and the run finished. The failed retrieval called the search tool with your name. The implementation uses a simple contains search based on PostgreSQL ILIKE; the phrase your name is not present in User's name is Samuel. The tool can accept a correctly typed string while that string is still a poor query.
ILIKE performs case-insensitive pattern matching; surrounding % wildcards give it contains behavior. The relevant PostgreSQL pattern-matching behavior can be isolated with the same memory and failed query:
sql
SELECT
'User''s name is Samuel' ILIKE '%your name%' AS matches;
-- matches: false
The mismatch is about the words in the query, not capitalization. Inspecting the actual tool argument explains a failure that the final response alone would not explain.
On the later attempt, the model searches for name. That substring is present, so the tool retrieves the stored memory:
| Stored memory | Search argument | Retrieval |
|---|---|---|
User's name is Samuel | your name | No match |
User's name is Samuel | name | Memory returned |
The stored fact did not need to change. The model-generated search argument changed, and the lookup succeeded. The completed trace shows the search, returned memory, and final answer together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The trace also exposes time and cost
With User's name is Samuel returned, the memory example completes successfully. The trace also shows how long individual calls took. Colvin closes by pointing out pricing at both levels: individual spans and the aggregate agent trace. The same view used to explain the retrieval failure therefore also exposes the operational cost of the calls that produced the answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The PyCon US listing for Colvin's May 2025 presentation on Pydantic AI and Logfire.
Further reading
- Building effective agentsArticle
Anthropic's guide to agent loops, environmental feedback and explicit stopping conditions.
The semantics of LIKE, ILIKE and wildcard patterns used for substring searches.
Updates since the talk
Current examples for typed outputs, structured-output modes, validation and corrective retries.
How to pass typed application dependencies into tools through RunContext.
Current documentation for inspecting LLM exchanges, token usage and costs in Logfire.
Read the complete timestamped transcript
- 0:00
[upbeat music] I'll assume, given the time we have, that you kind of get who I am and what Pydantic is to some extent.
- 0:19
So I will, I will move on. This is, this-- I'm using the talk I gave at PyCon, so, uh, it-it was building, uh, AI applications the Pydantic way, which is, uh, I guess, somewhat akin.
- 0:30
As I say, I'm not gonna be able to get to the eval stuff today, um, but I, I can talk about these two. So everything is changing really fast, as we all get told repeatedly in ever more hysterical terms.
- 0:42
Actually, some things are not changing. We still wanna build reliable, scalable applications, and that is still hard. Arguably, it's actually harder with GenAI than it was before, whether that is using GenAI to build it or using GenAI within your application.
- 0:55
Um, so what we're trying to talk about here is, is, uh, some techniques that you can use to build applications, uh, quickly but also somewhat more safely than, than you might, you might do if you, uh, otherwise.
- 1:08
Um, I'm a strong believer that type safety is one of the really important parts of that. Not just for in production avoiding bugs, but if you-- no one starts off building an AI application knowing what it's gonna look like, so you're gonna have to end up refactoring your application multiple times.
- 1:23
If you build your application in a type-safe way, if you use frameworks that allow it to be type-safe, you can refactor it with confidence much more quickly. If you're using a coding agent like Cursor, it can use type safety or running type, type checking to get, get-- basically mark its own homework and work out what it's doing
- 1:38
right in a way that you can't do if you use a framework like LangChain or LangGraph, who either through decision or inability decided not to build something that's type-safe.
- 1:46
Um, I'll talk a bit about MCP if I have a moment. Um, and I won't talk about how eval's fit in 'cause I don't have time. Um, so before-- Look, nothing I'm gonna say here on what an agent is, is controversial.
- 1:59
This is, um, reasonably well accepted now by, by most people as a definition of an agent. This, uh,
- 2:10
image here is from Barry Zhang's talk at AI Engineer in New York in February. This is his definition or the, the, the Anthropic definition of what an agent is now being copied by us, by OpenAI, by Google's ADK.
- 2:23
I think generally the accepted definition of an agent. This, although very neat, doesn't really make any sense to me. This, however, does make sense. So what, what they say is that an agent is effectively something that has, has an environment.
- 2:37
There are some tools which may have access to the environment. There is some system prompt that describes to it what it's supposed to do. And then you have a while loop where you call the LLM, get back some actions to run in the tool, run the tools, that updates the state, uh, and then you call the LLM
- 2:52
again. There is, however, even in his, whatever it is, six-line pseudocode, a bug, which is there is no exit from that loop. And sure enough, that points towards a real problem, which is that there isn't-- it is not clear when you should exit that loop, that, that loop.
- 3:07
And so there are, there are a number of different things you can do. You can say when the LLM returns plain text rather than calling a tool, that is the end.
- 3:14
Or you can have certain tools which are kind of, uh, what we call final result tools, which basically trigger the end, end of the run. Or if you have models like OpenAI or Google which have structured output types, you can use that to end your run.
- 3:28
But it i-it's not necessarily trivial to work out when the end is.
- 3:32
So enough pseudocode. Let me run a, a real minimal example of Pydantic AI. So this is, uh, a very simple, um, Pydantic-based model with three fields. Uh, and then we're gonna use Pydantic AI to extract structured data that fits that, that person, uh, schema from unstructured data, this sentence here.
- 3:50
Now, here, obviously, to fit this into on screen, uh, this is,
- 3:56
um, a very, very simple example, but this could be a PDF, uh, tens of megabytes. Well, probably not tens of megabytes necessarily in context, but like definitely an, you know, enormous document.
- 4:06
And, and this schema is very simple, but this could be an in-incredibly complex nested schema. Models are still able to do it. And sure enough, if we go and run this example, and the gods of the internet are with us, sure enough, we get the, the Pydantic model, uh, printed out.
- 4:20
So, but the-- Some of you will notice that this example is simple enough that we don't actually need an agent or this loop. We're doing one shot. We make one call to the LLM, returns the structured data.
- 4:30
We call under the hood. We call a final result tool. Pydantic AI performs validation, and we get back the data. But we don't have to change that example very much to start seeing the value of the agentic loop.
- 4:42
So here I'm being a little bit unfair to the, to the model. I've added a, a field validator to my person model, which says that date of birth needs to be before [REDACTED:dob].
- 4:53
And obviously, the, the actual definition here is abstract. Uh, uh, is, uh, doesn't define what year we're going to be, um... W-well, sorry, w-which century we're talking about. You would obviously-- The model will, for the most part, assume eighty-seven is [REDACTED:dob].
- 5:09
We'll then get a validation error when you do the validation, and that's where the agentic bit kicks in because we will take those validation errors and return them to the model basically as a definite and say, "Please try again," as I'll show you in a moment.
- 5:21
And the model is then able to use the information from the validation error to, to try again. Obviously, if you were trying to do this case in production, you would add a, uh, a doc string to the DOB field saying it must be in the 19th century.
- 5:32
But there are definitely cases where models, even the smartest models, don't, uh, pass validation. And being able to use this trick of returning validation errors, um, to the model is a, is a very effective way of fixing a lot of the simplest use cases.
- 5:46
So if we run this, you see we had two calls to Gemini here. And if I come and open... The other thing you'll see in this example is we instrumented, um, this code with, uh, with Logfire, our observability platform, so we can actually go in and see exactly what happened.
- 6:04
So you'll see our agent run. We had two- Uh, two calls to the model, in this case Gemini Flash. And if we go and look at the,
- 6:13
the exchange, you can see what's happened here. So
- 6:17
we... I'll just try and make it big enough that you can see it. We first of all had the user prompt, the description, it called the final result tool, as you might expect, the date of birth being [REDACTED:dob].
- 6:27
Uh, we then responded, the tool response was validation error incorrect, please try... And then we, we add on the end, please fix the error and try again. And sure enough, it was then able to return, uh, correctly call the final result tool with the right date of birth and succeed.
- 6:44
Cool. I've got five minutes. I feel like I'm in one of those, uh, see how fast I can go. Uh, I'm on the wrong window, am I?
- 6:53
I am. Here we are. Um, I think the other thing that's worth, worth saying here, even if I don't have that much time, is if you take a look at the exa- this example, I talked about type safety.
- 7:06
If you look, the way that we're doing this under the hood, agent, because of the output type, is generic in, in this case person. And so we can act- when we access, uh, result or output, both in typing terms, it's an instance of person, and, uh, runtime, we're guaranteed from the Pydantic validation that it will really be
- 7:23
an instance of person. So if I access here .name, all will be well. If I access first name, uh, we suddenly get a validation, we get a runtime, we get the, the nice error from typing saying this is a i- incorrect field.
- 7:37
So that's the kind of, the, the kind of very beginning of the value of, uh, static typing, uh, of, of our typing support. We go a lot further. You will have seen, or some of you might have noticed there's a second generic on agent, um, which is the deps type.
- 7:51
And so if you register tools with this agent, they, you, we can have type safe dependencies to tools, which I will show you in a moment.
- 7:59
Um, so what... The other thing you will, you will notice is missing from this example is any tools. So let's look at an example with tools. So if I open this example here, we have...
- 8:11
Well, this is, uh, an example of memory, long-term memory in particular, where we're using a tool to record memories and then, uh, another tool to be able to retrieve memories.
- 8:19
So you'll see we have these two tools here, record memory and retrieve memory. Tools are, are set up by registering them with the agent.tool decora- uh, decorator. But this is where the typing, as I say, gets more complex.
- 8:32
Now, you will see that we've set deps type when we've defined the agent, and so our agent is now generic in that deps type. The return type is string because that's the default.
- 8:41
And so we... When we call a tool decorator, we have to set the first argument to be this run context to parameterize with our deps type. And so when we access context.deps, that is an instance of our, of our deps data class that you see there.
- 8:55
And if we access one of its attributes, we get the actual type. And if we change this to be
- 9:01
int, let's say, suddenly we get an error saying we've used the wrong, the wrong type. So we get this guarantee that the type here matches the type here, matches the attributes you can access here.
- 9:12
And then when we come to run the agent, we need our deps to be an instance of that deps type. So again, if we put- gave it the wrong type, we would get a val- a, a typing error saying, "You're using the wrong type."
- 9:22
And as far as I know, we're what? The only agent framework that works this hard to be type safe. And, uh, it is quite a lot of work on our side.
- 9:29
I'll be honest, there's a little bit of work on your side as well as in it's not necessarily as trivial to set up, but it makes it incredibly easy to go and refactor your code.
- 9:37
Um, and yeah, you, we run this here, and we give it the, the... I'm pretty sure I don't have Postgres running.
- 9:45
Uh, do I have Docker running? I don't know if I have time to make that work.
- 9:50
I will... That's Docker running. I'll just try and run this very quickly. Uh, Docker run.
- 9:58
Hopefully, that is enough. If I now come and run this example,
- 10:03
what you will see is it successfully failed. Great. [laughing] Um, I will try one more time and see if I get lucky. I don't know quite what was going on there.
- 10:19
Ah, and I have no idea. Well, we can look in Logfire and see what happened, uh, to make it fail. I promise you I hadn't set that up to fail the first time to demonstrate the value of observability, but maybe it can help here.
- 10:29
So if you look, um, this first time, we, um, our first agent run, you'll see that we re- used the, uh, the tool call, uh, record memory. The user's name is Samuel.
- 10:46
Um, and then it, it returned finished. And then the second time, uh, you can see that the, when it did retrieve memory, where it called the, that tool, the parameter or the, the argument it gave was your name, um, which was not, does, is not contained within the, the query the previous time.
- 11:05
We're just doing a very simple I like here. So your name is not a substring of user's name is Samuel, and so that's why it, why it failed that time.
- 11:15
Um, so this has turned into a very useful example of where, where Logfire can help. And if we look at the, that second time,
- 11:23
you'll see user's name is Samuel, and then when it, when it ran the agent, it just asked for name. Name is obviously a substring of, of the user's name is Samuel, and so it was able, it got the response.
- 11:35
User's name is Samuel and therefore succeeded. The other thing we get here is like, obviously, we get this tracing information, so we can see how long each of those calls took.
- 11:44
Um, and we also get pricing on both aggregate across the whole of the trace and individual spans. Um, I am told that I am running out of time, so thank you very much. [outro music]