Human seeded Evals — Samuel Colvin, Pydantic

How do you build a reliable, scalable application when you do not yet know what it will become? Adding generative AI does not remove that problem. It can make it harder, whether a model is helping write the application or running inside it. Despite the recording’s title, Samuel Colvin explicitly leaves evals out of this shortened presentation, drawing instead on Building AI Applications the Pydantic Way. The practical focus is building and changing AI applications safely.

Type safety makes repeated refactoring less precarious. An AI application is likely to change shape several times as its requirements become clearer. A type checker can reveal the places where a change breaks an existing contract. A coding agent such as Cursor can also run those checks and use the diagnostics to correct its work.

Colvin contrasts this with what he considers inadequate type safety in LangChain and LangGraph. That is his assessment of their guarantees in the presentation, not a claim that they have no typing facilities: current LangGraph documentation includes typed state schemas. The distinction that matters for the coming examples is whether types remain connected across an agent’s output, its tools, and the dependencies supplied when it runs. MCP is mentioned as a possible topic, but the demonstrations concentrate on these contracts and observability.

Colvin starts with an agent diagram he attributes to Barry Zhang’s February AI Engineer talk in New York and Anthropic, describing the definition as one adopted by Pydantic, OpenAI, and Google’s ADK. The operational version is straightforward: an agent has an environment, tools that may access it, and a system prompt describing its job. It then repeats a cycle: call the model, receive tool actions, execute those tools, update state, and call the model again.

The displayed loop has a revealing omission: there is no exit condition. Repeating model and tool calls is easy to describe; deciding that the work is complete requires another contract. This criticism concerns the displayed code, rather than Anthropic’s broader guidance—Building effective agents also discusses explicit stopping conditions.

Three possible completion signals appear in the presentation:

Colvin points to OpenAI and Google models as supporting the structured-output option. Choosing among these signals is part of defining the agent’s behavior, not an incidental detail of the loop.

The first Pydantic AI example extracts a three-field Person model from an unstructured sentence. Person inherits from Pydantic’s BaseModel, so the requested result has a concrete schema. The sentence and schema are small enough to fit on screen, but the same approach can target larger documents and nested models. Colvin suggests a large PDF, then qualifies the idea: document size still has to fit the model’s available context. Running the example prints a Pydantic model.

So far, this does not require repeated reasoning or tool use. The model makes a final result tool call, Pydantic AI validates the returned data, and the application receives the result. The successful extraction is a single-call case; the framework’s loop becomes useful when validation rejects an answer.

Colvin adds a field validator requiring the date of birth to precede 1900. The sentence gives the year as 87, leaving the century ambiguous. The model generally assumes a twentieth-century birth year, which is plausible from the sentence but invalid under the new rule. A compact Python version of that constraint is:

from datetime import date

from pydantic import BaseModel, field_validator


class Person(BaseModel):
    name: str
    dob: date

    @field_validator('dob')
    @classmethod
    def born_before_1900(cls, value: date) -> date:
        if value >= date(1900, 1, 1):
            raise ValueError('Date of birth must be before 1900')
        return value

This reduced schema isolates the rule: a date can be well formed and still violate an application constraint.

The validation failure becomes input to the next model call. Pydantic AI returns the error to the model and asks it to try again, giving it information it did not use in the first answer. Colvin deliberately makes this example unfair by withholding the intended century from the field description. In a real application, he would document the nineteenth-century requirement on the date-of-birth field. Even with clear instructions, however, capable models can produce invalid output; returning the specific error gives them a way to repair it.

The instrumented extraction run makes two calls to Gemini Flash, with the second call producing an output that passes validation. Logfire exposes the exchange inside the agent run, making the correction visible rather than leaving it hidden behind the final Person.

The trace shows the sequence:

1. The user supplies the description.
2. The model calls the final result tool with an initially invalid date of birth.
3. The tool response returns the validation error and asks for a correction.
4. The model calls the final result tool again with a date that satisfies the constraint.

The validator does more than protect the caller from an unacceptable value: its error supplies actionable feedback for another iteration.

The same output declaration also carries information into the editor. With output_type=Person, the agent’s output generic is Person, and result.output is statically typed accordingly. Pydantic validation separately ensures that a successfully returned output is a Person instance at runtime. These checks act at different boundaries: validation checks the model’s data; static typing checks how application code uses the resulting object.

Accessing result.output.name is valid. Accessing result.output.first_name produces an editor diagnostic because Person has no such field. That is a static typing error, not the date validator rejecting model output. The agent also has a dependency generic, which extends this connection into registered tools.

The next example introduces long-term memory with two tools: one records memories and another retrieves them. Both are registered with @agent.tool. Unlike the extraction example, this agent uses the default string output, while deps_type specifies the application dependencies available to its tools.

The dependency contract connects the declaration to the tool and then to the call site:

Changing the tool’s context parameterization to int creates a typing error. Passing the wrong dependency type when running the agent does too. The benefit comes from keeping these declarations connected, so changing one part of the application reveals incompatible uses elsewhere.

Colvin says that, as far as he knows, no other agent framework works as hard at this level of type safety. He also acknowledges the cost: the framework has to do substantial work to preserve the relationships, and users have a little more setup to write. The intended payoff is easier refactoring as the application grows and changes.

Running the memory example introduces a different class of problem. Colvin suspects the local Postgres instance is not running, checks Docker, and attempts to start the database. The example then fails, prompting another attempt. He turns to Logfire to understand what happened, explaining that the failure was not staged.

The trace shows that recording worked: the record-memory tool stored User's name is Samuel, and the run finished. The failed retrieval called the search tool with your name. The implementation uses a simple contains search based on PostgreSQL ILIKE; the phrase your name is not present in User's name is Samuel. The tool can accept a correctly typed string while that string is still a poor query.

ILIKE performs case-insensitive pattern matching; surrounding % wildcards give it contains behavior. The relevant PostgreSQL pattern-matching behavior can be isolated with the same memory and failed query:

SELECT
    'User''s name is Samuel' ILIKE '%your name%' AS matches;
-- matches: false

The mismatch is about the words in the query, not capitalization. Inspecting the actual tool argument explains a failure that the final response alone would not explain.

On the later attempt, the model searches for name. That substring is present, so the tool retrieves the stored memory:

The stored fact did not need to change. The model-generated search argument changed, and the lookup succeeded. The completed trace shows the search, returned memory, and final answer together.

With User's name is Samuel returned, the memory example completes successfully. The trace also shows how long individual calls took. Colvin closes by pointing out pricing at both levels: individual spans and the aggregate agent trace. The same view used to explain the retrieval failure therefore also exposes the operational cost of the calls that produced the answer.