Playground in Prod - Optimising Agents in Production Environments

Improving an agent in production requires two separate capabilities: a way to discover better behavior, and a way to deliver that behavior to the running application. Samuel Colvin brings these together using Pydantic AI, an agent framework, and Pydantic Logfire, an observability platform maintained alongside Pydantic Validation. Logfire builds on OpenTelemetry logs, metrics, and traces; Colvin treats AI observability as a feature of that broader system, with evaluations and managed configuration extending what ordinary tracing can do.

GEPA supplies the optimization mechanism. Its name comes from genetic Pareto optimization: retain successful candidates, combine useful components, and evaluate the resulting candidates. Colvin compares this to breeding successful racehorses. The object being optimized can be prompt text or a string containing JSON, so the search need not stop at wording. Managed variables supply the delivery mechanism: instead of remotely editing only a prompt, an application can consume an entire configuration object defined by a Pydantic model. The workshop connects these pieces manually; autonomous optimization inside the platform is still under development at the time of the recording.

The concrete task begins with a question from The Rest Is Politics: how many MPs come from political families? Colvin originally used Pydantic AI to inspect each MP’s Wikipedia article for political relatives. He tentatively recalls an answer of about 24%, but that original run was not evaluated. The workshop returns to the extraction task to measure its behavior and improve its instructions.

An automatically downloaded archive supplies the Wikipedia HTML, avoiding a fresh scrape for every experiment. The data model describes MPs and their relations, including a relative’s name, role, relationship, and optional party. For the dynasty question, the intended boundary is the parents’ generation and earlier: spouses, siblings, and children should not qualify merely because they are politicians.

That boundary is surprisingly difficult to express reliably. Once a model sees a request for relations, it tends to include spouses, children, or in-laws. Colvin’s original workaround was to extract relations broadly and filter out same-generation and younger relatives afterward. The new experiment asks whether better instructions can make the extractor apply the exclusions itself. It starts with a minimal prompt and a more detailed, human-style expert prompt, then searches for something better.

The agent’s output type is a list of political relations. This is the same basic pattern as extracting addresses from email or invoice lines from a PDF: define the result structure, prepare the source text, and ask the model to populate it. Beautiful Soup strips the archived HTML down to text before the agent receives it. The core Python shape is small:

from bs4 import BeautifulSoup
from pydantic import BaseModel
from pydantic_ai import Agent

class PoliticalRelation(BaseModel):
    name: str
    role: str
    relation: str
    party: str | None = None

async def extract_relations(
    html: str, model: str, instructions: str
) -> list[PoliticalRelation]:
    text = BeautifulSoup(html, "html.parser").get_text(" ", strip=True)
    agent = Agent(
        model,
        instructions=instructions,
        output_type=list[PoliticalRelation],
    )
    result = await agent.run(text)
    return result.output

A valid list does not guarantee valid membership. The difficult errors are semantic: whether a relative belongs to the eligible generation, and whether their public role actually counts as political.

The downloaded MP directory contains metadata, URLs, and HTML pages. A separate golden-relations file supplies reference answers. Stephen Kinnock illustrates why extraction and filtering must be distinguished: his reference data includes his wife, a former Danish prime minister, even though a spouse should be excluded from the dynasty calculation. Colvin generated the reference data with Opus 4.6 and checked it extensively himself; it is a useful reference set, not an exhaustively human-certified ground truth. In the single-case terminal demonstration, the instructed extractor finds two qualifying relations and excludes Kinnock’s wife.

Following the workshop requires model credentials and, for managed variables, a Logfire account. The model names use a gateway prefix to route through Pydantic AI Gateway. Colvin explains that removing that prefix lets participants use their own provider credentials instead. For the session, he creates a temporary shared gateway key with a spending limit and plans to delete it afterward; a formal team setup would give people individual keys and limits. Participants export the supplied environment variable, install dependencies with uv sync, and select a Logfire project.

The gateway is a routing and operations layer: one key can reach Anthropic, OpenAI, Groq, and Gemini, while request observability, caching, and fallback provide additional capabilities. It is convenient for the workshop, but not required. The workshop repository contains the 2026-04-ai-engineer directory. First-time Logfire users also need browser authentication to establish their machine credentials. A practical setup sequence is:

1. Enter 2026-04-ai-engineer and run uv sync.
2. Configure your model credentials.
3. Run uv run logfire auth if this machine is not authenticated.
4. Run uv run logfire projects use demo, replacing demo with your project name.

Project selection configures the telemetry destination; managed-variable access has an additional authorization step later in the workshop.

The evaluation dataset contains one case per MP. A custom evaluator compares the extracted relations with the reference and emits several metrics and assertions. Colvin prefers deterministic comparisons when the task permits them: an LLM judge introduces another model’s judgment into the measurement. These particular checks are a starting point, but they make individual disagreements inspectable.

The CLI dispatches to an evaluation function, which uses Pydantic AI’s override mechanism to substitute the candidate instructions and, optionally, a different model. dataset.evaluate receives the task function and runs cases with maximum concurrency set to five. Named experiments make the results identifiable in Logfire. Telemetry uses if-token-present so missing credentials do not prevent execution; console output is disabled to keep concurrent runs readable. Colvin also disables scrubbing for this public-data demonstration, where words such as password or auth might appear in source text. Pydantic AI instrumentation captures agent calls, and print instrumentation will later capture GEPA’s progress.

The initial run uses the test split and the simple prompt. Logfire connects each case to its cost, instructions, Wikipedia text, and structured output; the Evals view groups those cases into an experiment. Inspect the disagreement behind a score. One example identifies a father correctly but describes his UKIP-related role differently from the reference, earning partial credit rather than a clean pass or failure.

The initial run reports an 85% custom evaluation score. Although Colvin describes this conversationally as getting everything right that often, the published evaluator uses partial-credit relation matching and a soft F1-style score. It should therefore be read as an aggregate task score, not the percentage of completely correct cases. Its value becomes clearer when compared with another prompt on the same task.

Participants do not all see the same runtime. One reports a run lingering near the beginning after several minutes, while others finish quickly. Colvin considers model and network differences and reports Anthropic availability problems that morning. He selects GPT-4.1 for its fast responses and adequate extraction quality in this session, attributing some of its speed to lower demand.

The compare command evaluates both starting prompts. This comparison uses 65 test cases drawn from the corpus of 650 MPs. Repeated executions create new spans and experiment runs; older runs can be archived. Selecting two experiments in Logfire exposes their comparison view. In this paired run, the expert prompt scores about 92%, versus 87% for the initial prompt. That initial-prompt result is a separate execution from the earlier 85% baseline.

The first case shows a useful correction. In the second, both prompts disagree with the reference, which treats the historical governor as a public figure outside the task’s political category. Looking at these examples explains what the aggregate improvement consists of—and what remains unresolved. This is the kind of feedback GEPA will use to propose the next candidate.

Comparison graphs appear only after selecting two runs and choosing Compare. The two starting prompts live in task.py: one is a brief instruction, the other resembles a reasonable prompt a human might write or ask a coding assistant to improve. Neither is GEPA-optimized. Colvin recalls a stronger result from an earlier optimization run, but the next step is to watch that search happen.

Optimization can pursue different objectives: raise quality at a fixed model choice, or preserve sufficient quality while reducing latency or cost. Colvin describes a Shopify example that moved from supplying entire websites to GPT-5 toward an agent using a Qwen model and GEPA for fraud or tax-category analysis. He reports annual cost falling from $5 million to approximately $73,000, alongside improving performance. His account combines model, architecture, and prompt changes; it does not isolate the effect of prompt optimization or specify the workload and quality metric. The workshop takes the narrower route: keep the fast task model fixed and improve extraction quality.

The optimization entry point loads training and validation data, then constructs a GEPA adapter. This adapter is a dataclass subclass whose build_proposer_agent method returns a Pydantic AI agent. GPT-4.1 therefore plays two roles: one agent performs extraction, while another proposes better instructions for it. In the workshop integration, GEPA’s synchronous execution creates fresh asynchronous contexts for proposer calls. The adapter creates a fresh HTTPX connection to avoid errors from reusing a connection across those contexts. Colvin identifies this async boundary and weaker typing as integration friction in the version he is using.

The adapter closes the feedback loop:

1. Give the proposer instructions to improve the extraction prompt, together with GEPA’s evaluation context.
2. Generate a replacement prompt.
3. Run the extraction evaluations using that candidate.
4. Assign scores, including scores for failures.
5. Return an EvaluationBatch so GEPA can decide which candidates deserve further exploration.

The proposer is not merely polishing prose. It receives evidence about where previous candidates succeeded and failed, and the evaluator determines whether its revision helped.

Colvin finds a budget of 50 calls too small for this setup and launches the demonstration with --max-calls 400. The published CLI defines this as a maximum number of metric calls, not a count of every provider request including proposer calls. Logfire shows task evaluations, proposer inputs, generated prompts, and subsequent evaluations. Captured print output makes GEPA’s progress readable alongside those traces. Cost is already accumulating during inspection, so the budget is an operational control as well as a stopping condition. The visible loop remains straightforward: propose, evaluate, reuse successful components, and repeat until the search stops.

The live questions expose which controls this integration actually establishes. Colvin has observed different evaluation batch sizes but does not pin down GEPA’s sampling policy. He also has not investigated stopping after a specified number of unimproved iterations. Easier local and open-source integration, alongside background optimization in Logfire, is planned; no packaged optimization skill is demonstrated.

Inspecting the proposer trace reveals a whole replacement prompt, rather than a diff. The candidate in this demonstration is effectively a dictionary with one system-prompt entry. A richer candidate could expose separate entries for model choice, tools, or prompt components, allowing successful values to be combined. Colvin connects this to DSPy’s established pattern of selecting examples to include in a prompt. GEPA can operate within DSPy, but it is not confined to that framework.

This search is especially attractive when making a cheaper or faster model perform a bounded task. Colvin expects less room for improvement when a strong model already has everything it needs. Private context changes that calculation: a bank’s internal operating specification may contain exactly the rules a model needs but has never learned. Choosing the right pieces of that specification can matter more than refining a generic instruction. The MP example is weaker in this respect because public political facts may already be memorized. Colvin considers fresh news or recent Premier League results as possible public substitutes for unfamiliar data, while recognizing that neither is an ideal demonstration dataset.

Unconstrained rewriting has a cost: prompts tend to grow as edge cases accumulate. In an earlier trial, GPT-5 Mini produced very long instructions and slower execution, without a clear quality improvement in Colvin’s account. His proposed alternative is bounded selection—for example, choosing 20 sentences from a library of 200. That constrains length by construction, whereas whole-prompt generation has more freedom to describe newly discovered exceptions. The completed GEPA run reports a 96.7% performance score, compared with roughly 92% for the earlier expert-prompt comparison. The optimized instructions elaborate relationship eligibility, but are relatively verbose; the exact evaluation split behind the final score is not established in the recording, so this is not a verified same-split gain.

The golden-relations JSON in the cases directory makes this experiment tractable. Without reference answers, the first problem is finding a useful judgment signal:

• Human annotation: label a representative subset of cases.
• Executable checks: test whether generated code runs, is valid, and uses available libraries.
• Outcome feedback: return results to their original source and observe whether they were useful or correct.

These signals answer different questions. Code that runs is not necessarily correct. Colvin’s deliberately extreme example is a smoking-cessation agent: the desired health outcome unfolds over decades, while an evaluation must return promptly. Practical proxies might check required advice or reject suggestions to take up cigars. The hard work lies in relating those checks to the outcome that matters.

Stochastic results create another problem. An attendee runs the same exercise and gets a different result. Colvin recommends increasing the repetition count per case to obtain more stable measurements, while noting that extensive repeated evaluation can become expensive. The adapter’s EvaluationBatch.scores is a list of floats, so an application can define a score that incorporates its chosen quality and variability objectives. That flexibility does not itself specify a good objective. The batch can also include trajectories: sequences of agent steps, analogous to traces, that help explain how an outcome arose.

An attendee reports a 96% score despite a prompt that explicitly excludes aunts and uncles appearing in the reference answers. This exposes a policy distinction hidden by the loose word “ancestors”: direct ancestors and older-generation relatives are not interchangeable categories. The attendee says Claude Code spotted the contradiction and suggests an independent final check. Colvin suspects that the optimizer’s smaller sample did not include those relationships, allowing an overly simple exclusion rule to score well. His proposed remedy is separate training and evaluation sets that each cover the task’s categories. Overfitting is his diagnosis to investigate, not an established root cause. Aggregate quality can conceal a systematic exclusion.

Changing the model generally means evaluating and optimizing again. A new release can erase the practical advantage of painstaking prompt work on the old one. The economics depend on repetition: Colvin’s hypothetical private-equity example involves hundreds of millions of invoices, where weeks of specialist work could justify moving a large workload to a smaller model. A broad coding agent has a different problem: its trajectories span so many tasks that representative evaluation becomes difficult. Colvin’s account of informal coding-agent development illustrates that difficulty, rather than establishing how much evaluation any particular product performs.

The search space can extend beyond prompts to model choice, compaction strategy, tool registration, and code mode. For a small set of candidate models, simply evaluating each one may be the clearest approach: compare the quality, price, and latency that matter for the application.

Breadth makes optimization harder. Even thousands of coding-agent tests may represent only a small fraction of real usage, and optimizing them aggressively can overspecialize the agent. Selecting representative examples remains valuable for a private domain with substantial context, but a narrow benchmark cannot stand in for an open-ended workload. Fine-tuning faces a related economic tradeoff: training investment may be overtaken by a newer model. Colvin suggests that harness improvements or a better waiting experience may be more valuable for ordinary workloads, while very high-volume applications—particularly in finance—can justify both fine-tuning and careful agent optimization.

The second demonstration puts the MP data behind a small FastAPI application with two endpoints: one serves the page, and one accepts a form submission. Its search tool runs regular expressions over the archived HTML. Asked who represents Hammersmith, the agent returns Andy Slaughter. Logfire connects the incoming HTTP request to the agent run and the search tool call, showing how the answer was obtained.

A managed variable moves the agent’s configuration outside the deployment cycle. The code defines its type and defaults; the running application resolves a value and applies it to the agent. The published web example makes the demonstrated fields precise:

from pydantic import BaseModel

class AgentConfig(BaseModel):
    instructions: str
    model: str
    max_tokens: int

The spoken walkthrough also mentions temperature, but this example’s three fields are instructions, model, and integer max_tokens. The distinction matters because the configuration schema defines what the remote editor can change. This is particularly useful when a production deployment requires a lengthy CI and release process.

The application’s variable-push command publishes code-defined variables to Logfire and asks before overwriting an existing definition. In the recorded setup, logfire projects use does not also authorize managed-variable operations: a separate API key from Settings needs the appropriate variable permissions for pushing and reading values. The managed-variable interface then exposes update history and percentage-based targeting. Colvin describes its OpenFeature foundation as a route for other compatible clients to connect. Initially, all requests use the code default.

To make a configuration change obvious, Colvin adds an instruction to reply in French, saves the value, and targets the latest version at 100%. The same English question about Hammersmith then receives a French answer naming Andy Slaughter. He accidentally stops and restarts the server during this first trial, so that particular sequence does not by itself show an uninterrupted update. The application is intended to resolve the variable whenever the relevant function is called.

He then changes the instruction to German and repeats the question without that restart. The response arrives in German. Next, he changes the configured model from gateway-routed Sonnet 4.5 to GPT-4.1. The visible response subsequently identifies itself as ChatGPT; that self-description is the shown observation, while the configuration edit is the mechanism. Typed managed variables let the application change several settings together, rather than treating the system prompt as the only adjustable part of an agent.

The intended next step is automatic optimization of managed variables against evaluations. That remains a planned platform feature, so the workshop prototypes the connection manually. A second variable holds MP-relations instructions. The web agent has both an MP-search tool and an extraction tool; the latter calls the same extraction function optimized earlier. Resetting the search configuration’s targeting to zero restores its code defaults before testing this nested workflow.

A request for Stephen Kinnock’s political relations selects the extraction tool. The trace shows the tool receiving Kinnock’s name and invoking the nested extraction agent. It returns the expected data, but that makes it a poor demonstration of improvement: the simple prompt already succeeds on this case. Colvin searches the earlier evaluations for a case where changing the prompt might visibly help.

After discarding a comparison he initially reads in the wrong direction, he selects Andrew Gwynne. The reference answer is empty, but the weaker extraction treats Gwynne’s sports-commentator father as a political relation. The live request reproduces that false positive. This is a useful test because the failure concerns political eligibility, not whether the family relationship exists.

The attempted fix reveals a wiring problem first: the web application is not using the task variant that reads managed instructions. Colvin changes the code and restarts the server, then publishes the GEPA-generated prompt through the managed variable. The next request still returns the sports commentator. The final trial does not demonstrate improved extraction. It leaves the remaining cause unresolved, while making an essential distinction visible: delivering a new configuration and improving the result are separate things to verify.

User feedback can supply new evaluation cases, but collecting it is difficult. Asked about NPS-style ratings and deliberately negative feedback, Colvin recommends keeping explicit controls to thumbs up and thumbs down. Logfire annotations can seed a reference dataset. A richer signal may already exist in a conversation: the next user turn. A correction or request to try again is evidence of a problem; thanks suggests satisfaction. A user leaving is more ambiguous and should not become an unquestioned correctness label. Colvin compares these signals to a search user immediately returning to choose a different result.

The current page is a minimal form generated with Claude, not a complete conversational interface. Colvin mentions Pydantic AI’s Vercel AI protocol integration, then improvises a conversion using to_web(). Calling it directly, and guessing at a .run method, does not launch the interface. After checking documentation, he defines an ASGI application and serves it with Uvicorn. The pattern is:

# chat.py
from task import relations_agent

app = relations_agent.to_web()

uv run uvicorn chat:app

The current web UI documentation identifies this built-in interface as a local development and debugging facility. In the recording, the browser interface loads after the launch correction.

A conversational interface would allow follow-up questions with retained context, but the extraction agent is designed to return structured data rather than chat naturally. The visible result establishes that the interface has loaded, not that a complete conversational workflow has succeeded.

The closing questions return to a concrete internal use case. Logfire has an agent that converts natural-language searches into SQL, and the team optimizes it for producing good queries. Its request volume is low enough that token cost is not the primary concern. Representative examples are difficult to show publicly because a query can contain private identifiers, such as a customer’s invoice number. The useful optimization target is SQL quality, even when the underlying examples cannot become demonstration material.

Sensitive workloads can separate measurement from content export. In response to a question about medical data and obfuscation or encryption, Colvin proposes recording performance metrics while omitting raw prompts, inputs, and outputs. This loses debugging detail but preserves a measurement channel; it is not a demonstrated encryption scheme. Enterprise self-hosting inside a VPC is another option. He cites legal workflows at Legora and Harvey to illustrate the constraint: categorical grades can leave the protected environment without exporting generated text containing client data.

A smaller implementation question has a similarly practical answer. Colvin uses dataclasses for internal structures that do not need validation and Pydantic models where validation is required. He treats their construction overhead as negligible beside model calls, rather than choosing between them to optimize this workload’s speed.

The final question extends optimization to summarization, fresh contexts, and compaction. Colvin agrees that compaction strategy belongs in the search space for a more complex agent, without reporting a combined experiment here. He returns to the Shopify architecture example: where GPT-5 could receive a whole website, a smaller Qwen model needed an agent to issue targeted queries for relevant terms. That changes the information presented to the model, not just the wording around it. For a smaller model, selecting the right context can be what makes the task feasible and its behavior more consistent.