Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust

An agent can generate a great deal of telemetry without making its problems easy to find. A final answer tells you what the user received; it does not explain which tool ran, what input it received, or where the workflow went wrong. Doug Guthrie, a solutions engineer at Braintrust, builds this workshop around that gap between collecting traces and using them to improve an agent.

Tracing records the intermediate steps that lead to an output, including tool inputs and outputs. An untraced part of the system remains a black box for both an engineer and a coding assistant investigating a failure. More detailed tracing creates more data to examine, but it also supplies the evidence needed to distinguish a bad final response from a broken step upstream.

The workshop uses a deliberately simple Python support agent built with the OpenAI Agents SDK. That choice makes the integration concrete without making the framework the lesson. Guthrie describes other integrations, including OpenTelemetry and custom orchestration: the essential requirement is to capture the agent’s work, whatever produces it.

The intended development loop starts with a change, evaluates whether it improves or regresses behavior, and deploys it when the evidence supports doing so. Production then supplies the next set of observations. Guthrie’s compact instruction is that “production should inform development.” At scale, this requires another step between observation and editing: turn many traces into a smaller set of useful findings.

A scorer turns a question about behavior into a result that can be compared or filtered. The question might be mechanical—did the agent call the required tool?—or subjective—did this conversation resolve the user’s problem? Different questions need different scoring mechanisms.

• Code checks: Validate a schema or check whether tool A ran before tool B. These checks avoid an LLM call and are generally fast and cheap.
• LLM judges: Give a model explicit criteria for assessing an output or conversation. This allows judgments that are harder to express as deterministic rules.
• Human calibration: Review the judge’s results and reasoning to decide whether its judgments match the intended quality standard.
• Code with model calls: Put branching or other complex logic in a code-based scorer, invoking an LLM where the logic needs a subjective assessment.

A judge needs maintenance. Its explanation helps a reviewer understand why it assigned a score and how the criteria might need to change. Guthrie recommends starting with harsh binary judgments and difficult examples; a perfect result can mean the dataset or scorer is too forgiving. Examples in judge prompts and a stronger scoring model can help, though he presents these as practices that have worked with customers rather than rules for every workload.

Offline evaluation runs a candidate against test cases before release, potentially in CI, to compare improvement and regression. Online scoring applies judgments to incoming production traffic. A routing score of zero, for example, gives an investigator a way to isolate failed cases from the larger stream. The scoring criteria provide the signal; filtering makes that signal usable.

An evaluation suite covers the behaviors its authors thought to test. Users can still encounter failures outside those cases. Braintrust’s Topics feature looks for patterns in trace data, groups them, and exposes their frequency. The resulting examples can suggest new evaluation cases or new scorers. Patterns in user intent can also reveal a capability users want that the agent does not yet provide.

A facet defines what to look for. The built-in facets cover task, issues and sentiment: what users are trying to accomplish, what goes wrong, and how users feel during the interaction. The model creates a taxonomy through the pipeline rather than requiring the user to supply every category in advance. A custom facet supplies a prompt for a more specific question.

How does a large execution trace become something an investigator can filter? The diagram follows the transformation. A preprocessor reduces the trace to the user, assistant and tool-call exchange. A facet summarizes that exchange for its particular concern. The backend embeds the summary, builds a topic map, and uses the result to label and classify traces. Those labels connect aggregate patterns back to individual examples.

Cost matters because a discovery process that sees only a tiny sample may miss the tail of behavior it is meant to uncover. Guthrie describes running Topics on 100% of traces as a design goal and quotes six cents per million input tokens and 40 cents per million output tokens for the hosted processing discussed here. Those are workshop-era figures, not a complete estimate of the cost of processing a particular application.

An audience question challenges the premise: if an agent repeatedly uses the same tools and workflow, won’t its traces be too similar for embeddings to distinguish useful patterns? Guthrie points to customer experience and the forthcoming demonstration, while acknowledging that the workshop data is scripted demo data. The practical question remains application-specific: whether the facets separate meaningful differences in the traces your users actually produce.

In response to a competitor question, Guthrie emphasizes integration flexibility and Brainstore, Braintrust’s purpose-built backend. He describes moving away from ClickHouse after encountering scaling problems with customer workloads. That is his account of the product’s architectural motivation; the workshop does not provide a comparative database benchmark. The relevant capability for the later demonstration is direct SQL access to traces and spans.

Customization starts before clustering. A custom preprocessor can include metadata or spans the default message-oriented preprocessor does not capture, and a custom facet can ask a domain-specific question. Long conversations also require settings that reflect how turns are distributed across traces and how long an interaction lasts. These choices determine what evidence reaches the pipeline and when it is processed.

The same approach can examine internal coding-agent sessions. Guthrie names integrations for Claude Code, Codex, Cursor and OpenCode, with Topics helping teams understand what engineers do in those sessions. Understanding usage is a starting point; he treats measuring the tools’ broader effect on engineering efficiency as an area that still needs additional machinery.

Deployment can separate the control plane from the data plane. The web app and control-plane metadata stay with Braintrust, while traces, datasets and experiments can reside in the customer’s infrastructure. This distinction matters for teams evaluating where their trace data will be stored; it does not describe a wholly self-hosted web application.

Venue internet problems turn the hands-on portion into a walkthrough. The setup still makes the dependencies clear: create an organization, create the AIE-Workshop project expected by the repository, configure AI providers, and generate a Braintrust API key. The project name in the application and repository must agree so logs and evaluation assets land in the intended place.

Provider configuration supplies the models used by LLM judges. Guthrie adds OpenAI and Anthropic credentials, and describes custom providers for organizations that route model calls through their own gateway. A firewalled gateway may need network access configured as well. Ordinary judge calls use the organization’s configured models; Topics uses Braintrust’s built-in models.

Locally, uv sync installs the Python dependencies and creates a virtual environment. The example .env supplies a Braintrust API key and a default model available through the configured providers, with an option to choose a different judge model. The Braintrust CLI adds the operations needed later: query SQL, inspect logs, create datasets and run evaluations. Skills teach the coding assistant how to use those operations, while BT init creates a config.json identifying the organization and project.

After seeding the database and checking configuration with make ready, a chat interaction produces a trace in Braintrust. The integration sets the OpenAI Agents SDK trace processor to a Braintrust tracing processor, directing emitted traces and spans to Braintrust. Guthrie uses explicit setup because the example also accesses configured providers through Braintrust’s gateway; he notes a simpler auto-instrumentation option.

The initial evaluation loads a small dataset, invokes the support agent for each row, and stores the telemetry and results as an experiment. Its scores mix a code-based required-tools check with LLM judgments such as support resolution and tool-use quality. This establishes something to compare against after a change: the next experiment can show whether a fix improves the new case while damaging previously tested behavior.

Publishing the scorers makes the repository’s evaluation logic available for online use. The workshop’s push-scores command uploads them to Braintrust, where automations can invoke them on incoming logs. Keeping scorer definitions in code also allows a playground user to reuse a scorer whose source remains version-controlled.

The judge’s input should match the question being asked. The thread variable gives it the back-and-forth conversation, useful for multi-turn resolution. User messages, assistant messages, or a particular span’s input and output support narrower checks. Judging a policy lookup’s intermediate result requires different evidence from judging whether an entire support conversation succeeded.

• Span scope: Assess an intermediate step, using filters to select the relevant span. Selecting span scope defaults to the root.
• Trace scope: Access the spans within one trace. The demonstration uses this scope for communication quality, support resolution and tool-use quality.
• Group scope: Combine related traces when conversation turns are recorded separately. A shared conversation ID provides the relationship.

Guthrie sets judge sampling to 100% for the demonstration so the incoming examples receive broad scoring coverage. That setting incurs model compute for the selected judges. Sampling is therefore a cost-and-coverage decision separate from choosing the scorer itself.

Topics receives the built-in facets plus a custom support workflow issue facet from a Markdown prompt in the repository. The custom prompt outputs none when its no-problem criteria are met, and a regex excludes those outputs. This keeps the issue map focused on failures instead of filling it with successful interactions.

Processing settings control which traces qualify, what proportion is sampled, and which time window supplies the topic map. Timing and group scope also address conversations spread across many traces or a long period: the pipeline can use the shared conversation ID and an invocation interval suited to the interaction. These settings can be configured through the CLI as well as the UI.

The import supplies enough data to examine patterns: BT sync loads 968 sample traces containing 5,260 spans from an S3 bucket. Each conversation turn is represented by its own trace. In one example, the current turn carries eleven previous messages; the OpenAI Agents SDK has attached that history, so it is already available without separately retrieving every earlier trace.

Consider the user checking the status of a returned-item refund. The incoming trace initially records the interaction. The configured automations then attach additional interpretations: negative sentiment reflects frustration with the pending transaction, while the custom workflow facet identifies an order-record retrieval failure. Both the specific order-ID lookup and the broader customer search failed to locate the order. A vague unsuccessful support exchange has become a case with a named workflow problem.

The trace views answer different investigative questions. Thread view shows the conversational exchange; timeline view helps locate bottlenecks and inspect information such as cache hits across model calls. Loop, the embedded assistant, can also create a review UI over trace data so another participant can examine agent outputs or calibrate judges.

In a project where processing has already finished, the topic map shows grouped patterns and their frequency. The built-in issues map has not appeared: Guthrie explains that map generation requires at least 100 matches or generated facets, and the default issues prompt does not fit this agent well enough to produce them. The custom support-workflow facet does reveal record-lookup failures. An absent default issue map therefore cannot be read as an absence of problems.

The useful change here is in what can be investigated, rather than a demonstrated repair to the refund workflow. Task, sentiment, workflow labels and online scores now give an engineer—or an assistant—ways to select examples and inspect the tool failures behind them. The custom facet makes a domain-specific failure visible that the initial evaluation and default issue classification had missed.

Manual investigation combines labels and scores to narrow the logs, then opens traces and spans to discover what happened. That workflow is useful but reactive: someone must keep choosing filters and reading examples. Guthrie asks Loop, “Help me understand what I should improve in my support agent based on recent logs.” Loop responds by issuing SQL queries against Brainstore to retrieve relevant context.

The labels do important work before the assistant reads the detailed evidence. Pulling every trace into context would repeat the original scaling problem. Queries can instead find relevant groups and representative failures, after which the assistant examines span inputs and outputs. Classification selects where to look; the execution trace explains what went wrong.

A selected failure can then become a dataset row for offline evaluation. The refund lookup problem illustrates the sequence: discover the workflow failure, inspect its trace, preserve a useful case, change the agent, and rerun the evaluation. A new lookup-failure scorer may also be needed so the same kind of problem can be detected before the next release.

An audience clarification makes the purpose precise: this is an evaluation dataset, not a training set. Adding a trace does not itself update model weights. It expands the cases used to test the agent. The evaluation combines a dataset, a task that invokes the agent, and scorers that judge the results; preserving older cases helps detect regressions while new cases extend coverage.

The agent-improvement skill packages the investigation loop for a coding assistant: write SQL, view traces and spans, create scores, modify the agent, and run evaluations. After installing it, Guthrie starts Codex in the repository. The CLI connects the assistant’s local code access to the scored and classified traces in the Braintrust project.

Codex queries the 968 support traces and identifies scored failures, including a visible cluster of errors in the escalate-to-human tool. It then examines classifier outputs and representative failed traces, continuing to query and drill into individual examples. The demonstrated investigation uses aggregate signals to find candidates, then detailed execution evidence to understand them.

This division of work matters. Topics and scorers provide searchable observations; SQL retrieves a manageable subset; the coding assistant supplies access to the repository where a fix can be made. The workflow joins observability and evaluation through tools the assistant can invoke, rather than requiring it to infer a code change from a dashboard alone.

While Codex runs, the questions return to deployment. Hybrid deployment is an enterprise option in the workshop: traces, experiments and datasets remain in the customer’s infrastructure, while the control plane stays with Braintrust. In BYOC, Braintrust manages the customer-hosted data plane; in the other hybrid arrangement, the customer’s team manages it using deployment tooling such as Terraform or Helm.

Model flexibility has a specific limit. Evaluation can help test a swap between generation models without introducing regressions, and judges can use configured provider models. Topics summarization and vector embedding, however, are restricted to Braintrust-hosted models. Guthrie says the team previously allowed substitutions but got poor topic results; the hosted models were trained for this pipeline. Custom preprocessors and facet prompts do not imply that every model inside the pipeline is replaceable.

Classifications also support trends over time because they are stored as trace metadata. A saved view can filter for classifications.sentiment being non-null, group by its label, and chart sentiment or tasks over time. That moves the question from which patterns exist in a snapshot to how their frequency changes.

• Sampling as the agent matures: Guthrie observes higher judge sampling for newly deployed agents, with lower rates as known failures become less frequent and the marginal signal declines.
• Classification cost: Topics may replace some taxonomy-oriented judge work. He describes Dropbox experimenting with this because broader coverage was more affordable than its existing classification scores.
• Conditional scoring: A code-based scorer can call one scorer, branch on the result, call another, and return an array of score objects. The UI does not directly wire that branching graph; a filter can also gate invocation on the presence of a scorer span.

Projects group the logs, datasets and scorers relevant to an agent or use case. Monitoring is project-scoped in this demonstration; an organization-wide view requires queries that pull data across projects. Guthrie describes a first-class agent object within projects as forthcoming, rather than an existing capability shown here.

Evaluation assets should change when the agent’s behavior or responsibilities change, without requiring constant churn for a simple, stable agent. A prompt edit may need only a few additional examples; a new tool or runtime change may require a larger adjustment. Production observations give the team a reason to expand coverage rather than treating the original baseline as a complete model of user behavior.

Scoring can run in Braintrust or in the application’s own infrastructure. Published code-based scorers can bundle their dependencies, allowing Braintrust to handle asynchronous triggering. Teams that retain that machinery themselves can update existing spans in place by ID, attaching scores, metadata or metrics after the original trace arrives.

Trace access and improvement access need not be identical. Guthrie proposes restricting the raw-data project with role-based access control and allowing a service account to run the investigation. Engineers could then receive aggregate analysis, SQL queries and a proposed pull request without direct access to the underlying traces. The proposal separates who can inspect raw conversations from who can review a resulting code change.

Returning to the coding session, Guthrie shows scorer changes, additions to the agent’s tools, and new examples in a dataset. The session also queries an evaluation experiment. The recording establishes that the workflow reaches code and evaluation assets, but does not present a numerical before-and-after result for the support-agent repairs.

How can this run without someone starting the terminal session each time? Guthrie shows a separate supervisor-agent example whose GitHub Action runs Claude Code with the Braintrust CLI and improvement skill. A schedule or event can trigger the workflow. The platform itself does not yet have a single button that performs the whole loop; the demonstrated automation assembles the available pieces.

The pull request packages what changed, why the assistant recommended it, evaluation impact, links to experiments, and regression examples drawn from the investigation. The reviewer then decides whether the proposal is meaningful. Guthrie reports that one customer generates five to 10 additional PRs per day with this approach; that is an output-volume observation, not a measure of merged changes or quality improvement.

The final demonstration opens the improvement loop to collaborators who work on prompts and models rather than repository code. Remote evaluation exposes an evaluation task and its parameters to a playground. In the supervisor-agent example, those parameters include the system prompt and model, plus prompt and model settings for the math and research agents. A product manager or subject-matter expert can adjust the exposed controls without needing a custom application for every experiment.

Where does the agent run when a playground user clicks Run? The diagram separates configuration from execution. An evaluation dev server can run locally, with the demonstrated setup looking for localhost:8300, or be exposed in the team’s infrastructure. Braintrust sends a POST request to the evaluation endpoint; the agent’s code executes on that server, and evaluation results stream back to the playground.

Guthrie’s hosted example runs on Modal and uses a create-app function to expose the evaluations in an evals directory. The relationship is useful: the playground presents the parameters engineers choose to expose, while the server retains the agent’s tools and execution logic. Collaborators can explore prompt and model changes against the actual evaluation code.

The ending turns the same methods back on coding assistants themselves. Braintrust is working on ways to evaluate the skills those assistants use and understand their Claude Code and Codex sessions. The next object to inspect is the development workflow: which skills help, how sessions unfold, and whether the coding assistant is making engineers more effective.