Evals 101 — Doug Guthrie, Braintrust

When you change an application's model or prompt, how do you know whether its output improved? That question connects the development and production workflows in this workshop. Doug Guthrie introduces himself as a solutions engineer at Braintrust, where the platform UI and SDK connect local development, evaluations, production logging, and human feedback. He traces the company's origin to Ankur Goyal building similar infrastructure at two previous workplaces, and describes customers already using Braintrust for evaluations and generative-AI observability.

LLM outputs are nondeterministic, so a prompt edit or model substitution needs more scrutiny than looking at one appealing response. Evaluations make those changes comparable and help detect regressions. Guthrie recounts Goyal's framing that evals let developers “play offense”: they support deliberate exploration of better application behavior as well as guarding against breakage.

The useful unit is a feedback loop, not an isolated test run. Filter production logs, select relevant spans, add them to a dataset, and use those cases in the next offline evaluation. Guthrie attributes faster development, more shipped AI features, and better application quality to customers using this workflow, without giving numerical outcomes here. In that loop, the playground supports prompt prototyping, evaluations measure changes, and observability exposes what happens with real users—including their feedback and the records humans need to review.

An evaluation is a structured test of an AI system's quality, reliability, or correctness. It needs three ingredients:

The task can be a single prompt or an entire workflow that calls tools. Its complexity does not change the basic input/output contract. The dataset supplies realistic cases; the scorer supplies the logic that makes their results interpretable.

A code-based scorer can check a binary condition or compute a heuristic measure. An LLM judge receives the output and assessment criteria, then returns a judgment that maps to a numerical score. Guthrie gives labels such as excellent, fair, and poor, with score values such as 0, 0.5, and 1. The rubric must specify what those labels mean and how they map to scores.

Tracing records model inputs, outputs, intermediate steps, and tool calls; scoring assesses those records. Depending on instrumentation, traces also expose tokens, cost, latency, and duration. Keeping these responsibilities distinct matters: collecting a trace gives you evidence of behavior, but it does not by itself judge whether that behavior was good.

Start with a baseline you can improve. Waiting for a perfect golden dataset delays the first useful comparison. Once you have a baseline, compare your own assessment of the output with the score:

Both mismatches indicate that the measurement needs attention. A high score is only useful when it tracks the behavior you actually want.

A prompt in the platform starts with an underlying model and a system prompt. Mustache templates supply variables such as the user's question, chat history, or metadata. Metadata also gives you fields to filter later when inspecting logs. Additional system, assistant, user, and tool messages let the prompt represent a multi-turn interaction.

Tools extend the task beyond text generation: a prompt might retrieve documents in a RAG workflow or search the web. The prompt needs access to the tool it is instructed to use. Guthrie then introduces prompt chaining, which was in beta at recording time: one prompt's output becomes the next prompt's input, and each step can have different tools. That makes a sequence of tool-assisted steps available for evaluation as a larger workflow.

The dataset defines what that workflow will encounter. Only input is required. An optional expected value enables comparisons between the generated output and a reference; Guthrie names Levenshtein as a way to measure string differences. Optional metadata supports filtering, including when loading the dataset into an application codebase.

A small initial dataset can grow from production evidence. Reviewers filter logs to meaningful cases, inspect them, and select spans or traces to add to the offline dataset. This turns a problem encountered by a real user into a case the next application change can be tested against.

Code-based scorers can be written in Python or TypeScript, either in the UI or in a codebase and then pushed to Braintrust. Publishing makes them available to colleagues who do not work in that codebase. LLM judges instead use a criteria prompt: for example, a rubric can explicitly assign 1 to good output and 0 to bad output. Both kinds of scorer can serve offline and online evaluation.

AutoEvals supplies ready-made code-based scorers and LLM judges. Guthrie suggests that even a simple Levenshtein comparison can establish a low-effort baseline, while acknowledging that it is unsuitable for many tasks. The baseline gives you something concrete to inspect and replace with a more task-specific measure.

Guthrie reports customers using higher-quality models as judges even when the application itself uses a cheaper model. He then introduces the running example: generating a changelog from commits. Instead of asking one judge to assess accuracy, formatting, and correctness together, give each dimension its own scorer. A formatting failure then remains distinguishable from a content failure.

For a concrete Python formatting check, suppose the chosen rule requires a changelog heading followed by at least one bullet. This narrow rule can be expressed without involving a model:

def changelog_format_score(output: str) -> float:
    lines = [line.strip() for line in output.splitlines() if line.strip()]
    if len(lines) < 2 or lines[0] != "## Changelog":
        return 0.0
    return float(
        all(line.startswith("- ") and line[2:].strip() for line in lines[1:])
    )

changelog = "## Changelog\n- Fix login timeout."
score = changelog_format_score(changelog)

That checks only the stated format; it cannot tell whether the commits support the claim about a login fix. A separate content scorer needs the relevant commits and generated text. For LLM judges, test the criteria prompt in the playground and avoid burying the actual input and output under unrelated context.

The playground brings a prompt or agent together with a dataset and selected scorers. Running it evaluates the task across those cases. This shared interface lets developers and product managers participate; Guthrie also describes doctors at a healthcare customer using playgrounds and human review. Experiments preserve evaluation snapshots so changes can be compared over time. Both the UI and SDK can produce those runs.

The live application takes a GitHub repository URL, fetches recent commits, and generates a changelog. After generation, the user can provide feedback. Guthrie has already created two prompts, a dataset, and scorers in code and pushed them into Braintrust. Those shared assets become the ingredients for the comparison.

The playground procedure is straightforward:

1. Create a playground and load the existing prompt.
2. Duplicate the prompt and change the duplicate's model to GPT-4.1.
3. Attach the dataset and configured changelog scorers.
4. Run the tasks across the dataset; cases execute in parallel.
5. Open the summary layout to compare the base and comparison tasks.

The dataset and scorers are essential to the comparison: changing a model alone does not define what improved performance would mean.

Guthrie reports that the base task scored better on average completeness but worse on accuracy than the comparison task. Both changelog variants scored 0% on formatting under the demo's configured scorer and dataset. Those separate dimensions reveal a tradeoff and a shared formatting problem; they do not identify an unqualified winner.

Guthrie next opens Loop, which he says was released that day. Its Cursor-like interface accepts a request to optimize a prompt. The crucial input is the existing evaluation results: Loop can inspect them, propose a prompt change, rerun the evaluation, and compare the new scores with the old ones. That makes the evaluation criteria part of the editing process.

In the demonstration, Loop fetches results and produces a proposed diff. Accepting the change is supposed to trigger another evaluation, but the rerun does not complete; Guthrie suspects an issue with his Anthropic API keys. The visible proposal therefore does not establish a successful optimization result.

Saving runs as experiments preserves the comparisons beyond the current playground session. You can inspect the latest change against the preceding run or look across a longer development history. The minimum remains a task, a dataset, and at least one scorer; experiments make the results of that combination available for later inspection.

The same workflow can start in a codebase. Guthrie describes Python and TypeScript as the most-used SDKs and also names Go, Java, and Kotlin. His Python demonstration first defines a prompt, then a scorer and dataset. Code-defined assets can live in source control; prompts created in the UI are version controlled as well.

There are two distinct SDK operations:

In the recording, braintrust push publishes the assets. Current function deployment documentation uses bt functions push; the recording's command belongs to its historical workflow. Defining an evaluation is a separate step: it combines the dataset—including one stored in Braintrust—with the task and scorers, then runs it to produce an experiment visible in the platform.

Publishing scorers also makes them available for online scoring, so production behavior can be assessed against criteria used during development. Evaluations can run in CI/CD to check whether configured scores improve or regress. Guthrie points to a GitHub Action integration for incorporating that comparison into the development workflow.

Moving into production starts with logging the application and its tool calls. This lets you measure quality on live traffic and find cases the original dataset missed. A low user-feedback score or a comment can identify a record worth reviewing; selected records can then return to the offline dataset.

Instrumentation can grow in layers:

1. Initialize a logger. Authenticate and select a project. A project groups the prompts, scorers, and datasets for a feature, while allowing assets to be used across projects.
2. Wrap the model client. Guthrie shows wrapping an OpenAI client to capture metrics such as tokens, duration, and cost.
3. Trace functions. A trace decorator records operations such as tool calls.
4. Define custom spans. Lower-level span APIs let you choose the input, output, and metadata to record.

Start with the client wrapper, then add detail where the application needs diagnosis.

Online scoring applies configured scorers to incoming records. Guthrie gives 10% and 20% as example sampling rates, so teams need not score every log. Automations can alert when a score falls below a threshold. Custom views then filter the rich log stream into a manageable set of cases for human review.

The production code demonstration wraps a Vercel AI SDK model to capture metrics, then explicitly defines span inputs and outputs. The span's data shape should match the offline dataset's data shape. If the logged record already has the structure the task expects, selecting it for a dataset does not require reconstructing the request from unrelated trace fields.

Guthrie then configures live scoring:

1. Open the configuration pane and choose Online scoring.
2. Create a rule and select the desired scorers.
3. Set a sampling rate; the demonstration uses 100% so incoming changelog records will be scored.
4. Choose the span the rule should evaluate.

A rule can target an individual span rather than the root span. In a RAG workflow, for example, a scorer could assess whether retrieved material is relevant to the user's query. That is a different question from whether the final generated answer is good.

Running the changelog application again produces new logs and online scores. This connects the offline criteria to actual application usage: the team can inspect how the same dimensions behave over time in production and identify where another development iteration is needed.

The changelog application's feedback control offers thumbs up, thumbs down, and an optional comment. The resulting user-feedback score and comment appear in Braintrust alongside the log. Guthrie filters for feedback equal to 0 and saves that filter as a view. The live demonstration initially has no matching rows, but the saved view defines where reviewers can find future negative-feedback cases.

Pressing R, or using the review button, opens a pared-down human review pane. It pairs the record's input and output with configurable review fields. Those fields can include free text and custom scores, so reviewers assess the dimensions the team has chosen rather than merely leaving an unstructured impression.

The changelog trace exposes more than the final model call. It shows the application finding the latest GitHub release, fetching commits since that release, and generating the changelog. Instrumenting these operations makes the workflow inspectable at the level where a failure may actually occur.

Guthrie switches to a conversational analytics application to show why intermediate scoring matters. Before returning data, the application rephrases the user's question using chat history. If that rephrasing loses the user's meaning, later steps may answer the wrong question even if they execute correctly. A dedicated span scorer can assess the rephrased question; another can assess the inferred intent. Those checks can run both offline and online.

Someone still needs to supply and maintain the human judgment behind these assessments. Guthrie recounts Sarah Sachs's Notion workshop, describing a review role that combines product-management judgment with LLM expertise. In smaller organizations, engineers may do this work. Human assessment grounds the quality and reliability checks that automation applies at scale.

Internal review and user feedback contribute different evidence. Internal reviewers inspect logs and apply configured scoring criteria. Users respond from inside the application, often through a rating or comment. Feedback-based views route those experiences to reviewers, who can decide which cases should influence the next offline evaluation.

The audience first asks how feedback and log data can reveal intent and improve model performance. Guthrie answers through dataset curation: filter to meaningful records, select individual rows or all relevant rows, and add them to the dataset used for offline evaluations. The demonstrated bridge is the transfer of reviewed cases into future tests; it does not automatically modify the model or extract intent from feedback.

For comparing multiple models in production, Guthrie assumes the application serves or swaps among the models. Scores can then be grouped by model and inspected in comparative views. This supports analysis of production variants, although the demonstration does not show how traffic is allocated among them.

Human reviewers can disagree just as automated scorers can misjudge. Asked whether individual reviewers' scoring can be compared, Guthrie recommends starting with a shared rubric and clear guidelines. The platform shows who scored a record; he is unsure whether those reviewer scores can be extracted as a dataset for disagreement analysis. The immediate intervention is to make the expected judgment explicit before inconsistent criteria spread through the review process.

Another question distinguishes relative judgments on a fixed test set from standalone judgments about new production samples. Guthrie confirms online use of LLM judges and recommends evaluating the judges themselves: inspect their rationales and test whether their assessments are sound. He again references a process described at Notion. This gives a direction for checking judge quality, but does not establish an absolute calibration method for a single new answer.

A participant working with a government organization needs to demonstrate accuracy before deployment. Subject-matter experts already have thousands of questions. Guthrie identifies this as an offline-evaluation use case: use the existing questions and configured measures to establish a baseline before exposing the application to production traffic.

The resulting score history can help stakeholders judge progress. Guthrie's hypothetical progression from 20% to 30% to 40% illustrates improvement across iterations, not measured accuracy or a launch threshold. The purpose is to make the application's development visible enough that confidence rests on evidence people can inspect.

The final question comes from a team building an ML platform that wants monitoring data in a unified dashboard. Guthrie says experiments and datasets can be retrieved through the SDK, and describes a customer building its own UI from logged data and experiments. The evaluation workflow can therefore supply a team's existing operational interface as well as Braintrust's own views.