[Evals Workshop] Mastering AI Evaluation: From Playground to Production

How do you move from trying a prompt to knowing whether an AI feature works well enough for users? Carlos Esteban and Doug, introducing themselves as Braintrust solutions engineers, structure this workshop around that progression: evaluation fundamentals, hands-on work in the UI, the SDK, production logging, and human review. Doug brings a background in data and finance; Carlos previously worked with Terraform and Vault at HashiCorp. The workshop guide, distributed through Slack and a QR code, accompanies the exercises.

The practical questions are specific: Which model should you use? What does acceptable quality cost? Does the application handle edge cases, maintain the right brand voice, and improve as you change it? A prompt edit can look better on the example in front of you while causing regressions elsewhere. Evals turn those development decisions into questions you can test across examples.

Carlos connects that testing discipline to faster releases, less manual review, better model economics, and participation from both technical and nontechnical colleagues. He mentions customer improvements in productivity and product quality without supplying numerical results. The working loop has three connected concerns: prompt engineering changes the behavior, evaluation measures the change, and observability reveals what happens when users encounter it.

An eval is a structured test of quality, reliability, and correctness across user scenarios. Its basic contract has three parts:

A task can be one model call or an entire agent workflow. The essential boundary is an input and an output. Scoring can use an LLM judge or a code function; the numeric scores discussed here fall between 0 and 1 and are displayed as percentages.

You do not need a mature production dataset to begin. Synthetic examples can establish the first test cases. As the application develops, bring real interactions into the dataset so its coverage increasingly reflects what users actually do.

Offline evals exercise proposed behavior during development, through the playground or SDK. Online evals grade captured live traffic. The latter reveal problems that the development dataset may never have represented: an unfamiliar request, a missing edge case, or a weak area of the product. Those discoveries become candidates for the next offline test set.

A score is useful only if it tracks the quality you care about. Compare your assessment of an output with its automated score before deciding what to fix.

The two disagreement cases are especially valuable. They identify a measurement problem before a team spends time optimizing the application toward the wrong target.

For a single prompt, Mustache templating inserts dataset values into the message. More complex tasks keep the same input/output boundary while expanding what happens inside it:

• Conversation history: Extra messages supply the exchanges between user and assistant, including simulated tool calls, so the evaluation can consider the conversation as a whole.
• Tools and retrieval: A prompt can use tools to obtain information or contact external services, including retrieval-augmented generation workflows.
• Prompt chains: One prompt's output becomes the next prompt's input. Carlos uses a three-prompt chain to illustrate evaluating the full sequence end to end.

The dataset separates the request from its reference answer and supporting information.

Start with a small set and expand it deliberately. Internal and staging interactions can supply realistic cases before substantial production traffic exists. Human review then improves the reference answers, especially the expected field, rather than treating the initial synthetic set as permanent ground truth.

Choose each scorer according to the question it must answer. An LLM judge suits contextual criteria that require interpretation; code suits exact conditions, required structure, and binary checks. Using both gives you different ways to inspect quality. For an LLM judge, Carlos recommends a stronger model than the cheaper model being evaluated, a single focused criterion, explicit evaluation steps, and only the relevant input/output context. The judge itself should be tested against human assessments.

The UI provides two places to work with these ingredients. Playgrounds support rapid changes and side-by-side prompt or model comparisons. Experiments retain results for comparison over time, including work originating in either the UI or SDK. Saving a playground run as an experiment gives the team a historical reference for its next change.

The first audience question exposes a boundary in the workshop's UI: its described conversation roles are assistant, user, and tool call. Representing additional user roles or branching interactions requires the SDK's flexibility. The general task contract still holds; the workflow does not have to fit a fixed number of turns or a simple prompt editor.

The local exercise requires Node, Git, a Braintrust account and organization, and a project called Unreleased AI. The demonstration uses OpenAI credentials, although Doug also describes other providers, including Bedrock and custom providers. Asked about local models, he points to the remote-evals bonus activity; the session does not demonstrate that configuration or an entirely local Braintrust installation.

Unreleased AI takes a GitHub repository URL, retrieves commits since the latest release, and summarizes them into a changelog for developers. In the workshop checkout, .env.local holds the Braintrust and OpenAI API keys, with an optional GitHub credential to reduce rate-limit problems. Running pnpm install installs dependencies and publishes workshop resources, including two changelog prompts and a dataset. The public repository has since differed in project naming and prompt count, so the two-prompt setup here follows the recording.

The connection between local files and UI objects is a publishing step. The install script invokes braintrust push, which publishes the SDK-defined resources into the Braintrust organization. This lets prompts and other assets live in version control alongside application code. Once published, the two prompt variants appear in the UI, where their Mustache variables map to values from dataset rows.

Each changelog case supplies a series of commits, the repository URL, and a since field identifying the last release. That input establishes what the model has to summarize. The workshop's resources.ts defines three checks: accuracy, completeness, and formatting. Accuracy and completeness use separate LLM judges; combining them into one instruction would make it harder to tell which quality changed. The accuracy editor shows GPT-4.1, commit and output placeholders, and criteria focused on factual correctness.

Formatting uses code because it asks whether the output matches an expected structure. For example, a changelog contract that requires three named sections can be checked directly in TypeScript:

const requiredHeadings = [
  "## Features",
  "## Fixes",
  "## Other changes",
] as const;

export function formattingScore(output: string): number {
  const lines = new Set(output.split(/\r?\n/).map(line => line.trim()));
  return requiredHeadings.every(heading => lines.has(heading)) ? 1 : 0;
}

This checks the chosen heading contract. Whether a listed feature is supported by the commits, or whether an important fix was omitted, remains a separate question for the accuracy and completeness scorers.

To compare the prompts in the playground:

1. Configure the provider key in the Braintrust account so the selected OpenAI models can run.
2. Load both prompt variants as tasks.
3. Select the changelog dataset.
4. Attach accuracy, formatting, and completeness scorers.
5. Run the comparison. The playground processes the examples and scoring in parallel.
6. Open the summary layout to compare the baseline task with the comparison task.

Doug also permits npm for setup, although he uses pnpm throughout the demonstration. The comparison now covers the selected dataset rather than a single hand-picked output.

The Experiments button opens a creation dialog already populated from the playground. Saving it preserves the comparison for later changes and CI workflows. When an attendee asks what completeness actually measures, Doug opens the judge: it applies the instructions in its prompt to the supplied data and generated output. That definition needs evaluation too. Changing the task model and rerunning the same cases gives evidence about the effect of the model change under those particular scoring criteria.

A setup failure in the room makes the authentication path concrete. Cloning the repository does not connect it to Braintrust. Copy .env.local.example to .env.local, create an API key in the Braintrust organization settings, and put it in that file at the cloned repository's root. After the credentials are populated, the workshop's install hook can authenticate and publish the configured resources. The file does not belong in an assumed home directory, and the key is not created in project settings.

The SDK moves the same work into a version-controlled workflow; Python is available as well as the TypeScript used here. Prompts, scores, and datasets can be defined with the application, while evaluation definitions specify the task, data, and at least one scorer. Publishing assets and running tests are separate operations:

The distinction is also visible in the repository's package scripts. Evaluation files use .eval.ts, allowing a directory of evaluations to be discovered without listing every file. Current documentation uses bt eval; the command names above preserve the workshop's CLI workflow. Source control supports consistent assets across environments, while published scorers can also support online scoring.

The SDK discussion returns to a common starting barrier: waiting for a golden dataset. Doug recommends beginning with synthetic examples or logs from a newly released feature, then using the results to decide whether to improve the application or the scorer. A small set provides a place to start that investigation; waiting for a large curated collection postpones the feedback.

An attendee identifies GPT-4.1 as the completeness judge and asks whether consecutive runs can receive different scores. Carlos confirms that they can. A stronger judge does not eliminate nondeterminism. He describes a customer practice of running five trials and averaging the scores; that is an example of repeated evaluation, not a guarantee that five runs resolve every source of variation.

The percentage also needs interpretation. The judge selects a rubric category, and the configured mapping turns it into a number: Doug confirms that excellent maps to 1. A displayed percentage is a scorer value under a rubric, not automatically an empirical accuracy rate. Inspect the logged rationale to understand why an output received its score. Carlos recommends investigating a judge that gives every output full marks, and comparing changes against a baseline on the same dataset and criteria instead of demanding an arbitrary percentage. Drill into the task's calls, tool invocations, and scorer results together when the outcome looks wrong.

Asked whether that iteration could be automated, Carlos previews Loop, forecasting its first release within one or two weeks of the workshop. The proposed assistant would help improve prompts, datasets, and scorers. Doug describes a tool-using workflow that consults earlier experiments, changes prompts, and runs new comparisons under user direction. Carlos says Braintrust evaluates its own AI feature with its logs and scores, and that CEO Ankur periodically benchmarked the use case until a model met the team's expectations. He does not identify the model or provide the benchmark results.

Judge rationales are accessible through the API, Doug explains, so a team can build another evaluation layer around the explanations. An attendee proposes using this to detect weaknesses in a strict, workflow-specific evaluation process. The trace view makes the immediate inspection possible: open the scorer result, read its rationale, and decide whether the explanation supports the assigned score.

To build confidence, Carlos recommends human review of both the score and its appropriateness. A code scorer can provide another check on a similar criterion using regular expressions or other logic. If the judge gives a low score while the code check gives a much higher one, that disagreement identifies a case needing attention. It does not establish which evaluator is correct; the output-versus-score matrix still determines the investigation.

Another attendee asks about traditional machine learning between rule-based checks and LLM judges: intent classification, entity recognition, sentiment classification, and clustering. The exchange leaves that choice open. Carlos reports customers taking both code-focused and LLM-focused approaches, depending on their use case. The attendee also distinguishes probabilistic classifiers from deterministic code. Neither instructor offers a definitive recipe for that middle ground; their practical recommendation is to compare candidate scorers and use human judgments to determine which ones reflect the desired quality.

Doug then runs the code-defined evaluation and shows it executing. Its ingredients are unchanged: task, scores, and dataset. An audience question prompts a sharper distinction between the two interfaces: the playground is the place for quick, comparatively ephemeral iteration; an experiment is the retained record for historical analysis. Their UI capabilities have converged, Carlos says, but saving an experiment preserves the work for future comparisons.

SDK evaluations produce experiments. To retain code-defined behavior while iterating interactively in the playground, Carlos points to remote evals. The useful boundary is that application evaluation code can execute on your infrastructure while Braintrust triggers it and displays the results. The workshop refers participants to the bonus activity rather than demonstrating that setup.

After deployment, the question changes from how the application performs on selected cases to how people actually use it. Logging reveals missing scenarios and makes production traces available as future dataset rows. It also gives product managers and subject-matter experts a way to inspect requests and feedback alongside developers.

Instrumentation begins with an authenticated logger bound to a Braintrust project. Wrapping an LLM client captures its prompts, responses, token counts, latency, and errors. Carlos names wrap_openai and also describes Vercel AI SDK wrapping and OpenTelemetry integration. For behavior outside the model call, trace arbitrary functions and use span.log to attach additional information such as metadata. The wrapper provides the initial visibility; explicit spans describe the application around it.

Online scoring measures selected live logs. Sampling controls how much traffic receives scores, while regression alerts should reflect the scorer's meaning and the established baseline. Tagging traces with prompt variants allows production comparisons between prompt A and prompt B. To configure the workshop's rule:

1. Open the project's Online Scoring configuration.
2. Add a rule and select the scorers.
3. Set the sampling rate, starting lower if appropriate while you establish confidence in the measurements.
4. Select the span to score: the root span by default, or a particular nested child span.

Saved views then preserve filters, sorting, and selected columns so colleagues can repeatedly inspect the same subset of logs.

For the live exercise, pnpm dev starts the application on localhost:3000. A repository selection triggers commit retrieval, changelog summarization, and categorization. In Braintrust, Doug opens the top-level generate-changelog request and follows its child operations: obtaining release information, fetching commits, and loading the prompt from Braintrust. The model span exposes prompt tokens and estimated cost; the small demonstration does not yet provide a meaningful historical trend.

The application's generate route.ts connects those traces to the code. It wraps the AI SDK model for model-call logging, then adds explicit inputs, outputs, and metadata where the application needs more detail. Metadata becomes especially useful when filtering logs into views: it preserves the context needed to distinguish otherwise similar requests.

A saved view can direct reviewers toward requests with low completeness. From a selected trace, the UI can add a span to a dataset as a new row. This closes the development loop: evaluate offline, deploy, observe a real interaction, review it, and bring that case back into offline testing.

An attendee asks whether only good responses belong in the dataset. Doug explains that both successful and unsuccessful interactions can be useful: good responses provide examples of desired behavior; failures expose cases the application needs to handle better. Once an online scoring rule exists, it supplies scores for matching logs without someone manually launching an evaluation for each interaction.

Doug configures the demonstration rule to sample all logs, explicitly distinguishing that choice from a likely normal operating rate, and generates another changelog. Doug reports an accuracy score of 25% and a completeness score of 100% for the demonstrated online changelog output. These are the rubric-derived values for that output. He then saves a view selecting accuracy scores below 50%, giving human reviewers a concrete queue to investigate.

Existing applications do not need to move all their prompts into Braintrust-managed resources to obtain tracing. In response to a comparison with LangSmith, Doug points to client wrappers and function decorators as the integration mechanism. Turning a logged span into a dataset row does have a structural requirement: the logged data must map to the dataset's schema.

For event-conditioned scoring, Carlos suggests creating a span only when the relevant condition occurs and targeting the scoring rule at that span. Sampling then applies to occurrences of the conditional span rather than the whole stream of root traces. Selecting every occurrence of that span can ensure the event receives evaluation without scoring every unrelated request.

Human review supplies domain knowledge that automation can miss. Doug describes product managers, subject-matter experts, and doctors reviewing outputs, identifying hallucinations, and establishing reference judgments. Their assessments help determine whether the product meets real expectations, including nuances that an automated grader may overlook.

There are two distinct sources of human input:

• Reviewer annotations: People enter the platform to label, score, and audit interactions.
• End-user feedback: People using the application submit a thumbs-up, thumbs-down, or comment about a particular result.

Use them together by filtering for negative user feedback, asking reviewers to inspect those interactions, and adding useful failure cases to the dataset. A dissatisfied user identifies a case worth reviewing; the reviewer determines what actually went wrong.

Doug submits a thumbs-up with the comment “Really good” in the changelog application. Doug shows the submitted thumbs-up as a user-feedback score of 100%, alongside the comment “Really good.” The logFeedback function associates that feedback with an already-created span, allowing saved views to select feedback values of 1 or 0. The score here encodes the submitted feedback; it is separate from the automated accuracy and completeness scores.

Human review mode hides fields that are irrelevant to reviewers. Before it is useful, configure the review fields: options, free-form input, or sliders, with binary judgments or richer categories as needed. Reviewers then assess the logged input and output through those fields. Carlos describes using a team's annotations across several criteria to evaluate an LLM judge: make the judge prompt the task in a playground, supply the reviewed cases, and compare its judgments with the human labels. The evaluator becomes the system under test.

How much preparation is enough before this loop becomes useful? Carlos recommends starting with one or two scores and ten dataset rows, then adding cases and refining criteria as evidence accumulates. The goal is to begin learning, rather than wait for a golden dataset or an extensive collection of scorers.

The next question identifies a different maintenance problem. An application's workflow can grow from two turns to five across sprints. If the evaluation contains a separate imitation of the old workflow, even an excellent scorer will be grading obsolete behavior. Carlos initially recommends durable quality criteria that survive changes in the task, but the attendee clarifies that the task implementation itself must follow the application. These are separate responsibilities: keeping the tested code current and keeping the definition of quality valid.

The resolution is to have the eval call the application's task directly. A pull request then evaluates the proposed implementation, including its changed number of steps, rather than a separately maintained mimic. The binding can be a thin TypeScript function:

export function bindEvaluationTask<Input, Output>(
  applicationTask: (input: Input) => Promise<Output>,
): (input: Input) => Promise<Output> {
  return input => applicationTask(input);
}

Pass the current application's exported task to this binding; do not reproduce its conversation or tool sequence inside the evaluation. With that arrangement, an implementation change alone does not require rewriting the .eval.ts file. Changes to the input/output contract or the qualities being measured still require deliberate maintenance.

Staffing human review remains organization-specific. Asked about Braintrust's own specialized reviewers, Carlos says the company is not then using SMEs internally and has only recently added an AI component. He describes customers in domains such as healthcare and legal technology using external annotation services or bringing reviewers into Braintrust with dedicated roles and annotation views. The session does not offer a fixed staffing formula.

The final question asks whether high-scoring examples can improve future prompts without fine-tuning. Carlos explains that dataset-row metadata can hold few-shot examples referenced during evaluations and playground runs. The attendee then distinguishes that from production behavior: can a live request automatically draw good examples from the accumulated dataset? At the time of the recording, Carlos says Braintrust does not natively provide that live-traffic-to-few-shot workflow. It can be built with custom SDK and application logic. Collecting, reviewing, and testing examples creates the material for improvement; deciding how those examples enter a production prompt is another application behavior to implement and evaluate.