The Future of Evals - Ankur Goyal, Braintrust

After an eval finishes, what should you change in your code or prompt? Running evaluations can become routine long before answering that question does. After almost two years working with AI product teams at Braintrust, Ankur Goyal describes substantial evaluation activity: Goyal reports that the average organization signing up for Braintrust runs almost 13 evals a day, while some customers run more than 3,000 a day. Some advanced customers also spend more than two hours a day in the product working through evaluations. These are reported product-usage figures; the observation window and customer cohorts are unspecified.

Even teams building highly automated agents still do much of this improvement work manually. An eval produces a dashboard; a person inspects it, leaves the dashboard, and decides which code or prompt changes might improve the next result. The missing step is turning evaluation results into useful changes. A better dashboard helps people inspect the evidence, but it still leaves that decision with them.

Loop is an agent built into Braintrust to help perform that improvement work. Its development depended on evaluations of the models doing the improving: every quarter for two years, Braintrust tested frontier models on their ability to improve prompts, datasets, and scorers. Goyal says those models performed poorly until very recently.

Goyal reports that Claude 4 performed almost six times better than the previous leading model in Braintrust’s internal tests of evaluation-development tasks. The talk does not specify the scoring metric, test-set composition, inference settings, previous leader, or which Claude 4 variant was used. The comparison explains Braintrust’s decision to pursue this workflow, but does not provide enough detail to reproduce the result.

Loop’s scope extends from optimizing prompts to improving complex agents. It also helps build datasets and scorers, because evaluation quality depends on all three working together.

This makes the evaluation setup itself part of the work: an agent can help improve how a system is tested as well as the system under test.

The launch-era setup presented in the recording is short:

1. Sign in to Braintrust or create an account.
2. Enable the feature flag named Loop.
3. Use the default Claude 4 model or select another model you have access to.

Goyal names OpenAI models, Gemini, and users’ own LLMs as alternatives. These are the instructions and model choices announced at the talk; today’s Loop documentation describes access within the product and model selection through providers configured for the organization.

Loop runs directly inside Braintrust, preserving the ability to inspect data and prompts while working on them. That matters because an optimization suggestion may change different parts of the evaluation. The interface shows proposed changes side by side, including:

• Data edits: changes to the examples being evaluated.
• Scoring ideas: suggestions for how to assess outputs.
• Prompt edits: changes to the instructions driving the system.

The proposal remains connected to the material it would change, so users can examine the edit directly in the working interface.

For users who want more autonomy, Goyal describes an optional toggle as “Just go for it”: Loop proceeds with optimization rather than stopping at suggestions for inspection. He says that mode works well, without providing a separate measurement of its effectiveness. The product therefore offers both visible proposed edits and an option to let the agent optimize autonomously.

Evals have become a critical part of building AI products, while the work of interpreting and improving them remains labor intensive. Speaking at the June 2025 AI Engineer World’s Fair, Goyal forecasts that frontier models will transform evaluation itself over the following year. Braintrust’s intended direction is to incorporate those advances into the tools used to develop and evaluate AI systems.

That direction is still being shaped. Goyal invites users to try Loop and provide feedback, acknowledging that substantial work remains. He also invites people working on UI, AI, and infrastructure to join the team, closing with an on-screen QR code as a route to get in touch.