From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

How do you know whether a changed agent will work with your company’s real tools, API services, policies and workflows? A useful benchmark must reproduce those conditions and keep growing as production exposes new cases. Rustem Feyzkhanov, who leads AI platform work at Snorkel AI, treats benchmark construction as an engineering discipline. Snorkel produces benchmarks using agents and subject matter experts; Feyzkhanov reports that it runs millions of agent simulations per month.

Start with a production trace: an input prompt, the actions the agent took and its final output. An evaluator can inspect that record to determine whether the agent succeeded or encountered an edge case. The opening slide makes this concrete with a billing-dispute trace and an observed failure. That record gives you evidence about what happened.

Testing what would happen with a different configuration is harder. Production A/B tests encounter changing database contents and tool versions, so two agents may not face the same problem. Offline simulation turns a trace into a repeatable experiment: construct a task from the production interaction, then run different agent configurations against controlled conditions. Those experiments can run in parallel and compare success rate, cost, latency and retries.

Public benchmarks help establish expectations about capabilities. SWE-bench tests repository issue resolution; Terminal-Bench tests work in a terminal. Computer-use benchmarks cover another interaction surface. Your company’s benchmark must go further into its own use cases, tooling, policies and production environment. Public results help you form a prior; private evaluations help you decide whether to ship.

Pass rate describes only part of the production tradeoff. A successful agent may still cost too much, respond too slowly or require too many retries. Simulation lets you compare those dimensions under the same conditions. It also lets you evaluate the entire agent stack, rather than treating model selection as the only variable.

The unit being evaluated is the full system. A model change, a different reasoning budget or a revised skill can each affect behavior; keeping the environment and evaluators fixed makes those changes easier to assess.

The same benchmark supports successive stages of agent development:

1. Make it work. Check edge cases, select a model and inspect traces to understand failures before the first release.
2. Make it reliable to change. Put the benchmark in the release path as an integration test, checking that modifications to the stack or harness do not introduce regressions.
3. Optimize it. Tune cost and latency, and use simulation traces for reinforcement learning.

Feyzkhanov cites a task-specific example in which fine-tuning a smaller Qwen model brought it to the performance of a larger Qwen model; he does not give model sizes or measured scores in the talk. The broader architectural point is that a simulation benchmark can serve three roles: evaluation suite, release integration test and source of training data.

A benchmark task has an execution sequence:

1. Give the agent an input prompt.
2. Let it interact with APIs, MCP tools, databases and files in the environment.
3. Collect its trace, the final environment state and any output artifacts.
4. Run verifiers over those results to produce success and quality metrics.

The output is therefore more than the agent’s last message. It includes the world the agent leaves behind.

A task also needs an oracle solution: a constructed solution that runs through the same sequence in place of the agent. Its purpose is to establish that the task can actually be solved. Without that check, an agent failure may be evidence of a broken task rather than a capability gap.

Harbor, created by the Terminal-Bench team, gives this contract a file-based structure. The talk groups the files by what they expose:

The packaging is deliberately straightforward. Instructions, an environment and hidden evaluation code are enough to turn an interaction into an experiment that can be repeated.

The environment should behave like a miniature production system without requiring a full production deployment for every experiment. Its databases, API services, tools and files need to match the interactions the agent will encounter. An agent should not get an easier or recognizably artificial task simply because it is being evaluated.

A real user cannot be packaged into that environment. A simulated user can instead be an LLM with its own prompt and additional context describing the behavior and interaction it should reproduce. The user’s replies become part of the environment that the verifiers inspect after the run.

The implementation patterns resemble integration tests and development environments:

• Database snapshots: provide the relevant state instead of copying an entire production database into every run.
• Sidecar containers: place the agent in a main environment and expose API services, databases and MCP tools through other containers.
• Mocked services: reproduce the required service interactions without deploying every production service.
• Intermediate task steps: for work that spans hours, give each step its own prompt and verifiers. Stop the simulation early when a step fails instead of spending resources completing a trajectory that has already failed.

These choices bound the experiment while preserving the interactions that matter to the task.

Output checking is familiar from coding tasks: run tests against the code the agent produced. An agent simulation exposes more evidence. The database may have changed, APIs may have returned particular responses and a user may have supplied additional information. Verifiers should inspect the final state, trace and artifacts, choosing evidence appropriate to the task.

• Deterministic checks work well when correctness is directly testable, such as an exact output requirement or a tool call.
• LLM, harness or agent judges can assess trace quality and whether the agent planned appropriately.
• Combined evaluation uses both when the task contains mechanically checkable outcomes and behavior that requires judgment.
• Subject matter experts review selected traces and outputs where discrepancies warrant human attention, rather than reviewing every run.

An executable task is not automatically a trustworthy release gate. An agent may recognize the simulation and exploit it to obtain a reward. A task may be too easy, or its verifiers so permissive that incorrect behavior passes. A defective verifier may reject every attempt. Success may also vary substantially across runs. These are defects to investigate during benchmark development, not signals to accept uncritically as model performance.

The benchmark is software and needs its own CI pipeline. That starts with ordinary engineering checks: pinned dependencies, the correct base image and complete fixtures. For a single-container task, even a small shell check can catch missing inputs before an expensive simulation starts:

#!/usr/bin/env bash
set -euo pipefail

cd "${1:?Usage: check-task TASK_DIRECTORY}"

required_files=(
  instruction.md
  environment/Dockerfile
)

for path in "${required_files[@]}"; do
  if [[ ! -s "$path" ]]; then
    printf 'Missing or empty task file: %s\n' "$path" >&2
    exit 1
  fi
done

This illustrates a file-presence check using Harbor’s current documented directory layout. It is only the first layer: dependency, image and task-specific fixture checks still belong in the pipeline.

Behavioral validation comes next:

1. Run the oracle solution and require the verifiers to pass.
2. Run the verifiers without executing the solution and require them to fail, checking that the task does not pass in its initial state.
3. Run the agent several times to assess solvability and difficulty.
4. Tag the task as simple, medium or hard based on observed success frequency, then approve it for inclusion in the benchmark.

The positive and negative checks matter together: a verifier that accepts the oracle but also accepts untouched state is not testing the intended work.

With a validated benchmark, improvement becomes a controlled loop. Establish a baseline, inspect the failed cases, change one thing and rerun the experiment. Record the results with an experimentation tool such as Arize. After the targeted fix works, rerun the full evaluation before releasing to production.

This process gives you an alternative to continually adding prohibitions and urgent instructions to the prompt. Because simulation exercises the whole system, it can test a fix in the component responsible for the failure:

A longer prompt may change behavior, but it should not become the default storage location for every missing system capability.

Once the agent is deployed, two loops operate together. The benchmark expansion loop takes failures from production observability and turns them into additional tasks. The release loop runs simulations against the expanded benchmark or a new agent configuration, records the experiments and uses the results as a gate before deployment.

Observability and experimentation need to be connected because each supplies what the other lacks. Traces reveal edge cases that actually occurred in production. Simulations let you test what would happen under a changed configuration. Keeping that connection alive makes the benchmark part of agent operations rather than a dataset used once before launch.

The audience discussion adds an important constraint to repeated experimentation: maintain a holdout that the agent has not seen during development. Asked about example counts and dataset splits, Feyzkhanov suggests an 80/20 training–validation split, depending on the use case. He does not prescribe a benchmark size. The central requirement is that the holdout remain unseen throughout experimentation, so it can provide a separate check on the resulting agent configuration.

Coverage requires more than collecting memorable failures. An audience question contrasts handpicked production runs with tasks created before an agent reaches production. Feyzkhanov’s guidance is to include both everyday workflows and edge cases: the happy paths users depend on, tool failures and database problems. The integration-test analogy applies again—successful routine behavior and failure handling both need coverage.

Task construction need not be entirely manual. In response to a follow-up about simulation versus handcrafting, Feyzkhanov points out that agents can already write much of the necessary code. The valuable foundation is to build the environment once and supply context that reproduces production interactions; automation can then help construct tasks around that foundation.

The final question returns to human involvement in verification. Snorkel uses subject matter experts at scale, but the goal is not to have experts review everything. Direct their attention to disagreements among agents and verifiers: a task appears solved but receives a failure judgment, or a trace is judged insufficiently good despite an apparently acceptable outcome. Experts can then tune the agent performing the review. The evaluator itself becomes something to improve, with human expertise concentrated where automated judgments diverge.