The Art & Science of Benchmarking Agents

Coding agents feel more capable, and their model-card scores keep improving. Yet ask an enterprise whether it is ready to let those agents operate in a high-stakes environment, and enthusiasm becomes hesitation. The ability to measure agents is falling behind their capabilities. Vincent Chen brings this problem from Snorkel AI’s intersection of academic research and enterprise deployments: connections to Stanford, UW, and Wisconsin sit alongside forward-deployed and applied engineers who encounter the consequences of model behavior in practice.

Closing that gap requires several kinds of evidence. Field deployments put engineers and researchers into finance, insurance, and healthcare settings, where an evaluation corresponds to a real outcome. Red teaming, private human evaluations, and crowdsourced labeling reveal other aspects of behavior. Open benchmarks belong alongside these methods because they do more than record yesterday’s progress: they establish a goal that other teams can work toward. Terminal-Bench, METR’s long-horizon evaluation, and ARC-AGI exemplify these public capability targets.

Snorkel’s Open Benchmarks Grants provide the practical background for this framework. Chen reports a $3 million commitment to open benchmarks and more than 120 applications reviewed across academia and industry labs. Those proposals expose two different questions: what makes a benchmark a credible empirical instrument, and what makes it influential enough to change the field’s direction?

Borrowing a Timothée Chalamet reference, Chen frames the examples as an opportunity to honor benchmark greats. Their contributions fall into two complementary groups:

The first group makes a score worth trusting. The second makes the benchmark worth adopting and improving against.

An individual task needs real-world complexity, well-posed instructions, and a verifiable solution. Difficulty does not excuse ambiguity. Other domain experts must be able to solve the task, and validating that property can require more than one expert’s judgment. GPQA, a benchmark of graduate-level scientific knowledge, makes this quality-control problem part of its methodology rather than leaving it implicit.

The review process gives a difficult question several chances to reveal its defects:

1. An expert writes the question and its intended answer.
2. Another expert attempts it and provides feedback.
3. The writer has an opportunity to revise it.
4. A second expert validates the question, followed by non-expert attempts that test whether it remains difficult outside the specialty.

Chen emphasizes the author, reviewer, and adjudication roles, along with the opportunity for revision. The point is not merely to collect votes: a disagreement can expose a defective question that needs repair.

The incentives matter too. Chen highlights agreement-dependent payouts; GPQA’s incentive design more specifically rewards questions that experts answer correctly while non-experts struggle with them. That combination encourages answerable difficulty. It borrows the adversarial spirit of peer review while making task quality an explicit object of design.

A collection of excellent tasks can still be a poor benchmark if it samples the wrong things. Begin with a clear domain taxonomy, then distribute tasks deliberately. There are at least two legitimate targets:

• Representative workloads: Use production traces or traffic to reflect what an agent usually encounters.
• Consequential failures: Allocate explicit coverage to rare situations whose failure cost makes them important.

Self-driving illustrates the second target. Yellow lights, pedestrians, and motorcyclists may appear less frequently than ordinary driving conditions, but their importance is not proportional to their frequency. A benchmark designer must decide which distribution the score is intended to summarize.

MMLU organizes knowledge evaluation across 57 academic and professional subjects, including STEM and the humanities. Chen attributes part of its endurance to this deliberate taxonomy: broad coverage was designed into the benchmark rather than assumed to emerge from collecting enough questions.

A useful frontier benchmark should remain unsaturated, expose real capability weaknesses, and distinguish models that otherwise look similar. Merely making tasks harder is insufficient. Human-solvable tasks offer a reference point: they identify a capability or efficiency that exists but that models have not yet acquired.

Chen connects ARC’s progress to the o1-style reasoning push over the preceding 18–24 months. The version history matters here: the long history and initial o1/o3 breakthroughs belong to ARC-AGI-1; ARC-AGI-2 was introduced in March 2025. The underlying design lesson is to leave headroom for a meaningful change in capability, so a major improvement represents something more than squeezing another point out of an already saturated test.

Chen reports that frontier models scored below 1% at ARC-AGI-3’s launch, while every task was human-solvable to some degree. ARC-AGI-3’s launch result concerns scoring against human action efficiency in interactive environments, not a simple percentage of tasks completed or a statement about current model performance. This human–model gap is what makes a new result on the benchmark informative.

Accuracy alone does not describe an agent’s usefulness. Cost, latency, reasoning-trace quality, intermediate steps, and tool use may all matter to the capability being evaluated. Where appropriate, these dimensions can also supply supervision or reward signals. The hard requirement is measurement validity: a reproducible evaluation must actually measure what it claims to measure.

τ-bench makes this concrete through multi-turn interactions with a simulated user. The agent must complete the task while respecting domain policy. Chen’s example is an airline booking: an agent that books the correct flight but violates fare-class rules still fails. Completion and compliance are separate conditions, and success requires both.

That rule can be expressed as a small evaluation predicate. Given checks from an evaluator, the flight example has the correct booking but fails the policy check:

def evaluate_booking(checks: dict[str, bool]) -> dict[str, bool]:
    completed = checks["correct_flight_booked"]
    compliant = checks["fare_class_rules_followed"]
    return {
        "task_completed": completed,
        "policy_compliant": compliant,
        "success": completed and compliant,
    }

checks = {
    "correct_flight_booked": True,
    "fare_class_rules_followed": False,
}
result = evaluate_booking(checks)
assert result["success"] is False

The predicate exposes the evaluation contract; producing reliable checks for each condition is the substantive benchmark-design work. A completion-only score would discard the very constraint that makes the booking acceptable in practice.

Sound measurement establishes credibility. A field-shaping benchmark adds a research thesis: which capability deserves attention, and why will it matter? The thesis can revisit familiar capabilities under a new set of assumptions or make a broader prediction about how agents will operate.

Terminal-Bench’s bet is that the command line is an interface for general-purpose computer use, not merely a place to write code. Chen points to Claude and Codex building broader enterprise capabilities on coding and CLI tools as evidence that this bet is paying off. The terminal supplies an abstraction through which an agent can interact with the world across many tasks. Measuring that capability gives teams a concrete target to improve, and Chen argues that the benchmark helped accelerate this direction.

A benchmark can also act as a roadmap by making new research questions easier to formulate. SWE-bench starts from a simple software workflow: use existing repository issues and the coding work represented by pull requests to evaluate whether an agent can solve real problems. That starting point supports many subsequent questions about task selection, validation, language coverage, and the information an agent needs.

Chen lists Lite, Verified, Pro, Multilingual, and Multimodal among the benchmark directions that followed. The slide connects SWE-Bench with Lite, Verified, Multilingual, and Multimodal, showing how a single foundation can branch into a research program. Its influence is therefore broader than one leaderboard: it changes which coding-agent problems researchers can isolate and investigate.

The next questions remain open. How should these workflows apply to vibe coding? What should an evaluation measure when developers work at a higher level of abstraction than individual patches? A productive benchmark gives successors something to extend as the underlying work changes.

Researcher UX is part of benchmark design. A benchmark has users who need to run models and agents, contribute tasks, extend the environment, and reuse signals for reinforcement learning or later tuning. Every difficult interface adds friction between an interesting research question and an experiment. Making these operations straightforward is a product requirement, not a finishing touch.

Stanford CRFM’s HELM offers an example of a standardized, modular harness for reproducible evaluation across scenarios and models. Chen also highlights Harbor, describing it as infrastructure shipped with Terminal-Bench 2.0 that has become widely used by agent teams. Both examples point to the same adoption question: how easily can another researcher plug in a system, extend the benchmark, and start improving against it?

The next benchmark opportunities arise where current evaluations simplify work too aggressively. Chen proposes three directions for increasing realism:

• Environment complexity: Represent dynamic operating conditions that professionals and scientists actually encounter.
• Autonomy horizon: Measure how long agents remain reliable, while deliberately choosing where the task sits between a copilot and a fully autonomous agent.
• Output complexity: Evaluate richer work products with differentiated reward signals, including new artifacts for agent–human and agent–agent interaction.

These are design dimensions, not a demand that every benchmark maximize all three. An intentional choice about autonomy, for example, matters as much as increasing task duration.

Consider what surrounds a real coding task. The repository operates under organization-specific policies. Relevant context may live in Slack messages or screenshots. Toolchains are flaky, CI is distributed, and human reviewers carry preferences and knowledge that have never been written down. Meanwhile, several contributors may be changing the system in parallel.

A benchmark that supplies only code and an isolated task captures a fraction of that environment. Chen sees this missing context as a major source of agent failures and an opportunity for better evaluations, both in coding and in other professional domains. Increasing realism means deciding which surrounding constraints and interactions are essential to the work, then representing them in the task environment.

Autonomy horizon asks how long an agent can operate before reliability breaks down. A customer-experience agent may need context supplied weeks earlier. Retaining that context is only the beginning: integrations change, product specifications alter requirements, and reorganizations shift priorities while work is in progress.

The environment therefore cannot always remain fixed while the agent takes more steps. Long-horizon evaluations need to represent changing state and the continual-learning problem that comes with it. An agent must carry useful information forward while adapting to requirements that are no longer the same as when the task began.

More representative work also requires more nuanced judgments. Software, strategic recommendation reports, proposals, and roadmaps do not all have a single easily verified answer. What makes a recommendation good may depend on organizational context and informed human judgment. Benchmarks need to capture these distinctions in signals useful for both evaluation and training rewards.

Trustworthiness adds another output requirement: the agent should recognize and communicate its own uncertainty. Sometimes the appropriate behavior is to stop or ask for more information. Evaluating that decision expands the notion of a good output beyond producing a polished plain-text answer; it includes knowing when the available evidence does not support continuing.

The closing invitation is to examine the work already in front of you: which unresolved capability could become a meaningful benchmark? At the time of the talk, the Open Benchmarks Grants program was still accepting proposals, and Chen invited builders to apply or contact him directly. The aim is to turn practical experience into a public target that helps determine what agents should become capable of next.