Why Agent Hype can fall short of reality – Joel Becker, METR

How capable are AI systems today, and how quickly are they improving? Benchmark results and measurements of actual work can give surprisingly different answers. Joel Becker, a researcher at METR, approaches that gap through two studies: Measuring AI Ability to Complete Long Tasks, which measures autonomous task completion, and a randomized developer productivity experiment, which measures what happens when developers can use AI in their existing work.

METR—Model Evaluation and Threat Research—is an independent research nonprofit that informs the public, policymakers and AI labs about potential catastrophic risks. Its evaluation work studies capabilities and propensities; its threat research connects those findings to possible harms. Understanding what a capability measurement actually establishes is therefore part of the underlying research problem.

Consider a percentage score on SWE-bench or GPQA. In Becker’s GPQA illustration, chance performance is 25%, an approximate expert reference is 75%, and the ceiling is 100%. Chance is not the minimum possible score, and the expert reference depends on how answers are assessed: the original GPQA paper reports 65% raw expert accuracy and 74% after discounting retrospectively identified clear mistakes. A model scoring 50% sits halfway between the illustration’s chance and expert references—but that position does not translate directly into a useful description of what it can do.

Even matching the expert reference leaves a question unanswered: does the model match an expert in the setting that matters to you? Meanwhile, Becker observes that benchmarks increasingly move from informative to saturated over a short interval. A score can become harder to interpret just as the benchmark runs out of room to distinguish stronger models.

METR starts with human baseline measurements across tasks of very different difficulty. The humans are relevant experts, but they have little context on the particular task: think of a skilled software engineer in their first day or week on a job, rather than a maintainer who already knows the codebase. The same distinction applies to the machine learning and cybersecurity work in the evaluation.

The task mixture spans a deliberately wide range:

• HCAST: Software tasks requiring autonomy, including tool use, interaction with an environment and reasoning through a problem rather than simply answering a question.
• Software Atomic Actions, or SWAA: Small problems potentially within reach of GPT-2. Becker’s example is identifying which of four files contains passwords when one is named passwords.txt.
• RE-Bench: Novel, open-ended machine learning research engineering challenges that are difficult even for top human experts.

The range matters: a capability scale needs tasks that distinguish weak models as well as tasks that still challenge strong ones.

The measurement procedure then connects human effort to AI performance:

1. Measure how long relevant human experts take to complete the tasks.
2. Evaluate the models on those same tasks under conditions made as identical as possible.
3. Use human completion time as the scale against which to describe autonomous AI success.

The resulting unit is familiar—minutes or hours of human work—but the quantity being estimated is the model’s ability to complete tasks of that difficulty.

In Becker’s illustration, Claude 3.7 Sonnet succeeds on essentially all tasks taking humans roughly four minutes or less, while struggling on the longest tasks. Between those extremes lies a range of partial success. The tendency for success to decline as human task duration increases appears robust across the models in this task distribution. It is an empirical relationship, not a law that every possible task must obey.

METR fits a success curve against human completion time. The 50% time horizon is the duration at which that fitted curve predicts a one-in-two chance of success:

Here, T_human is human completion time and H₅₀ is the model’s time horizon. A longer horizon means success on harder tasks as measured by human time; it does not mean the AI ran autonomously for that long. Becker acknowledges debate about the choice of a 50% threshold.

Repeating the fit for other models produces a comparable quantity:

These are Becker’s approximate readings for this task distribution, measured in human completion time. Plotting each model’s horizon against its release date turns the collection of success curves into a history of capability improvement.

On the logarithmic chart, a straight line represents exponential growth. Becker describes the observed trend as surprisingly steady—more so than METR expected when starting the project. He reports that subsequent updates continued to track it through GPT-5.1-Codex-Max, the latest model shown in the talk.

The benchmark evidence establishes something substantial: models can sometimes complete tasks that are exceedingly difficult for humans, and their capabilities are improving quickly. Becker describes the RE-Bench tasks as beyond his own abilities, while AI systems can make successful attempts at them.

Three limitations complicate the translation into workplace performance:

• Human context: A relevant expert encountering a task for the first time is a different baseline from someone deeply familiar with that work. The latter may finish much faster, reducing the model’s apparent advantage.
• Benchmark ceilings: Becker describes GPQA as approaching saturation. Time horizons help connect task sets across difficulty levels, but they still require harder tasks to measure stronger models. Becker reports that model time horizons in this historical trend doubled approximately every six to seven months, making it difficult to keep extending the underlying evaluations.
• Environmental messiness: METR tasks, SWE-bench and other agentic benchmarks often use contained environments with limited human coordination. Large repositories and work involving the outside world can introduce complications absent from those settings.

A more interpretable scale does not, by itself, eliminate the limitations of the tasks underneath it.

Field experiments offer a complementary measurement. They can study developers who already know their repositories and measure productivity on tasks arising naturally in their work. Productivity also retains room to improve after a benchmark reaches its ceiling: Becker illustrates this with a possible progression from 1.5× or 2× speedups to 3×, 4× or 5×. Those multipliers describe the measurement’s headroom, not observed gains. Real work supplies the messiness that a contained evaluation can leave out.

METR recruited 16 experienced developers working on large, mature open-source projects, each completing about 16 real GitHub issues on average. These were issues from their normal work, with the very longest excluded. The intervention was assigned at the task level: an issue was randomly allocated to either AI disallowed or AI allowed.

The researchers purchased Cursor Pro for participants, who predominantly used it. The study documents Claude 3.5 Sonnet and Claude 3.7 Sonnet as the main models. These are the documented study versions; the spoken model label in this passage differs. Crucially, the treatment was permission to use AI, not an obligation to use it on every task.

For each issue, METR recorded completion time and compared the two randomized conditions. The repositories included the Haskell compiler, scikit-learn and Hugging Face Transformers. Becker describes repositories averaging more than a million lines of code and more than ten years of history. Participants averaged approximately third place among their repositories’ contributors, with about five years of contribution experience. These were developers with unusually deep knowledge of the systems they were changing.

Before revealing the measured result, Becker compares several expectations:

Tasks took 19% more time when AI was allowed than when it was disallowed. The contrast is not merely between outside forecasts and observed behavior: developers’ retrospective impressions also pointed in the opposite direction from the measured effect.

Becker initially suspected a problem with the experiment. Participants could always choose not to use AI, so why would giving them another option make them slower? To investigate, the researchers examined many hours of screen recordings and categorized hypotheses about what was adding time.

The recordings suggested several contributors, without establishing a complete causal decomposition:

• Overoptimism: Developers who expect AI to help may keep using it where its actual contribution does not justify the time spent.
• Implicit repository context and familiarity: An expert may already know the solution. If implementing it is largely a matter of typing, explaining the task to an AI introduces work that direct implementation would not require.
• Large, complex repositories: Their demands can exceed those of the smaller, contained environments used in benchmarks.
• Low reliability: Even useful generated work needs checking, and faulty work needs correction. Becker illustrates the issue with success rates such as 20%, 50% or 80%: none removes the need to inspect the output.

The relevant comparison is the whole assisted workflow against what the developer would otherwise have done, including instruction, review and repair.

One prominent public explanation—below-average skill or effectiveness in using AI tools—remained in the unclear category. Becker describes evidence both for and against it, with the research continuing. It should not be promoted from a possible factor into a settled explanation of the slowdown.

The population also limits what the result says. These were exceptionally experienced contributors working in unusually complex, long-lived repositories. Becker believes AI helped his own work on smaller repositories where he had less of that accumulated expertise. The study’s setting is interesting precisely because it differs from many developers’ everyday circumstances.

The tools belong to early 2025. Becker emphasizes March; the full paper covers February–June and reports that most issues were completed in February and March. Given the rapid capability gains already discussed, he leaves open whether the result had changed by the time of the talk.

The apparent puzzle extends beyond this one experiment. Becker places labor-market effects alongside field studies as economic evidence that can look less impressive than benchmark progress. Before trying to reconcile the two, he considers whether features of the study could explain its result.

• Learning the tools: Participants might benefit more as they gain familiarity during continued experiments. Becker is skeptical that this explains the result, but keeps the possibility open.
• Payment incentives: Developers were paid by the hour, a choice intended to resemble real employment conditions. That could weaken incentives to finish quickly, although Becker did not see behavior in the recordings that persuaded him this was happening.
• Study size: A small study should produce a restrained update in beliefs. METR was continuing the experiment and pursuing larger studies, rather than treating the initial estimate as the last word.

These possibilities concern how much confidence to place in the field result. The next question is how the two kinds of evidence could coexist even if the result holds.

First, the reliability required to save time may be much higher than the reliability needed to score well on a benchmark. Becker suggests that correctness around 95–99% might be necessary for developers to accept suggestions rapidly without spending substantial time verifying them. That is a proposed explanation, not a measured threshold from the experiment. A model that often solves a problem can still leave its user with expensive review work.

Second, passing an automated evaluation and producing mergeable code are different standards. SWE-bench-style scoring is inexpensive to apply at the margin, but unit tests do not fully capture whether other developers can build on a patch or whether it meets the repository’s broader quality expectations.

A model can improve on the first while still imposing substantial human work to satisfy the second.

Other differences change both sides of the comparison. A highly familiar maintainer supplies a stronger human baseline than an expert new to the task. Real issues also come from a messier task distribution. And the agent around a model matters: METR invests substantial effort and token expenditure in eliciting strong performance, while Cursor at the time of the study may not have elicited the same capabilities. A benchmark result therefore reflects the evaluation setup as well as the underlying model.

Finally, subtasks are not necessarily independent. Suppose a human can complete tasks A and B, while an AI can complete A faster but cannot complete B. Delegating A seems attractive until B requires knowing how A was done. The human may have to reconstruct that context before proceeding, making it rational to do both tasks personally. Becker believes this need to maintain context across subtasks contributes to the gap. Faster completion of one task does not guarantee faster completion of the work that depends on it.

METR’s next research steps follow both lines of evidence: longer autonomous tasks, more ambitious randomized experiments and additional measurements that help triangulate AI capabilities. Becker closes with an invitation to contribute to that work through METR careers. At the time of the recording, he highlights research engineers, research scientists and a director of operations, explicitly welcoming people with scrappy startup experience as well as academic backgrounds.