Long Tasks and Experienced Open Source Dev Productivity

Suppose compute growth slows around 2028. If the growth of AI capabilities depends proportionally on that input, how much later would an agent reach a task that takes a human one month? Joel Becker opens with two curves: compute over calendar time, and the logarithm of AI task horizon over calendar time. The compute input could mean research spending on compute, experimental compute, or training compute. The output is the human-work duration of tasks an AI can complete, as developed in Measuring AI Ability to Complete Long Tasks.

Now bend the compute curve so that its growth rate halves. Under the proposed proportional relationship, the log-horizon curve also becomes less steep. A horizontal line marking a one-month task horizon intersects the slower trajectory much later. A modest change in the growth rate can produce a large delay at a distant capability milestone. The chart illustrates that conditional delay; it does not establish that compute growth will actually halve in 2028.

Why might compute growth slow? Becker distinguishes physical constraints, including power, from the willingness and ability to pay. He describes forecasts in which physical limits may remain manageable through 2030, while suggesting that financial limits could become more important: even large technology companies and nation states cannot keep increasing expenditure indefinitely. The economic link is itself contestable. Becker considers proportional growth a reasonable business-as-usual model only while a software-only singularity—self-reinforcing progress without a corresponding expansion of physical inputs—remains unavailable.

An audience member raises the obvious exception: an architectural advance could improve capabilities without a proportional increase in compute. Becker treats straight lines on log-linear plots as useful forecasting defaults because they have persisted across substantial changes in scale, but allows for breaks in either direction. Slower inputs could flatten the line; another transformer-like advance or a software-only singularity could steepen it. There is also a measurement problem: if horizons keep doubling, constructing and validating sufficiently long tasks eventually becomes difficult within the time available between evaluations.

The word time can obscure what is being measured. The historical horizon is a human task duration associated with a specified agent success rate—50% in the original metric—not how long the agent runs. Becker says successful agents tend to accomplish their work relatively early in a run, so their elapsed time can be treated as small for this comparison, though never literally zero. Faster execution may therefore offer less headroom than improved reliability.

These evaluations do not measure the duration of a human–AI collaboration loop. Humans work without AI, and agents work without human assistance. That separation matters when comparing autonomous capability with the productivity of people using coding tools.

The discussion turns to METR’s experienced open-source developer experiment, where allowing AI produced a slowdown despite expectations of acceleration. Could unfamiliarity with the tool explain the result? An audience member recalls a Meta presentation measuring actual human effort per pull request and observing a J-shaped adoption curve: an initial loss followed by a gain. They cannot recall whether the transition took three or six months.

The mechanism is plausible. Experimenting with a tool takes time; that investment may later produce better prompting, better task selection, or a more effective workflow. Improving models can also make an earlier investment worthwhile even if the user’s proficiency changes little. Becker accepts this explanation in principle, but doubts that it accounts for the full discrepancy between expectations and measured outcomes. Follow-up work is intended to test how those effects evolve.

The audience distinguishes two survey questions: how productive someone felt, and how long a task actually took. One participant argues that duration recall is particularly unreliable, even when perceived-productivity responses correlate with other measurements. For Becker, the gap matters because researchers sometimes propose estimating R&D acceleration simply by asking developers or scientists how much faster AI makes them. Perceived acceleration cannot substitute for measuring completed work and the time it costs.

The expert forecasters had been told about the developers’ experience. Some anticipated smaller benefits because experienced maintainers already work efficiently or because large repositories are hard for AI to navigate. Becker says they did not emphasize tool familiarity. His spoken recollection describes widespread prior LLM use but much less Cursor familiarity; the paper’s recorded experience categories and proportions differ from that recollection. The useful distinction is between experience with LLM-assisted development and experience with a particular editor. After watching extensive screen recordings, Becker saw generally reasonable prompting, sometimes better than his own, rather than a conspicuous missing advanced workflow.

Participants nevertheless describe real skills acquired through use: recognizing when to plan, narrowing a task the model cannot handle, and thinking through an architectural decision before it causes trouble many conversation turns later. A familiarity threshold could matter even if it is hard to see in the original sixteen-person sample. The follow-up faces an additional identification problem: users are gaining experience while the models themselves improve.

Repository choice changes the comparison. An audience member proposes that mature open-source projects survive partly by being navigable enough for people to join and contribute. Enterprise software can persist despite poor documentation or awkward structure because it makes money. Another participant describes Claude Code being especially useful in an unfamiliar legacy repository with little documentation and an owner only intermittently available. In that setting, discovering the codebase is part of the work AI can help with; an expert maintainer may already possess that context.

The familiarity plots offer several ways to test the learning explanation. Grouping developers by prior AI experience, including whether Cursor was their primary IDE, produces little movement in the point estimates. Removing early AI-assisted issues also fails to reveal a clear improvement as developers accumulate exposure. But these are small subsets with wide uncertainty bars. Becker explicitly treats them as weak evidence, insufficient to rule out a learning curve.

One exposure-hours plot looks more compatible with a J-curve because its highest-experience bucket shows some acceleration. The bucket boundaries are less definitive than they appear, however. Becker describes a developer placed in the 30–50-hour bucket who had used Cursor as their primary IDE in 2024, logged 140 hours in time-tracking software, and conservatively estimated 50 hours of relevant use. Moving that person to the final bucket would, he says, bring its estimated effect back toward zero. An audience member counters that the people already in the final bucket may also have conservatively estimated their exposure. Becker accepts that possibility: the apparent shape is sensitive to measurement choices, and J-curve explanations remain open.

Expert maintainers working on mature repositories are a narrow population. That limits what the result says about other developers, unfamiliar projects, or new applications. It also makes the population relevant to a particular question: whether AI can accelerate an entire R&D process. Becker’s bottleneck argument is that important categories of work left unaccelerated can constrain the overall gain. Production inference codebases may resemble mature open-source projects more closely than disposable experiment scripts do. A larger follow-up includes greenfield work, but size alone will not make every result generalizable.

Participants raise concerns about vendor-associated productivity research and its methods. Becker acknowledges that much published work is affiliated with large technology companies, without establishing that affiliation as grounds to dismiss it. The audience also offers a defense of the small expert sample: holding expertise relatively constant removes some sources of variance and uncertainty about whether participants know how to do the underlying work. A study can have a useful internal comparison while remaining narrow in whom it represents.

To probe the brownfield-versus-greenfield explanation, Becker describes an unpublished hackathon in which half the teams were randomized to AI access and multiple judges scored each project. The score distributions overlapped substantially. Becker recalls an adjusted advantage of about four percentile points for AI-allowed teams in the unpublished hackathon. He immediately qualifies that recollection: the effective sample was smaller, the estimates were extremely noisy, the analysis had not completed publication review, and he might have made an error. Nonpublication also involved unspecified organizational reasons. The experiment therefore remains a tentative probe, not a settled greenfield productivity result.

Coding is only one component of AI research. Becker would like to study more conceptual work, potentially with mathematics PhD students, different developer populations, or teams inside major AI and technology companies. Tasks performed by fewer people may also be less represented in training data. The purpose is to find close analogies to AI-company work and obtain evidence with different strengths and weaknesses—not to claim those studies have already been conducted.

Enterprise analytics supplies a concrete candidate. An audience member recalls roughly 5,000 LinkedIn tables with impressions in their names, across datasets containing trillions of rows. An analyst asked to count page impressions must first identify which source is authoritative. Better SQL generation does not resolve that question. Becker connects the example to tacit knowledge locked inside companies: general models have been highly effective, but organization-specific data could create a role for specialized adaptation or fine-tuning.

Even identifying the table may not be enough. The participant describes a field called time_started whose meaning changes over time: first a date, then only a month, then seconds associated with when something finished. Correct analysis requires knowing the undocumented cutoff dates. One possible recovery strategy is to inspect SQL written by experienced analysts and work backward from existing reports, extracting the corrections those queries already encode. The missing dependency is a data contract, not query syntax.

Why does this remain unfixed? The immediate assignment is often to deliver an executive report, not to repair shared infrastructure. The discussion argues for explicit ownership of data specifications, infrastructure work, and fixes at the source. Otherwise, a problem deemed too complicated for the organization to solve upstream becomes the responsibility of individual analysts downstream. This is particularly relevant to ML work outside LLM development, where participants describe feature curation and cleaning as major parts of the job. Their hypothesis is that AI may be helpful with SQL, Pandas, or Polars while contributing much less to the complete analytical task; they do not offer a comprehensive benchmark establishing that result.

Consider the audience’s example: calculate the P90 time between deployments at Capital One. The request then expands to a breakdown by team hierarchy, an account of deployment start and end times, and the number and identity of pull requests in each deployment. After a recording-awareness interruption, the participant frames the missing-data details as an imagined deployment system. The general workflow exposes several dependencies:

1. Define the interval. Decide which deployment events establish the time between deployments and how ambiguous telemetry maps to those events.
2. Resolve organizational ownership. Locate the team hierarchy and join deployments to it.
3. Recover change membership. Determine which PRs belong to each deployment, including their counts.
4. Reach beyond the warehouse. If deployment records lack the necessary information, obtain additional evidence through a source such as the GitHub API.

The percentile is the final aggregation. Most of the difficult work precedes it.

For the narrow calculation, suppose a canonical deployments table already contains one row per deployment, a stable service_id, a resolved team_id, and an authoritative started_at. In PostgreSQL, the requested percentile can then be expressed directly:

WITH deployment_gaps AS (
    SELECT
        team_id,
        started_at - LAG(started_at) OVER (
            PARTITION BY service_id
            ORDER BY started_at, deployment_id
        ) AS gap
    FROM deployments
)
SELECT
    team_id,
    percentile_cont(0.9) WITHIN GROUP (
        ORDER BY EXTRACT(EPOCH FROM gap)
    ) AS p90_seconds_between_deployments
FROM deployment_gaps
WHERE gap IS NOT NULL
GROUP BY team_id;

This formulation measures gaps within each service and attributes each gap to the later deployment’s team. Different ownership or interval definitions require different logic. It makes the prerequisite visible: an agent needs authoritative event semantics and relationships before generating the aggregation.

The response proposes spending a day encoding the relevant context in a Cursor Rules file: where the hierarchy lives, how to interpret the records, and which systems to consult. That could turn repeated discovery into reusable instructions, but its effectiveness is a research question. Participants call for studies of actual data-scientist work, discuss access to large companies, and mention possible contacts involved in internal evaluations.

Lawyers, doctors, and mathematicians offer other populations in which capability and adoption may diverge. The discussion separates legal restrictions from organizational resistance. Restrictions can remain binding over time; a resistant firm may eventually face competition from a new entrant that organizes work differently. Trust and prior exposure can still shape how professionals use the tools today.

The audience contrasts a newly trained lawyer familiar with ChatGPT with a veteran who has decades of reusable briefs and knowledge of particular judges. Those are different starting points for assistance: producing text is only part of the veteran’s workflow. Discovery presents a more concrete opportunity because finding and reviewing evidence is expensive. Harvey is mentioned without a claim about demonstrated success. Cheap, fast, reliable discovery could change the cost of litigation, but available technology and its adoption remain separate questions.

Returning to the experiment, Becker resolves a misunderstanding about the experience plot. Its highest bucket means more than fifty hours of accumulated Cursor exposure, combining prior use with use during the experiment. It does not mean participants received fifty-hour tasks. The tasks were naturally occurring repository issues, not an identical exercise assigned to everyone. Maintainers brought years of project-specific mental context; Becker suspects some already knew how to execute a change before beginning it.

Becker describes developers predicting completion times near two hours without AI and around an hour and a half or less with it; observed AI-allowed tasks instead averaged somewhat above two hours, versus somewhat below two hours without AI. These are approximate readings offered to explain the chart. Tasks were randomized to AI-allowed or AI-disallowed conditions, so the measured direction contradicted the expected direction.

His spoken explanation moves between time ratios and speed multipliers. The distinction can be written precisely. Let T_allowed and T_disallowed denote comparable completion times:

A positive time change means slower completion, while a speed multiplier above one means faster completion. The paper estimates the time effect using a difficulty-adjusted regression, rather than simply dividing the displayed raw averages. Becker’s informal ratio discussion explains the intuition; it is not the full estimator.

An audience member recalls an article in which a journalist with no software-development background used AI to produce a feature and submit a pull request. This raises a different possibility from speeding up experts: AI may enable work someone could not otherwise perform. The discussion returns briefly to the hackathon, which included both more and less experienced participants but yielded noisy judge scores with little separation between conditions.

For the mature repositories in the study, a plausible change is not yet an acceptable change. Becker identifies several costs between generation and submission:

• Correctness: Verify that the change actually solves the issue without breaking something else.
• Project conventions: Check whether the solution follows the repository’s established approach.
• Component interactions: Account for how parts of the system depend on one another.
• Cleanup: Repair the output until it meets the project’s quality bar.

An expert can recognize and perform this work. A newcomer may not know what remains wrong. Becker nevertheless sees real enablement in his own AI-assisted internal PRs: he can produce changes he could not previously have written, while acknowledging that their quality standard differs from these expert contributions.

An audience maintainer adds that developer ergonomics can be sufficient reason to reject an otherwise functional contribution. In their account, open-source maintainers strongly value changes that make future maintenance easier, because maintenance burden drives people away. Business teams may face stronger immediate pressure to ship. Those incentives affect which outputs count as completed work.

The Haskell compiler repository illustrates the demanding review bar. Becker recalls a maintainer debating a contribution for longer than its implementation took, without confidently identifying the person or the frequency of such reviews. Becker reports zero minutes of post-review coding time for the median PR in the study. That does not establish perfect code: it suggests that experienced contributors often anticipate the expected standard before submission. There is also a long tail, including a Haskell case with much more extensive review work.

Keeping evaluations ahead of improving models requires more than extending the task list. Becker expects near-term longer-task construction to remain feasible, but sees increasing difficulty over the next two or three years. Raising the required success rate helps measurement, yet changes what a failure means. His personal assessment of GPT-5 at the time is that it is not existentially dangerous, partly because it cannot perform sufficiently demanding work. If a future system succeeded on very challenging long tasks 90% of the time rather than 99%, merely pointing to the remaining failures would provide a weaker incapability-based safety argument. Other safety arguments would then need to carry more weight.

One proposed extension measures a dangerous-task horizon while the agent is closely monitored. Other models would inspect its work transcripts for suspicious behavior, potentially escalating to humans. The task is then harder in a specific way: the agent must accomplish the dangerous objective while avoiding detection. Becker illustrates the idea with a hypothetical forty-hour horizon without monitoring and a two-hour horizon under close monitoring. The smaller effective horizon would provide additional room to measure capability growth before tasks become prohibitively long.

Becker conjectures that monitoring could reduce the effective dangerous-task horizon by one or two orders of magnitude. That is a proposed research outcome, not a measured reduction. An audience member identifies a communication risk: publishing a capability cost might encourage people to weaken controls. The evaluation therefore needs to make clear which capability it constrains—successful dangerous behavior under oversight—and why that constraint is desirable.

A second extension concerns output quality. Becker considers METR’s algorithmic scoring broader than SWE-bench-style unit-test checks, but still incomplete. An answer can satisfy the immediate task while leaving work that is difficult to extend. Adding that requirement could lower the measured horizon, much as increasing the required success rate does. His illustrative comparison is an hour-long task when future buildability does not matter, versus a thirty-minute task when it does. These more relevant measurements would still need forecasts that account for both compute slowdowns and possible acceleration from AIs improving AI.

Becker sketches an explicitly unofficial history of the measurement program. In his reconstruction, GPT-4 prompted a move from question-answering and exam-style tests toward asking whether a model could act and accomplish tasks. Repeating those tasks across successive models produced time-horizon trends. Randomizing AI access then supplied another view: what happens when people use the tools in real work? He was not present for the earliest period and does not present the reconstruction as institutional history. Its methodological point is that different measurements fail in different ways.

Real agent traces could provide another angle. Cursor, Claude Code, and similar systems leave diffs, action histories, and recorded reasoning. Those traces contain problems that are not packaged for an evaluation, including the ambiguous enterprise columns discussed earlier. Their potential advantage is realism and scale; their weakness is causal interpretation. An observed successful session does not by itself reveal what would have happened without the agent.

AI Village supplies a more open-ended setting. Its agents use computers, interact with people, and attempt fuzzy goals such as organizing an event in a park, running human-subject experiments, or operating a merchandise store. Becker describes frequent failures, while identifying reasons the setup may understate capability: computer-use interfaces were weaker than text or CLI interfaces, and the multi-model village arrangement introduces questions about orchestration.

A more informative version would give strong models appropriate text tools, invest in eliciting their best performance, and remove weaker participants. It would retain the fuzzy goals. The resulting observations could reveal where coherent work breaks down: a successful first step followed by aimless behavior, unproductive inter-agent interactions, failure to use external services, or poor resource management. Those failure mechanisms may help explain why automating an R&D organization remains harder than completing isolated tasks.

The audience frames part of the problem as a mismatch between uneven model abilities and a world designed around human capacities. Pack weights and tax obligations are examples of expectations calibrated to people. A neurodivergence analogy appears in the discussion as a way to describe uneven strengths, not as a diagnosis of models or people. Becker emphasizes that dependence on human assistance is a present limitation, not necessarily a permanent one. Strong performance on difficult GPQA science questions leaves the practical question unresolved: why is accomplishing a goal in the world still so hard?

Another participant compares repeatedly observing agent failure to observing that a car has once again failed to fly to the moon: perhaps the relevant machine has not been built. Becker responds by narrowing the proposed test to text-only work that a human could reasonably perform through a CLI. That reduces the interface objection without removing the need to choose appropriate tasks.

An audience member connects this to an earlier Anthropic presentation: transform a task into something closer to the model’s training distribution. A CLI interface may be more familiar to a model than interacting with Chrome. A harness can remove incidental friction, while appropriately scoped assignments can resemble good management of talented interns. These interventions help establish what the model can do under favorable conditions.

For automating R&D, however, the remaining question is who performs that reformulation. If a human must identify the real problem, find the necessary context, choose the decomposition, and turn each piece into a benchmark-like assignment, substantial work remains outside the agent. Models might eventually become excellent at creating those conditions for themselves. Until then, exceptional performance on well-shaped tasks can coexist with an inability to substitute flexibly for a worker handling whatever arises.

Task duration is only one axis. Text-based work, visual interaction, and physical action may sit at very different capability levels. Becker sees weak evidence that some task distributions could share similar growth rates or doubling times while differing substantially in their starting points. That distinction is visible on a log plot: similar slopes do not imply similar current capability. Vision-dependent work may lag even if its trajectory eventually resembles the text-based trend.

An audience suggestion that tactile tasks would simply score zero raises the elicitation issue again. Becker argues that the evaluation should provide an appropriate interface and try to make the system perform well before concluding that the capability is absent. The conversation touches on spatial judgment and Figure’s robotics work, then on how to define the full collection of capabilities one would expect from general intelligence. Becker mentions prospective work by Basil Halperin and Arjun Ramani without identifying a definite paper. Shared log-growth behavior remains a null hypothesis to investigate, not a finding established across every domain.

The opening compute constraint returns at a larger scale. Becker is skeptical that automating software research alone necessarily produces runaway progress: fixed hardware permits only so many sufficiently productive experiments. Automating chip design and production could change that constraint. A system that improves AI, builds more compute, and constructs the robots needed to keep expanding production would close a much broader feedback loop. A period of slower growth could then be followed by renewed acceleration.

What prevents that loop from closing? Possible bottlenecks include training data, physical hardware, and the intelligence needed to control it. Becker asks whether hypothetical superintelligence operating hardware available today could construct chip-production facilities. He explicitly lacks the manufacturing expertise to answer. Existing robotic steps in fabrication do not settle the question: operating a production machine differs from building that machine, its factory, and the supporting production system.

An audience member offers a counterweight from a software developer who moved into manufacturing protective equipment during COVID and encountered much slower, harder iteration than expected. Becker likewise recalls being surprised by the engineering expertise required in fabs. Physical iteration consumes capital and calendar time as well as reasoning. The audience initially suggests a very long timeline, but the discussion soon makes the uncertainty explicit.

Self-driving provides the analogy: reaching an impressive prototype can be much easier than achieving broadly acceptable reliability. Becker admits that he delayed getting a driver’s license because he expected autonomous cars, while pointing to their expanding deployment as evidence that the final stretch is not necessarily indefinite. Economic incentives could also accelerate robots building robots. When RepRap is offered as a reference point, Becker invokes GPT-2 and the possibility that robotics is at a similarly early stage of a rapid transition. The disagreement ultimately concerns practical mass production at consequential scale, not whether a system can produce a one-off demonstration.

Becker reads the robotics-compute chart as showing roughly similar growth rates but a level about two orders of magnitude below the comparison models. He wonders what closing that gap would produce, while explicitly stopping short of predicting autonomous chip production. More capable robots might emerge before a complete self-sustaining production system does. Additional compute could also support more parameters at a fixed amount of data, rather than requiring data volume to increase in lockstep. The proposed headroom is a reason to investigate scaling, not evidence that compute alone removes the remaining constraints.

The final exchange identifies nearer-term contributions to fabrication. An audience member describes lithography-mask design and checking that a design conforms to its specification as computationally expensive work where AI could help. Another possibility is identifying process parameters drifting out of range before they cause wafer-imaging failures. That would target yield: obtaining more usable output from an expensive and fragile production process. Both are specific forms of assistance within fabrication, rather than automation of the entire factory.

The participant uses chip binning—sorting manufactured parts by their usable characteristics—to illustrate why production quality matters. Their extension of that explanation to all NVIDIA GPU tiers being the same chip is incorrect: NVIDIA’s Blackwell architecture whitepaper identifies distinct silicon for the desktop RTX 5090, 5080, and 5070. The relevant opportunity remains the one immediately preceding that example: help engineers design, verify, and control a production process whose useful output depends on precision.