Rethinking Environments for Long Horizon Work

When an agent reaches a sixteen-hour task horizon, what has actually been measured: how long it runs, how much work it completes, or how difficult that work is? That distinction sets up this discussion of reinforcement-learning environments by two Theta Software co-founders: the company’s CTO and Rayan Garg, its CEO. Garg introduces his background as a founding engineer at DeepSilicon, researching ternary models.

As autonomous capabilities improve, the boundary of long-horizon work moves. A task that looked long a year ago may no longer stretch a frontier model. Long horizon is a relative measure, not a permanent category of tasks. The opening chart presents the accelerating trend; interpreting it requires a definition of the quantity being plotted.

METR’s task-completion time horizon uses human task duration as the reference. In the talk’s hypothetical example, a sixteen-hour horizon at a 50% success threshold refers to tasks that take humans sixteen hours, not an agent running for sixteen hours. More precisely, METR fits a success-probability curve and estimates the human duration at which it reaches the selected threshold. That does not guarantee the same success rate on every task of that duration. The human timing protocol matters, although the talk does not work through it.

A second approach measures work in units native to agents: trajectory tokens, steps, and tool calls. These reveal the computational demands of completing a task, but depend on the model and its harness. The CTO characterizes Codex models as more token-efficient than some Claude models; the useful point is that token counts are not interchangeable across systems. A task consuming 500,000 tokens on a GPT model does not establish how many tokens Claude will need.

Those measurements remain valuable because extended trajectories introduce their own failure modes. An agent may need context compaction, lose coherence across steps, or fail to preserve information needed later. Holding the model generation constant makes changes in these limits easier to interpret. The CTO offers an illustrative GPT-5.5 comparison: a larger context window or an improved compaction endpoint might enable a million-token trajectory. The example describes a possible improvement in autonomy, rather than a measured result. Human duration says something about the work’s human cost; model effort exposes a different technical frontier.

Neither measurement works well alone. Consider a financial analyst changing formatting or color themes throughout a large Excel workbook. Manual editing could take days, while an agent might write a Python script to apply the change throughout the file. The task’s human duration can be large even when its computational solution is straightforward.

For example, this Python operation replaces one explicit solid-fill color while preserving each matching cell’s other fill settings. It illustrates how a repetitive formatting task becomes a traversal over workbook cells:

from copy import copy
from openpyxl import load_workbook

workbook = load_workbook("finance.xlsx")
old_color = "FF4472C4"
new_color = "FF008080"

for sheet in workbook.worksheets:
    for row in sheet.iter_rows():
        for cell in row:
            fill = cell.fill
            if (
                fill.fill_type == "solid"
                and fill.fgColor.type == "rgb"
                and fill.fgColor.rgb == old_color
            ):
                replacement = copy(fill)
                replacement.fgColor = new_color
                cell.fill = replacement

workbook.save("finance-rethemed.xlsx")

This operation targets explicit cell fills; theme-based colors and conditional formatting require different handling. The capability being exercised is finding and applying the right transformation, rather than sustaining days of manual attention.

Human timing also depends on who performs the task and how the measurement is collected. An advertised average of sixteen hours cannot be cleanly compared with another provider’s twenty hours without matching methodology and expert experience. Estimates become especially noisy as tasks approach the frontier of human expertise—work that only the top 10%, 1%, or 0.1% of practitioners can do.

Human and agent workflows are diverging, with different strengths and bottlenecks. Keeping both kinds of measurement preserves information that either one would hide.

Duration is only one dimension of model capability. Garg turns next to environment complexity, beginning with tool coordination: how many external systems must an agent use, and how must it move information between them? Reading a file or a set of code files is different from combining Grafana observability, GitHub CI/CD, AWS CloudWatch logs, and database reads and writes. The challenge includes connecting evidence across those systems.

Adding independent tasks can lengthen a trajectory without creating meaningful dependence between its steps. A stronger test makes earlier decisions influence later decisions through tool use and changes in the environment. The important question is not just how much work remains, but how earlier actions change what the agent should do next.

In codebase analysis, separate investigations can often proceed at once and return to a coordinating agent. In a sequential investigation, a bad query or a misread result can send every downstream step in the wrong direction. Both tasks may be large, but they test different capabilities.

A third dimension is ambiguity: the information supplied through initial instructions and artifacts. Real work rarely begins with complete information. Testing an agent’s ability to explore documents and the surrounding environment therefore matters alongside testing its ability to follow explicit instructions. The trade-off is that exploration creates more possible routes to success—and more valid answers for an evaluator to recognize.

That leads directly to the verifier: how does the environment determine whether the agent’s work was correct and assign a training reward? Math and data-structure coding problems often permit hard checks through proofs, scripts, or test cases. Much economically valuable work is soft-verifiable: correctness cannot be fully captured by those techniques. A judge or critic model can evaluate aspects of the result that require contextual judgment.

The judge has two main sources of evidence:

• Final environment state: What changed, and does the resulting state satisfy the task?
• Trajectory: How did the agent reach that state, and were its intermediate actions valid?

Rubrics guide that assessment when a deterministic verifier would be impractical, brittle, or impossible to write. The final answer alone does not necessarily contain everything needed to assign reward.

A successful-looking output can conceal an invalid route. An agent might escape its sandbox and inspect privileged information, such as a hidden test suite for a coding task. Stronger environment safeguards and verifiers are necessary, but trajectory inspection gives the judge another way to catch that behavior. Passing the tests is not sufficient if the agent obtained the answers through prohibited access.

The opposite failure is making the permitted route too narrow. Requiring a specific sequence of actions collapses the space the agent can explore. A simple judge that compares the result with a reference answer—or the entire run with a sample trajectory—can reject legitimate solutions to ambiguous tasks. The goal is to constrain invalid behavior while leaving room for different correct approaches, rather than enumerating every acceptable path.

Judges are agents too. As environments gain tools and state, the judge needs a harness that can inspect them. Parts of the task agent’s harness—tool integrations and observability in particular—can serve the judge as well. A deployment repair makes the requirement concrete.

The proposed task asks the agent to:

1. Inspect GitHub CI/CD logs and CloudWatch logs.
2. Diagnose the deployment failure.
3. Apply the required code changes.
4. Open a pull request.
5. Trigger a redeployment after the pull request is merged.

These steps describe the repair workflow, not evidence that the repair succeeded.

To verify the outcome, the judge may need to read GitHub or AWS logs after deployment and determine whether the system is actually working. A record of tool calls is insufficient: requesting an operation does not establish its eventual result. The judge therefore needs direct environment access, with permissions that prevent it from changing the thing it is evaluating. Read-only access can allow inspection while blocking accidental mutations or another deployment.

This separates two questions that a long-horizon evaluator must answer: did the agent take an acceptable path, and did that path produce the required state? Open-ended work often requires both trajectory evidence and independent inspection of the outcome.

Direct environment access solves only part of the judge’s problem. Long trajectories can become too large and complex to evaluate by stuffing the entire history into a single context window. The judge needs a way to investigate the relevant parts of a run.

One approach stores the trajectory in a database and uses sub-agents to enrich it. In the deployment example, useful phases are log investigation, code writing, and post-change checking. Attaching phase labels and other step metadata lets the judge retrieve an apparent failure point and examine its surrounding evidence. A queryable trajectory turns a long transcript into something the judge can investigate, including checking whether an apparent failure was actually a failure.

An environment must be learnable, not merely difficult. Reward density depends on the rubric and how the judge applies it. Adding more detail can provide more feedback, but an overloaded rubric may exceed the judge’s ability to score consistently—especially on frontier problems that models cannot yet perform well. Rubric QA, task distribution, and the underlying data all affect whether training produces useful learning or wastes compute.

Two evaluation patterns help address different parts of this problem:

• Deterministic checks plus model judgment: A verifier can collect metrics or produce an artifact for the judge to inspect. Moving into soft-verifiable work does not make deterministic checks obsolete.
• Dynamic evaluation-time rubrics: A judge can assign conditional partial credit by accepting an assumption for the purpose of evaluating subsequent work. The analogy is grading an exam: if the first part is wrong, temporarily assume its result is correct and ask whether the rest follows correctly.

The second pattern distinguishes an early error from additional errors downstream, without declaring the original assumption correct.

Garg closes the rubric discussion with QA. Theta runs gold, no-op, and variance tests, and also considers coverage and expert agreement. The talk names these checks without specifying their procedures. Their importance grows as AI participates in rubric creation, rubric verification, and assistance to experts: increasing task length and complexity creates more places for an evaluator to miss a requirement or apply it inconsistently.

These design choices become concrete in the closing comparison of GDPval, BankerToolBench, and APEX-Agents. Garg questions whether their tasks are long enough to test the relevant frontier and suggests that shorter durations contribute to saturation. That comparison needs a domain boundary: METR’s principal task distribution concerns software, machine learning, and cybersecurity. A finance benchmark’s average human duration alone does not establish that it sits below an applicable model capability frontier.

The scores also answer different questions. GDPval uses expert preference comparisons, while BankerToolBench’s published study reports substantial unmet criteria and a lack of client-ready outputs. Those results should not be collapsed into a single saturation measure. Garg cites approximately 57% pass@1 for APEX-Agents’ investment-banking subset. In APEX-Agents, pass@1 estimates single-run success against every rubric criterion across repeated runs; it does not identify a fixed set of tasks that the model always solves. The cited figure is presented without a model, harness configuration, or evaluation date, so it cannot support a controlled model comparison.

Breadth is a separate concern. Garg characterizes GDPval’s finance Excel tasks as narrow and APEX-Agents’ finance coverage as concentrated in investment banking, pointing to credit, debt, and risk as areas needing more coverage. This is a critique of their finance coverage, not their entire scope: GDPval spans many occupations, and APEX-Agents includes consulting and corporate law alongside banking. For environment designers, the question is whether the task distribution exercises the particular capabilities they want models to learn.

Reward granularity is the final design concern. Garg describes Theta’s rubric detail using “20 different sub-criteria” and “10 different sub-criteria per criteria.” The description emphasizes granularity without establishing a total criterion count. His broader concern is whether benchmark rewards provide enough detailed feedback for training.

That concern should not be read as an absence of granular rewards in existing benchmarks. BankerToolBench’s repository reports an average of 150 rubric criteria. APEX-Agents uses 1–10 criteria and also reports mean criterion completion as a denser signal than all-criteria success. The useful distinction is between a headline score for evaluation and the feedback available to a learner—and, as the earlier rubric discussion establishes, whether the judge can apply that feedback consistently.

Theta reports an average human completion time of 15 hours per finance task over a 50-task sample. Garg says models also expend substantial effort on these tasks and still struggle across the finance domains Theta targets. The closing slide presents task statistics and model results by subdomain using mean@5, which Garg explicitly distinguishes from the preceding scores. Without a shared scoring definition and task distribution, those values should remain separate rather than becoming a cross-benchmark ranking. The closing example brings the measurements together: human effort, model effort, domain coverage, and reward design each describe a different requirement for useful long-horizon training.