# Rethinking Environments for Long Horizon Work

Rayan Garg

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/2aS7aKoXn64-rethinking-environments-long-horizon-work) · [Video](https://www.youtube.com/watch?v=2aS7aKoXn64)

## Rethinking Environments for Long-Horizon Work

Longer tasks are only useful training environments when their decisions have consequences, their outcomes can be verified, and their rewards teach models something.

Familiarity with tool-using agents, reinforcement-learning rewards, and basic software deployment workflows will help.

### What does a sixteen-hour task horizon mean?

[0:18](https://www.youtube.com/watch?v=2aS7aKoXn64&t=18s)

When an agent reaches a sixteen-hour task horizon, what has actually been measured: how long it runs, how much work it completes, or how difficult that work is? That distinction sets up this discussion of reinforcement-learning environments by two [Theta Software](https://thetasoftware.com/) co-founders: the company’s CTO and Rayan Garg, its CEO. Garg introduces his background as a founding engineer at DeepSilicon, researching ternary models.

As autonomous capabilities improve, the boundary of long-horizon work moves. A task that looked long a year ago may no longer stretch a frontier model. **Long horizon is a relative measure, not a permanent category of tasks.** The opening chart presents the accelerating trend; interpreting it requires a definition of the quantity being plotted.

![Slide titled “Task Horizons Are Accelerating” with model names, release dates, and time horizons increasing from 1 hr to 12 hrs.](https://ai.engineer/api/talk-essay-media/b7831e77e29eb5a9dd3a33ce640713f378a7e1bc67e4418a730daabe9f365a58)

Task horizons across four Claude models, as presented in the talk.

[METR’s task-completion time horizon](https://metr.org/time-horizons/) uses human task duration as the reference. In the talk’s hypothetical example, a sixteen-hour horizon at a 50% success threshold refers to tasks that take humans sixteen hours, not an agent running for sixteen hours. More precisely, METR fits a success-probability curve and estimates the human duration at which it reaches the selected threshold. That does not guarantee the same success rate on every task of that duration. The human timing protocol matters, although the talk does not work through it.

### Measure the model’s effort too

[2:28](https://www.youtube.com/watch?v=2aS7aKoXn64&t=148s)

A second approach measures work in units native to agents: trajectory tokens, steps, and tool calls. These reveal the computational demands of completing a task, but depend on the model and its harness. The CTO characterizes **Codex** models as more token-efficient than some **Claude** models; the useful point is that token counts are not interchangeable across systems. A task consuming 500,000 tokens on a **GPT** model does not establish how many tokens Claude will need.

Those measurements remain valuable because extended trajectories introduce their own failure modes. An agent may need context compaction, lose coherence across steps, or fail to preserve information needed later. Holding the model generation constant makes changes in these limits easier to interpret. The CTO offers an illustrative **GPT-5.5** comparison: a larger context window or an improved compaction endpoint might enable a million-token trajectory. The example describes a possible improvement in autonomy, rather than a measured result. Human duration says something about the work’s human cost; model effort exposes a different technical frontier.

### A spreadsheet can take days—or a script

[4:15](https://www.youtube.com/watch?v=2aS7aKoXn64&t=255s)

Neither measurement works well alone. Consider a financial analyst changing formatting or color themes throughout a large Excel workbook. Manual editing could take days, while an agent might write a Python script to apply the change throughout the file. The task’s human duration can be large even when its computational solution is straightforward.

For example, this Python operation replaces one explicit solid-fill color while preserving each matching cell’s other fill settings. It illustrates how a repetitive formatting task becomes a traversal over workbook cells:

```python
from copy import copy
from openpyxl import load_workbook

workbook = load_workbook("finance.xlsx")
old_color = "FF4472C4"
new_color = "FF008080"

for sheet in workbook.worksheets:
    for row in sheet.iter_rows():
        for cell in row:
            fill = cell.fill
            if (
                fill.fill_type == "solid"
                and fill.fgColor.type == "rgb"
                and fill.fgColor.rgb == old_color
            ):
                replacement = copy(fill)
                replacement.fgColor = new_color
                cell.fill = replacement

workbook.save("finance-rethemed.xlsx")
```

This operation targets explicit cell fills; theme-based colors and conditional formatting require different handling. The capability being exercised is finding and applying the right transformation, rather than sustaining days of manual attention.

Human timing also depends on who performs the task and how the measurement is collected. An advertised average of sixteen hours cannot be cleanly compared with another provider’s twenty hours without matching methodology and expert experience. Estimates become especially noisy as tasks approach the frontier of human expertise—work that only the top 10%, 1%, or 0.1% of practitioners can do.

| Measure | What it helps reveal | What must be controlled |
|---|---|---|
| Human completion time | Effort for a human reference population | Expertise and timing protocol |
| Trajectory tokens | Model effort and context pressure | Model, harness, and compaction |
| Steps and tool calls | Interaction burden | Tool interfaces and task decomposition |

Human and agent workflows are diverging, with different strengths and bottlenecks. Keeping both kinds of measurement preserves information that either one would hide.

### Long work needs consequential dependencies

[6:36](https://www.youtube.com/watch?v=2aS7aKoXn64&t=396s)

Duration is only one dimension of model capability. Garg turns next to **environment complexity**, beginning with tool coordination: how many external systems must an agent use, and how must it move information between them? Reading a file or a set of code files is different from combining Grafana observability, GitHub CI/CD, AWS CloudWatch logs, and database reads and writes. The challenge includes connecting evidence across those systems.

Adding independent tasks can lengthen a trajectory without creating meaningful dependence between its steps. A stronger test makes earlier decisions influence later decisions through tool use and changes in the environment. **The important question is not just how much work remains, but how earlier actions change what the agent should do next.**

| Structure | Example | Source of difficulty |
|---|---|---|
| Parallelizable work | Sub-agents inspect separate code files and return findings | Dividing and combining independent analysis |
| Sequential work | An agent investigates dashboards and logs | Early queries and interpretations shape later actions |

In codebase analysis, separate investigations can often proceed at once and return to a coordinating agent. In a sequential investigation, a bad query or a misread result can send every downstream step in the wrong direction. Both tasks may be large, but they test different capabilities.

![Slide titled “Environment Complexity — State Changes” with three takeaways contrasting unrelated tasks, cascading early decisions, and parallelizable documentation work.](https://ai.engineer/api/talk-essay-media/d554458ec5bf3614c4ac059a0519142e97e289753d1d650f66fc08ff816ec1eb)

State changes distinguish consequential task complexity from independent work.

### Exploration makes correctness harder to judge

[9:15](https://www.youtube.com/watch?v=2aS7aKoXn64&t=555s)

A third dimension is **ambiguity**: the information supplied through initial instructions and artifacts. Real work rarely begins with complete information. Testing an agent’s ability to explore documents and the surrounding environment therefore matters alongside testing its ability to follow explicit instructions. The trade-off is that exploration creates more possible routes to success—and more valid answers for an evaluator to recognize.

That leads directly to the verifier: how does the environment determine whether the agent’s work was correct and assign a training reward? Math and data-structure coding problems often permit hard checks through proofs, scripts, or test cases. Much economically valuable work is **soft-verifiable**: correctness cannot be fully captured by those techniques. A judge or critic model can evaluate aspects of the result that require contextual judgment.

The judge has two main sources of evidence:

- **Final environment state:** What changed, and does the resulting state satisfy the task?
- **Trajectory:** How did the agent reach that state, and were its intermediate actions valid?

Rubrics guide that assessment when a deterministic verifier would be impractical, brittle, or impossible to write. The final answer alone does not necessarily contain everything needed to assign reward.

### Catch reward hacking without prescribing one solution

[12:02](https://www.youtube.com/watch?v=2aS7aKoXn64&t=722s)

A successful-looking output can conceal an invalid route. An agent might escape its sandbox and inspect privileged information, such as a hidden test suite for a coding task. Stronger environment safeguards and verifiers are necessary, but trajectory inspection gives the judge another way to catch that behavior. Passing the tests is not sufficient if the agent obtained the answers through prohibited access.

The opposite failure is making the permitted route too narrow. Requiring a specific sequence of actions collapses the space the agent can explore. A simple judge that compares the result with a reference answer—or the entire run with a sample trajectory—can reject legitimate solutions to ambiguous tasks. The goal is to constrain invalid behavior while leaving room for different correct approaches, rather than enumerating every acceptable path.

### A deployment judge needs to inspect the deployment

[13:28](https://www.youtube.com/watch?v=2aS7aKoXn64&t=808s)

**Judges are agents too.** As environments gain tools and state, the judge needs a harness that can inspect them. Parts of the task agent’s harness—tool integrations and observability in particular—can serve the judge as well. A deployment repair makes the requirement concrete.

The proposed task asks the agent to:

1. Inspect GitHub CI/CD logs and CloudWatch logs.
2. Diagnose the deployment failure.
3. Apply the required code changes.
4. Open a pull request.
5. Trigger a redeployment after the pull request is merged.

These steps describe the repair workflow, not evidence that the repair succeeded.

To verify the outcome, the judge may need to read GitHub or AWS logs after deployment and determine whether the system is actually working. A record of tool calls is insufficient: requesting an operation does not establish its eventual result. The judge therefore needs direct environment access, with permissions that prevent it from changing the thing it is evaluating. Read-only access can allow inspection while blocking accidental mutations or another deployment.

This separates two questions that a long-horizon evaluator must answer: did the agent take an acceptable path, and did that path produce the required state? Open-ended work often requires both trajectory evidence and independent inspection of the outcome.

### Make the trajectory queryable

[15:27](https://www.youtube.com/watch?v=2aS7aKoXn64&t=927s)

Direct environment access solves only part of the judge’s problem. Long trajectories can become too large and complex to evaluate by stuffing the entire history into a single context window. The judge needs a way to investigate the relevant parts of a run.

One approach stores the trajectory in a database and uses sub-agents to enrich it. In the deployment example, useful phases are log investigation, code writing, and post-change checking. Attaching phase labels and other step metadata lets the judge retrieve an apparent failure point and examine its surrounding evidence. **A queryable trajectory turns a long transcript into something the judge can investigate**, including checking whether an apparent failure was actually a failure.

![“Scaling Agent Trajectories” slide separates the context-window bottleneck from recommendations to query trajectories and cross-reference environment state.](https://ai.engineer/api/talk-essay-media/f45717b048ecde89f28a4416959983b6dfbd71c758ed8c3a55f64c7d76928fa4)

Queryable trajectories let judges retrieve relevant turns, tools, and failure points.

### Dense rewards still need consistent judgment

[16:32](https://www.youtube.com/watch?v=2aS7aKoXn64&t=992s)

An environment must be learnable, not merely difficult. Reward density depends on the rubric and how the judge applies it. Adding more detail can provide more feedback, but an overloaded rubric may exceed the judge’s ability to score consistently—especially on frontier problems that models cannot yet perform well. Rubric QA, task distribution, and the underlying data all affect whether training produces useful learning or wastes compute.

Two evaluation patterns help address different parts of this problem:

- **Deterministic checks plus model judgment:** A verifier can collect metrics or produce an artifact for the judge to inspect. Moving into soft-verifiable work does not make deterministic checks obsolete.
- **Dynamic evaluation-time rubrics:** A judge can assign conditional partial credit by accepting an assumption for the purpose of evaluating subsequent work. The analogy is grading an exam: if the first part is wrong, temporarily assume its result is correct and ask whether the rest follows correctly.

The second pattern distinguishes an early error from additional errors downstream, without declaring the original assumption correct.

Garg closes the rubric discussion with QA. Theta runs gold, no-op, and variance tests, and also considers coverage and expert agreement. The talk names these checks without specifying their procedures. Their importance grows as AI participates in rubric creation, rubric verification, and assistance to experts: increasing task length and complexity creates more places for an evaluator to miss a requirement or apply it inconsistently.

### Read finance benchmarks through their task and metric choices

[18:25](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1105s)

These design choices become concrete in the closing comparison of [GDPval](https://openai.com/index/gdpval/), [BankerToolBench](https://arxiv.org/abs/2604.11304), and [APEX-Agents](https://arxiv.org/html/2601.14242v1). Garg questions whether their tasks are long enough to test the relevant frontier and suggests that shorter durations contribute to saturation. That comparison needs a domain boundary: METR’s principal task distribution concerns software, machine learning, and cybersecurity. A finance benchmark’s average human duration alone does not establish that it sits below an applicable model capability frontier.

The scores also answer different questions. GDPval uses expert preference comparisons, while BankerToolBench’s published study reports substantial unmet criteria and a lack of client-ready outputs. Those results should not be collapsed into a single saturation measure. Garg cites approximately 57% pass@1 for APEX-Agents’ investment-banking subset. In APEX-Agents, pass@1 estimates single-run success against every rubric criterion across repeated runs; it does not identify a fixed set of tasks that the model always solves. The cited figure is presented without a model, harness configuration, or evaluation date, so it cannot support a controlled model comparison.

Breadth is a separate concern. Garg characterizes GDPval’s finance Excel tasks as narrow and APEX-Agents’ finance coverage as concentrated in investment banking, pointing to credit, debt, and risk as areas needing more coverage. This is a critique of their finance coverage, not their entire scope: GDPval spans many occupations, and APEX-Agents includes consulting and corporate law alongside banking. For environment designers, the question is whether the task distribution exercises the particular capabilities they want models to learn.

### The training signal behind the headline result

[19:59](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1199s)

Reward granularity is the final design concern. Garg describes Theta’s rubric detail using “20 different sub-criteria” and “10 different sub-criteria per criteria.” The description emphasizes granularity without establishing a total criterion count. His broader concern is whether benchmark rewards provide enough detailed feedback for training.

That concern should not be read as an absence of granular rewards in existing benchmarks. [BankerToolBench’s repository](https://github.com/Handshake-AI-Research/bankertoolbench) reports an average of 150 rubric criteria. APEX-Agents uses 1–10 criteria and also reports mean criterion completion as a denser signal than all-criteria success. The useful distinction is between a headline score for evaluation and the feedback available to a learner—and, as the earlier rubric discussion establishes, whether the judge can apply that feedback consistently.

Theta reports an average human completion time of 15 hours per finance task over a 50-task sample. Garg says models also expend substantial effort on these tasks and still struggle across the finance domains Theta targets. The closing slide presents task statistics and model results by subdomain using `mean@5`, which Garg explicitly distinguishes from the preceding scores. Without a shared scoring definition and task distribution, those values should remain separate rather than becoming a cross-benchmark ranking. The closing example brings the measurements together: human effort, model effort, domain coverage, and reward design each describe a different requirement for useful long-horizon training.

![Theta slide with a task-statistics table above a Mean@5 table comparing four models across seven finance subdomains.](https://ai.engineer/api/talk-essay-media/9f51fb5c68441f9b8c0fb343667fa3202f21ba5875ac3bbc98a7132a64152e60)

Theta’s reported finance-task statistics and Mean@5 results by subdomain.

## Resources

- [METR task-completion time horizons](https://metr.org/time-horizons/) — Explains human-time baselines, fitted success thresholds and the limits of interpreting agent time horizons.
- [Introducing GDPval](https://openai.com/index/gdpval/) — Introduces professional-work evaluation across 44 occupations, including its expert grading process and one-shot limitations.
- [BankerToolBench](https://arxiv.org/abs/2604.11304) — Evaluates investment-banking workflows involving data rooms, financial tools and multiple deliverables against detailed expert rubrics.
- [BankerToolBench evaluation code](https://github.com/Handshake-AI-Research/bankertoolbench) — Open-source code and configuration for evaluating agents on BankerToolBench tasks.
- [APEX-Agents: original methodology](https://arxiv.org/html/2601.14242v1) — Describes cross-application tasks in banking, consulting and law, including human baselines, rubric judging and distinct success metrics.

## Slides

### Rethinking Environments for Long Horizon Tasks

Theta introduces RL environments in the context of long-horizon tasks.

[0:18](https://www.youtube.com/watch?v=2aS7aKoXn64&t=18s) [Title slide for Theta’s talk on long-horizon tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca8c691e4f879e17db95bcbc12e01a1979d64e0251fea6263bbd2f66dde38a9c.webp)

DATA QUALITY · AI Engineer World’s Fair 2026

Rethinking Environments for
Long Horizon Tasks

Theta

### Task Horizons Are Accelerating

Across four Claude releases dated February 2025 to February 2026, the slide lists METR time horizons rising from 1 hour to 12 hours.

[0:43](https://www.youtube.com/watch?v=2aS7aKoXn64&t=43s) [Table lists Claude model releases and time horizons increasing from one hour to 12 hours, citing METR.](https://assets.ai.engineer/sites/aiecode2025/sha256/6e1efd97ee166e4d6dd51a738b8d7c8b6c9ef3efac029f8628978e4eaea0627c.webp)

Task Horizons Are Accelerating

Model | Date Released | Time Horizon
Claude 3.7 Sonnet | Feb 2025 | 1 hr
Claude Opus 4 | May 2025 | 1.7 hrs
Claude Opus 4.5 | Nov 2025 | 4.9 hrs
Claude Opus 4.6 | Feb 2026 | 12 hrs

Source: METR, metr.org/time-horizons/

### How Do We Define Long Horizon?

The speaker treats long horizon as a relative measure: tasks can have longer horizons than others, and what counts as long changes over time.

[1:10](https://www.youtube.com/watch?v=2aS7aKoXn64&t=70s) [A question slide asks how to define long horizon.](https://assets.ai.engineer/sites/aiecode2025/sha256/1c39f206f232d21db9fc97d9a10eae8aa38a0341ccbe2994166ea25e5ab75f56.webp)

How do we define “long horizon”?

### Human Horizon

METR’s chart predicts a 12-hour 50% time horizon for Claude Opus 4.6: estimated success falls as human task time rises. Measurements above 16 hours are unreliable with the current task suite.

[1:48](https://www.youtube.com/watch?v=2aS7aKoXn64&t=108s) [METR chart shows a predicted 50% time horizon of 12 hours for Claude Opus 4.6; measurements above 16 hours are marked unreliable.](https://assets.ai.engineer/sites/aiecode2025/sha256/b094af4beea5b57d61390469291d1f7c941a0d862edf2b8d628d7f53b38dc592.webp)

1. Human Horizon

Predicted 50% time horizon: 12 hr for Claude Opus 4.6

Measurements above 16 hrs are unreliable with our current task suite

Y-axis: Estimated success rate
Y-axis ticks: 0%, 20%, 40%, 60%, 80%, 100%
X-axis: Estimated human time-to-complete
X-axis ticks: 1s, 4s, 15s, 1m, 4m, 15m, 1h, 4h, 16h, 64h

Source: METR, metr.org/time-horizons/

### Model Horizon

Model horizon measures work in tokens, steps, or tool calls. Token counts vary across models and harnesses, but holding those constant can help reveal long-trajectory challenges such as compaction and coherence.

[2:30](https://www.youtube.com/watch?v=2aS7aKoXn64&t=150s) [Slide discusses tokens, steps and tool calls as noisy model-horizon measures, and notes challenges in long trajectories.](https://assets.ai.engineer/sites/aiecode2025/sha256/e5e2936918b5e7b1336ba88e46b4eaf8a14dc76245c91541d20103dbd2886e83.webp)

2. Model Horizon

• Measuring tasks in ”model units”: tokens, steps, tool calls, etc.
• Can be much more noisy, especially across different harnesses and models
  • Some models are more token-efficient than others on the same tasks (e.g. GPT-5.5 vs Opus 4.8)
  • Easiest to interpret when holding these variables constant: if a task takes more tokens, probably harder / longer-horizon
• It’s an imperfect metric, but still an important one to help define technical frontiers for models: what makes a task difficult for models (e.g. compaction, memory, coherence over long trajectories, etc.)

### So What’s the Answer?

Use human and model horizon metrics together: an agent might script an Excel formatting task that takes a human hours, while human-time estimates depend on which experts are sampled and what work is measured.

[4:18](https://www.youtube.com/watch?v=2aS7aKoXn64&t=258s) [Slide recommends considering human and model horizon metrics together and describes limitations of human-time measurements.](https://assets.ai.engineer/sites/aiecode2025/sha256/99be273e9be42110e2692cc9028590343b4a9901d29aaa9db89e9270aabd2747.webp)

So What’s The Answer?

• All of the above...
  • No metric is perfect or paints a fully accurate picture
• What’s long-horizon for a human is not necessarily long-horizon for an agent
  • e.g. tedious, time-intensive tasks, like formatting an Excel sheet, might take a human many hours but might be fairly trivial for an agent (e.g. writes a Python script to fix all formatting issues, which a financial expert is unlikely to do)
• Measuring human time horizons depends a lot on the methodology and can be easy to skew and distort
  • What is the quality and efficiency of human experts you’re sampling from? What’s included in the measurement (reasoning/thinking for humans vs agents)? How accurate are measurements as we push towards tasks that only the top 10% of experts can do? Top 1%? Top 0.01%?
• As human work and AI work diverges further and further, each metric tells you less and less about the whole picture → important to look at metrics in tandem

### Environment Complexity — Tool Coordination

Tool coordination adds a dimension of difficulty: an agent may work on one file or move information across systems such as a database, Grafana and GitHub.

[6:53](https://www.youtube.com/watch?v=2aS7aKoXn64&t=413s) [Takeaways and diagram compare a single-file task with an agent coordinating multiple systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa5ef9e187ef7abe8372020e087995cc58177fe2eece74d70d47e82c5d2b0ce6.webp)

HOW DO YOU MEASURE MODEL CAPABILITIES?
Environment Complexity — Tool Coordination
How many tools and external dependencies the agent has to coordinate

TAKEAWAYS
• Realistic tasks pull in more systems. This is a different lever of difficulty than just running longer
• Low end: “update only this Python file” — one tool, one file, nothing else to track
• High end: the same bug fix spans a database, Grafana, an MCP deploy, and CI/CD — five or six systems at once

Low complexity
File
Agent
High complexity
Grafana
GitHub
+ many more

### Environment Complexity — State Changes

Early decisions can constrain later options: a bad query may cascade through a sequential task, while documentation can be summarized faster by reading files independently.

[7:50](https://www.youtube.com/watch?v=2aS7aKoXn64&t=470s) [Takeaways contrast sequential complexity with independently parallelizable work.](https://assets.ai.engineer/sites/aiecode2025/sha256/202f83fea53dda30feddd0a77677e98b808122325665b086bcbeb251d4cae5af.webp)

HOW DO YOU MEASURE MODEL CAPABILITIES?

Environment Complexity — State Changes

The degree to which the content of the environment changes throughout the task

TAKEAWAYS

• Tasks can be made artificially long horizon by chaining together unrelated independent tasks. However, merely making tasks take longer to complete misses the mark; a key component of adding complexity is that an agent’s initial decisions influence its options for later decisions
• Sequential complexity: a bad early query or misread dashboard cascades into every downstream step, like a deploy and CI run that takes longer to resolve end-to-end
• Parallelizable complexity: Summarizing documentation in a large codebase can be made much quicker by reading each file independently

### Ambiguity

Ambiguous instructions and starting artifacts require agents to explore, infer goals, and work through constraints. More possible valid paths make standardized evaluation harder.

[9:09](https://www.youtube.com/watch?v=2aS7aKoXn64&t=549s) [Takeaways describe interpreting underspecified tasks and the difficulty of standardized evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/57be299cef177d3182f4d5ac76bc7d20fe79346262b2e76bd9c5744f0269a1d3.webp)

HOW DO YOU MEASURE MODEL CAPABILITIES?

Ambiguity

To mirror the work humans actually do, models have to interpret and work with underspecified instructions and messy initial environments.

TAKEAWAYS

• To accurate reflect real-world tasks, models must determine the underlying motivations behind an ask, reason through constraints, and determine success criteria
• Multiple possible approaches and avenues demands more exploration, which requires more advanced reasoning
• The trade-off: standardized evaluation gets much harder

### Verifiers

Longer trajectories and softer domains make verification harder. Judge models assess correctness and assign reward by examining the final environment and the agent’s trajectory.

[10:21](https://www.youtube.com/watch?v=2aS7aKoXn64&t=621s) [Longer trajectories and softer verification domains motivate judge models for correctness and reward.](https://assets.ai.engineer/sites/aiecode2025/sha256/e34777162e2a38707b7dbc5b3bc121a53722e98fa5d6696e31c0956b45729c8b.webp)

Verifiers

01 Trajectories are growing longer and environments more complex

02 From hard-verifiable domains (math, Leetcode-style coding) to soft-verifiable domains (finance, legal, healthcare, open-ended SWE)

Judge models are a critical part of how we verify correctness and assign reward

### Why Rubric-Based Judge Models?

When deterministic checks are impractical, judges assess outcomes and trajectories. This can expose reward hacking, such as an agent accessing a hidden test suite, while allowing multiple valid solutions.

[11:30](https://www.youtube.com/watch?v=2aS7aKoXn64&t=690s) [Reasons for judge models include difficult deterministic verification and assessing solution paths.](https://assets.ai.engineer/sites/aiecode2025/sha256/b03c8fe95cc1f357ded8cb4ec7d0f33038c481646824a9f7955b12fb7486798d.webp)

Why Rubric-Based Judge Models?

• For entire classes of problems we care about, defining deterministic verifiers is impractical, brittle, or outright impossible
• Not all generated solutions are equal, and not all paths to solutions are equal (e.g. prevent reward hacking)
• Judges look at agent trajectories, not just final outcomes

### Judges Are Agents Too

A judge checking a deployment fix needs access to GitHub and AWS logs to verify environment state. Read-only permissions can prevent it from changing that state.

[13:39](https://www.youtube.com/watch?v=2aS7aKoXn64&t=819s) [Judges need tools to inspect complex environments, illustrated by a deployment investigation.](https://assets.ai.engineer/sites/aiecode2025/sha256/04fb49e729f980e10642a82392b7de8a962ccc57a2bac955dd6eea55c8b7819a.webp)

Judges Are Agents Too

• Agent harness isn’t the only thing that needs to keep up with complex environments: your judge needs to be a capable agent
• Many parts of judge harness design often mirror agent harness design: judges need to use the same tools to explore and verify sources of truth
• Especially true with environments where the state space is both large and critical for correctness

EXAMPLE
Task: Sifting through CI/CD and CloudWatch logs to investigate a broken deployment, fixing the underlying issue, and then opening a PR to redeploy with the fix
Judge: Needs tools to interact with Github and AWS just like the agent harness (with some added safeguards like read-only permissions to prevent accidental state contamination)

### Scaling Agent Trajectories

As tasks grow longer, judges need queryable trajectories to find relevant turns and failure points, then cross-reference them with environment state.

[15:28](https://www.youtube.com/watch?v=2aS7aKoXn64&t=928s) [Slide describes long trajectories as a judging bottleneck and recommends querying them and checking environment state.](https://assets.ai.engineer/sites/aiecode2025/sha256/e18189408cca836a83f9cec403131c6bdf6e8312631e538525c083a4c09c086b.webp)

Scaling Agent Trajectories

THE BOTTLENECK
Trajectory length scales with task horizon — becoming a bottleneck for judges

Dumping raw trajectories into a judge's context window no longer works

WHAT WORKS
Trajectories must be queryable, so judges can pull relevant turns, tools, and failure points

Robust judges should cross-reference trajectory data with environment state

### Environment Learnability

A final-file-only reward can miss progress toward correct forecasting assumptions. More granular rubrics can provide denser feedback, but judges need careful QA to apply them consistently.

[16:31](https://www.youtube.com/watch?v=2aS7aKoXn64&t=991s) [Slide discusses reward density, rubric quality and task distribution, with a financial forecasting example.](https://assets.ai.engineer/sites/aiecode2025/sha256/05d400ec50ad7d291553928fa7e811055cc92f593fe642be2a49358e0d3697dd.webp)

Environment Learnability

• Learnability affects how effectively your model can learn across rollouts: can waste tons of compute, or even worse, make learning infeasible
• One of the biggest levers in improving learnability: density of reward signal
  • More detailed, granular rubrics → requires careful QA and harness design
• Other learnability factors include distribution of data in environments, e.g. breadth of subdomains and depth of environment and task complexity

EXAMPLE
Task: Financial forecasting task in Excel for next 8 quarters of a company’s revenue, requires sifting through large data rooms, financials, and market data to even arrive at the correct inputs & assumptions
Reward Signal: Only giving feedback on final Excel file outputted would provide no reward signal where the file was wrong but progress was made towards correct assumptions

### Emerging Rubric-Judge Patterns

Dynamic rubrics award partial credit by grading downstream work against a model’s assumptions. Deterministic checks can run alongside judges or supply metrics for their review.

[17:19](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1039s) [Two columns describe dynamic evaluation-time rubrics and combinations of deterministic and judge-based verifiers.](https://assets.ai.engineer/sites/aiecode2025/sha256/519bcab042788d4681553af326fb8eee668aa76a290f41e8771aa0f0fdd1b010.webp)

Emerging Rubric-Judge Patterns

Dynamic evaluation-time rubrics

• Even when a model's assumptions, inputs, or early steps were wrong, the judge can grade downstream work as if they were correct, generating a tailored rubric on the fly
• Example: in the finance forecasting task, the judge parses out the model's assumptions, calls a tool to plug them into a golden financial model, and uses the financial model’s outputs to check for partial credit

Combining deterministic and judge-based verifiers

• Piecewise: independent, weighted components — e.g. test cases catch regressions in a code refactor, while a judge separately evaluates whether the changes were meaningful
• Deterministic artifact for judge review: precompute objective metrics, then hand them to the judge to interpret — e.g. exposing hardware-dependent performance metrics as a CSV for the judge to grade

### Real Work Is Messy; Benchmarks Need to Be Scoreable

The proposed benchmark keeps messy files, ambiguous instructions, and judgment calls in the workflow while constraining the answer space to make scoring reliable.

[17:50](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1070s) [Slide argues for constraining benchmark answer spaces while preserving messy workflows and reliable ground truth.](https://assets.ai.engineer/sites/aiecode2025/sha256/0176d2d7137bab9c54f0174f929f7098f4ea3165cda6af8144653c64ead4cd67.webp)

Real work is messy; benchmarks need to be scoreable.

PROBLEM
Real workflows include messy files, ambiguous instructions, and judgment calls.

OUR PERSPECTIVE
Remove all the messiness and you stop measuring real agent capability.

HOW WE ADDRESS IT
Constrain the answer space without over-simplifying the workflow.

The goal isn’t clean tasks — it’s messy tasks with reliable ground truth.

### Rubric-Judge QA

Rubric QA checks scores at the gold and initial states, grading consistency, and coverage of their differences. An expert also examines disagreements with an LLM judge.

[17:53](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1073s) [Slide lists gold, no-op, variance, coverage and expert agreement tests for rubrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e0f00363b173bf15a366727e52cca677e77463f6eaba6615198b337a0c63d75.webp)

Rubric-Judge QA

For each rubric we produce, we run 5 different tests:

• Gold: We grade a verified “golden solution,” ensuring every attempt scores >98%
• No-Op: We grade the initial starting stage, ensuring every attempt scores <3%
• Variance: Same attempt, same judge, repeated multiple times. We minimize variance at both the criteria and sub-criteria level. For each rubric, we keep variance <2% for the same attempt
• Coverage: % of differences covered by the rubric criteria. We ensure our rubric covers 100% of the differences in between the initial and gold states with no gaps
• Expert Agreement: We have an expert and an LLM-as-a-judge independently grade the task. Then, we have the expert examine disagreements and determine any relevant improvements to the rubric

### Benchmark Data and Evaluation

The speaker argues that these benchmarks’ 1.4–7 average human hours per task fall short of long-horizon work and that their scores suggest saturation. The benchmarks use different scoring metrics.

[18:21](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1101s) [Slide claims training and evaluation data are flawed and compares three benchmarks by task duration, metric, model and score.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c5476fa1d7242821f3379157d36c18b4b7b9bf88f643791ba0716807fb28654.webp)

Existing literature suggests a lot of the data being used to train and evaluate models right now is flawed

| | GDPval | Banker Tool Bench | APEX Agents |
| Published by | OpenAI | Handshake AI | Mercor |
| Average Human Hours Per Task | 7 hours | 5 hours | 1.4 hours |
| Score Metric | Win Rate | Score@3 | Pass@1 |
| Model Used | GPT 5.5 | GPT 5.4 | Gemini 3.5 Flash |
| Score | 84.9% | 58.1% | 57% |

### Theta finance-task statistics and model results

Across seven finance subdomains, mean@5 scores range from 27.21% to 71.62%. Tasks averaged 14.7 human hours in a 50-task sample; GPT-5.5 runtime averaged 92.4 minutes per task.

[20:25](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1225s) [Slide reporting average finance-task duration and spreadsheet size, with mean-at-five model scores across seven finance subdomains.](https://assets.ai.engineer/sites/aiecode2025/sha256/739e9a4c2c103d566e8fae97030445d51e3b69ba52c2cc53a05dec7bab417bcd.webp)

At Theta, we build and think deeply about long horizon environments for frontier models—the right way

Metric | Average
Human time to complete one task | 14.7 hr
Model runtime per task | 92.4 min
Cell changes per task | 3,600 cells
Sheets per Excel model | 5.8

* Runtime figures reported for GPT-5.5.

Mean@5
Subdomain | GPT-5.5 | Opus 4.8 | DeepSeek-V4-Pro | MiniMax M3
Corporate Finance | 57.94% | 54.21% | 42.72% | 31.12%
Investment Banking | 68.09% | 71.62% | 33.69% | 48.48%
Private Equity | 48.41% | 45.45% | 43.56% | 36.42%
Credit | 49.37% | 46.03% | 39.59% | 33.27%
Debt | 46.12% | 44.54% | 29.91% | 27.21%
Real Estate | 54.13% | 56.23% | 27.37% | 35.73%
Risk | 62.71% | 63.47% | 43.17% | 39.50%

## Full transcript

[0:00](https://www.youtube.com/watch?v=2aS7aKoXn64&t=0s) [on-hold jingle] It's, uh, great to see all of you here today. We're super excited to talk about, uh, one of our favorite topics, uh, here at Theta.

[0:18](https://www.youtube.com/watch?v=2aS7aKoXn64&t=18s) Um, before we get started, we just want to introduce ourselves. Um, so hi, I'm a co-founder and CTO at Theta Software.

[0:25](https://www.youtube.com/watch?v=2aS7aKoXn64&t=25s) Hi, I'm Rayan. I'm a co-founder and CEO at Theta Software. Prior to this, I was previously a founding engineer at DeepSilicon, where we did research into ternary models.

[0:34](https://www.youtube.com/watch?v=2aS7aKoXn64&t=34s) Awesome. So I can get us started with the topic today. Um, we're gonna be talking about RL environments, uh, within the context of long-horizon tasks. And I think the most important thing for us to start with at the beginning is just talk about the trends and what long horizon actually means.

[0:48](https://www.youtube.com/watch?v=2aS7aKoXn64&t=48s) So, you know, we all know that the horizon at which AI agents can work autonomously is accelerating really fast. Um, this is just some of the metrics that you can look at to see how this progress is really accelerating.

[0:59](https://www.youtube.com/watch?v=2aS7aKoXn64&t=59s) Um, but I think it's really important to actually define what the time horizon here actually means. Uh, we've gotten this, uh, data from one of the most common benchmarks out there that you've probably heard of for time horizons.

[1:08](https://www.youtube.com/watch?v=2aS7aKoXn64&t=68s) I'm sure you've seen it in your Twitter feed all the time. It's-- um, comes from METR. Uh, and METR kind of has one response or answer to this really important question about how do we actually define long horizon.

[1:20](https://www.youtube.com/watch?v=2aS7aKoXn64&t=80s) Um, it's really important to understand because what we considered long horizon a year ago probably isn't really long horizon in our definition today. And what's long horizon today probably won't be long horizon in a year or two.

[1:30](https://www.youtube.com/watch?v=2aS7aKoXn64&t=90s) Um, and I think that gets to our first point, which is that long horizon is really kind of a scalar metric. Uh, it's useful for kind of measuring relative tasks.

[1:39](https://www.youtube.com/watch?v=2aS7aKoXn64&t=99s) Like one task might be more long horizon than another, but it's really hard to define into kind of a binary category of this task is long horizon, this task is not, uh, especially as the kind of scope changes over time.

[1:49](https://www.youtube.com/watch?v=2aS7aKoXn64&t=109s) So I think the first way we can talk about defining this is how METR kinda looks at it, uh, which is human horizon. Meaning can we use humans as a benchmark of, oh, this task takes humans a certain amount of time, so if AI agents can do that, then they've reached this certain critical, uh, level of kind

[2:06](https://www.youtube.com/watch?v=2aS7aKoXn64&t=126s) of a time horizon. Uh, and the way METR kinda does it is they have thresholds for the tasks they care about. So, you know, they have a fifty percent threshold, meaning, you know, if a certain model reaches a sixteen-hour threshold on this, uh, benchmark, that means it can achieve tasks, uh, with a fifty percent su-success rate that

[2:20](https://www.youtube.com/watch?v=2aS7aKoXn64&t=140s) take a human sixteen hours. And, you know, there's a really rigorous methodology of how they actually measure how did it take humans sixteen hours, but we'll kinda avoid some of those details.

[2:28](https://www.youtube.com/watch?v=2aS7aKoXn64&t=148s) Um, the other way we usually think about what long horizon actually means is not with the reference of humans, but instead the reference of models. Uh, so some of the relevant model units we usually care about are things like tokens, how many tokens are consumed in a trajectory, uh, how many steps it took, how many tool calls

[2:44](https://www.youtube.com/watch?v=2aS7aKoXn64&t=164s) it kinda takes. Um, and these can be really noisy, right? 'Cause I'm sure you guys have used different models. Um, like, you know, a lot of the Codex models, uh, are seen as more token efficient than some of the Claude models.

[2:54](https://www.youtube.com/watch?v=2aS7aKoXn64&t=174s) Um, and it's a pretty noisy estimate for a couple reasons. One is that which model you're using, like I just said, and different harnesses you care about have a pretty big impact on how many tokens are actually consumed on a task, right?

[3:04](https://www.youtube.com/watch?v=2aS7aKoXn64&t=184s) So, um, this can be pretty hard to interpret when you're not holding variables constant. Um, you know, if a task takes a GPT model five hundred thousand tokens, that doesn't really tell you a lot about what that task would look like for Claude models until you actually run it on those Claude models.

[3:18](https://www.youtube.com/watch?v=2aS7aKoXn64&t=198s) But despite it being a pretty noisy metric, it's actually really useful, uh, and important for us to understand because, you know, the amount of tokens that are consumed tells us a lot about how difficult a task actually is for an AI agent to kinda tackle it autonomously, right?

[3:33](https://www.youtube.com/watch?v=2aS7aKoXn64&t=213s) Uh, you have to deal with things like compaction over long horizons. Um, you know, they don't really stay coherent over enough steps or trajectory, uh, length that you kind of achieve.

[3:41](https://www.youtube.com/watch?v=2aS7aKoXn64&t=221s) So even though it's kind of a noisy metric, uh, it can be really useful when, you know, if we look at what a GPT-5.5 model can do, uh, now, and then you use the same model generation and kind of see, oh, now it can actually achieve a thou-- a million trajectory based on an increased context window

[3:54](https://www.youtube.com/watch?v=2aS7aKoXn64&t=234s) or improved compaction endpoint. That tells us a lot about how autonomous, uh, AI agents can actually go for long periods of time in that sense. Um, and it really defines for us what the technical frontier actually means, uh, for models right now.

[4:07](https://www.youtube.com/watch?v=2aS7aKoXn64&t=247s) Maybe not really human-adjacent. It's really hard to say how many tokens a task takes for a human 'cause we don't really think in tokens, but still very useful in that kind of sense.

[4:15](https://www.youtube.com/watch?v=2aS7aKoXn64&t=255s) Um, so these are two different approaches we can think about, but what's actually the right way to think about this? Uh, the answer is that we probably wanna think about all of these, and if we just look at one of these metrics in isolation, it's probably not a great way of measuring things.

[4:28](https://www.youtube.com/watch?v=2aS7aKoXn64&t=268s) So I-I went through some of the weaknesses with measuring with, like, model-specific metrics like tokens and, and steps, but there's also a lot of weaknesses in the other approach of kind of relying on humans.

[4:38](https://www.youtube.com/watch?v=2aS7aKoXn64&t=278s) Um, you know, what's long horizon for a human isn't necessarily that difficult for a, a model, depending on what the actual task you care about is. You know, there's a lot of tasks that are really tedious and time-intensive.

[4:48](https://www.youtube.com/watch?v=2aS7aKoXn64&t=288s) Maybe, like, you know, some financial analyst has to go into, like, an Excel file and fix a bunch of formatting issues throughout, uh, the, the task. Maybe they're changing, like, the theming of, like, the colors in the-- in the actual file, right?

[5:00](https://www.youtube.com/watch?v=2aS7aKoXn64&t=300s) That might be really tedious for a human to take. It might take them, like, days to do that if it's a really big Excel file. But for a model, it can maybe write a Python script or find some other cool trick to do that really quickly.

[5:10](https://www.youtube.com/watch?v=2aS7aKoXn64&t=310s) And that's not really hard for it to do, but you would never really expect a financial expert to do that because, you know, they-- most of them don't really know how to write these Python scripts.

[5:16](https://www.youtube.com/watch?v=2aS7aKoXn64&t=316s) Um, so I think that's one thing to note. And the other that I kinda briefly touched upon before is that the methodology of how we actually measure this has a really big impact.

[5:25](https://www.youtube.com/watch?v=2aS7aKoXn64&t=325s) And if you, you know, someone is out there saying, "Hey, we have some tasks or environments that are sixteen hours long on average." Someone else says twenty hours. That's really hard to compare across people 'cause there's so many different things in the methodology that really impact, uh, kinda what that actually means.

[5:37](https://www.youtube.com/watch?v=2aS7aKoXn64&t=337s) It can mean, you know, the quality of the experts you're using. Some more experienced experts might actually be way more efficient at doing a certain type of financial or coding task, whatever it kind of is.

[5:46](https://www.youtube.com/watch?v=2aS7aKoXn64&t=346s) Um- And I think this becomes really, really important as we start shifting towards, uh, kind of the frontier of even human capabilities. So, you know, as... This METR talks about this, but as you shift towards more long-horizon tasks and tasks that only the top 10%, the top 1%, the top point...

[6:01](https://www.youtube.com/watch?v=2aS7aKoXn64&t=361s) one percent of humans can really do, these estimates start to get really, really noisy, and it's something that we really have to consider. Uh, and I think, you know, the way agents work is kind of developing in its own separate path, and there are a lot of different bottlenecks and different things that AI agents are better at than

[6:14](https://www.youtube.com/watch?v=2aS7aKoXn64&t=374s) even the way humans work. And with that in mind, uh, you know, as these paths kind of diverge of how humans do work and what their limitations are, and what agents do and what their limitations are, uh, it's really important to kind of keep both these metrics in mind because they kind of tell and paint different pictures,

[6:28](https://www.youtube.com/watch?v=2aS7aKoXn64&t=388s) uh, of, of what's actually relevant and, and you r- don't really get the whole picture by just looking at one in that sense.

[6:36](https://www.youtube.com/watch?v=2aS7aKoXn64&t=396s) Yep. So now the question becomes: How do you measure model capabilities? And this is a really important question because fundamentally, long-horizon tasks aren't the only thing we care about.

[6:46](https://www.youtube.com/watch?v=2aS7aKoXn64&t=406s) This is the larger question that we wanna think about every time we're trying to create tasks, create environments to train our models. And so,

[6:54](https://www.youtube.com/watch?v=2aS7aKoXn64&t=414s) the first way we can think about this is environment complexity, and specifically environment complexity related to tool coordination, right? So how many tools or external dependency does the agent have to coordinate?

[7:05](https://www.youtube.com/watch?v=2aS7aKoXn64&t=425s) How many tools or external dependencies does the agent have to move information across? So, if we start off kind of thinking about what the world looked like before a long-horizon task, uh, you know, world, we'll notice that there was, you know, a low complexity world where the agent maybe had to read one file or one set of

[7:20](https://www.youtube.com/watch?v=2aS7aKoXn64&t=440s) files in a code base, and that's kind of what a task entailed. But now w-we can see increasingly as these tasks become more long-horizon, what is important to define for measuring model capabilities is, okay, the, the agent should be using a ton of different tools like Grafana for observability to parse logs, or GitHub f-for CI/CD, or AWS

[7:41](https://www.youtube.com/watch?v=2aS7aKoXn64&t=461s) CloudWatch, or reading and writing to a database. And we're gonna notice that as we sort of start to have these agents and these environments use many tools, that we also start to think about environment complexity in regards to state changes, which is effectively the degree to which the environment changes throughout the task.

[7:57](https://www.youtube.com/watch?v=2aS7aKoXn64&t=477s) And so fundamentally, the way we wanna think about this is, right, ta- all long-horizon tasks aren't equal. So for example, one task can, you know, maybe be made by artificially long horizon by chaining together unrelated independent tasks.

[8:11](https://www.youtube.com/watch?v=2aS7aKoXn64&t=491s) However, that doesn't actually tell us or meaningfully measure the model capabilities. Instead, a key component of this is actually being able to have the earlier decisions in the, in the environment influence the later decisions.

[8:25](https://www.youtube.com/watch?v=2aS7aKoXn64&t=505s) And this comes back to how the agents are asked to interact with the tools, how these tools change the state of the environment, et cetera. So we can look at an, uh, a concrete example for this.

[8:34](https://www.youtube.com/watch?v=2aS7aKoXn64&t=514s) One example where you'll see parallelizable complexity, which is effectively not involving a lot of state changes, is when you can maybe have an agent analyzing a large code base, and then the agent needs to spawn off multiple sub-agents, and it can very easily parallelize this, right?

[8:46](https://www.youtube.com/watch?v=2aS7aKoXn64&t=526s) It can look at a lot of the different files in parallel, come back to the, to, to the master agent, and then kind of wrap this all up, right?

[8:52](https://www.youtube.com/watch?v=2aS7aKoXn64&t=532s) But s- meanwhile, if we look at sequential complexity, we'll see if you have to use a dashboard or logs, a bad early query or a misread can cascade into these downstream steps that really start to have major consequences later on, right?

[9:05](https://www.youtube.com/watch?v=2aS7aKoXn64&t=545s) It's all dependent on how you use those tools and how the state of the environment change. So the third area that we also need to consider for measuring model capabilities is ambiguity, right?

[9:15](https://www.youtube.com/watch?v=2aS7aKoXn64&t=555s) And ambiguity is defined as the information you give the agent and the environment when starting the task. So this could be the instructions, this could be the artifacts, et cetera.

[9:24](https://www.youtube.com/watch?v=2aS7aKoXn64&t=564s) And in-increasingly, as these agents work with more artifacts at the start, right? We, we want to have them mirror the work that humans really do. And the work that humans really do has a lot to deal with ambiguity, right?

[9:35](https://www.youtube.com/watch?v=2aS7aKoXn64&t=575s) They always are, are-- don't have the most complete information, and they want to let exploration happen. And so we believe that to measure mo-model capabilities, we need to test the model's ability to explore and explore throughout the environment as well, and explore these artifacts similar to how a human would.

[9:50](https://www.youtube.com/watch?v=2aS7aKoXn64&t=590s) Now, the trade-off with this, right? Is that if you are going to have ambiguity in the materials you give, there's a lot more possible paths that the agent could take.

[9:59](https://www.youtube.com/watch?v=2aS7aKoXn64&t=599s) There's a lot more ways the agent could be right, and that means that standardized evaluation gets much, much harder.

[10:06](https://www.youtube.com/watch?v=2aS7aKoXn64&t=606s) Awesome. So I'm gonna talk about one of the hardest things there are to build environments and one, one of the most complex things to really think about, where there's a lot of nuance, which is the verifier in the environment.

[10:16](https://www.youtube.com/watch?v=2aS7aKoXn64&t=616s) How do we actually know that the work the agent did was correct, and give it some reward signal during the training process? So I think there's a few challenges here.

[10:22](https://www.youtube.com/watch?v=2aS7aKoXn64&t=622s) Um, you know, I think just to give a high-level overview, um, you know, tasks are getting more complex, the environment's getting more complex, the trajectories are getting longer, and we've shifted a lot from, you know, a lot of the early RL that we were doing, uh, in, in recent times was really in hard verifiable domains, and that's

[10:37](https://www.youtube.com/watch?v=2aS7aKoXn64&t=637s) why we saw these gains in, in math and kind of, uh, like data structure style, uh, coding problems. But what's happened over time is now we really care about a bunch of economically valuable work in soft-verifiable domains is, is a way to put it, where, uh, you know, we can't just run a Python script or run test

[10:52](https://www.youtube.com/watch?v=2aS7aKoXn64&t=652s) cases or, or write a proof to really see whether or not the output was correct or whether or not the environment was changed correctly. Uh, we have to start using other techniques, and the main way we're really gonna use that is kind of introduce a judge model or a critic model, uh, as some people put it, and

[11:05](https://www.youtube.com/watch?v=2aS7aKoXn64&t=665s) they kind of can add a lot of nuance to how we actually look at a few things here. Um, you know, very critical for how we actually determine correctness and assign reward.

[11:14](https://www.youtube.com/watch?v=2aS7aKoXn64&t=674s) Um, they'll look at two things mainly. One is usually either the state, final state of the environment and kind of how it was impacted. Um, and the other is looking at the trajectory of how the model that you're actually training made, uh, kind of changes to the, the state of the environment as well and, and what kind

[11:28](https://www.youtube.com/watch?v=2aS7aKoXn64&t=688s) of correctness look like there. Um, so, you know, why do we actually use, uh, judges and usually rubrics, um, as a technique? I think that's really important to understand before we can even understand how to use them properly, which, you know, I think there's a few reasons.

[11:41](https://www.youtube.com/watch?v=2aS7aKoXn64&t=701s) One is that, like I said, for these soft-verifiable domains, there's like an entire class of problems that are really important and a lot of the problems we care about that really you can't really write a determi-deterministic verifier for.

[11:51](https://www.youtube.com/watch?v=2aS7aKoXn64&t=711s) Um, they would be really impractical, brittle, or just downright impossible depending on what the problem setup really is. Um, and You know, I think the other thing also is that, like I said, we're gonna look at the trajectory.

[12:02](https://www.youtube.com/watch?v=2aS7aKoXn64&t=722s) Uh, and, you know, not all solutions are really created equal, and not all paths to those solutions are equal either. Uh, you know, the worst case of a bad solution we can get is some reward hacking that happens.

[12:11](https://www.youtube.com/watch?v=2aS7aKoXn64&t=731s) Lots of different types of reward hacking can happen depending on the setup or the task you kinda care about. You know, a agent can es-escape a sandbox and see privileged information it shouldn't be seeing about maybe a hidden te-test suite for, like, a coding task.

[12:22](https://www.youtube.com/watch?v=2aS7aKoXn64&t=742s) Um, this is all behavior that we wanna prevent, obviously, 'cause those are not actually really valid solutions we really care about. Um, and, you know, a lot of this-- mitigating this is gonna require strengthening your verifier and your environment setup, but the judge is really, really important in actually catching this behavior, and that's an important reason of

[12:38](https://www.youtube.com/watch?v=2aS7aKoXn64&t=758s) why we actually look at the trajectory that the agent actually took to get there. Um, yeah, and I think there's a lot of careful things we wanna be doing here.

[12:45](https://www.youtube.com/watch?v=2aS7aKoXn64&t=765s) Uh, one is that there's nuance in how much guidance or, uh, like explicit rigidness you wanna add to the trajectories that the model can actually take. Um, if we kinda enforce this too tightly, we collapse the state space of how many actual, uh, paths the agent actually explores.

[13:00](https://www.youtube.com/watch?v=2aS7aKoXn64&t=780s) Uh, and that can be really bad, uh, especially because I think some of the more simple approaches we've seen with judges early on is, "Hey, we'll just give it a, a reference, uh, answer or a solution or maybe a sample trajectory of what a good solution looks like," and then just compare against, uh, what the model did

[13:13](https://www.youtube.com/watch?v=2aS7aKoXn64&t=793s) and say, "Hey, does it match up with that?" Uh, and that really does not work for these more ambiguous or open-ended tasks 'cause there are so many possible correct solutions.

[13:19](https://www.youtube.com/watch?v=2aS7aKoXn64&t=799s) It's basically impossible to account for every single one, and we wanna check for more robust methods that allow for these different solutions.

[13:28](https://www.youtube.com/watch?v=2aS7aKoXn64&t=808s) So now that we've kind of established why we use judges, uh, we wanna go through some of the general, uh, heuristics and kind of principles we think about when we're designing good judges, um, some of the things that we think about at Theta.

[13:39](https://www.youtube.com/watch?v=2aS7aKoXn64&t=819s) So, you know, I think the first important, uh, consideration to make is that judges are agents too. Um, you know, so as environments get really complex, uh, oftentimes we-- a consideration we kinda have is like, "Hey, we have to make sure the harness can kinda scale and, and kinda match up with whatever environment, uh, you kind of

[13:56](https://www.youtube.com/watch?v=2aS7aKoXn64&t=836s) have." Maybe that means introducing a bunch of new tools and making sure your harness can support those tools really well. The agent has clear observability over what's happening in the environment.

[14:03](https://www.youtube.com/watch?v=2aS7aKoXn64&t=843s) Um, but I think, like we said, the way the judge determines correctness is that it oftentimes has to look at the state of the environment itself as well. So a lot of the harness that you've designed for the agent might also be reused, uh, for the judge as well.

[14:15](https://www.youtube.com/watch?v=2aS7aKoXn64&t=855s) Um, I think the best way to illustrate this is the example we have here. Let's say you define a task where, you know, there's some deployment failure with a software engineering task of some platform you're deploying, and the agent's task is to, like, sift through the CI/CD logs on GitHub, look through the Cou-CloudWatch logs, figure out whatever

[14:29](https://www.youtube.com/watch?v=2aS7aKoXn64&t=869s) happened, uh, kind of apply the changes you care about to the code base, and then open a PR and, and kind of kick off a redeploy there, uh, once the PR is merged.

[14:37](https://www.youtube.com/watch?v=2aS7aKoXn64&t=877s) Um, for a lot of a-- for a lot of that, if the judge actually wants to verify, uh, whether or not this is correct besides just, like, looking at the tool calls the agent made, which are usually not very reliable, it actually has to also check the GitHub logs.

[14:48](https://www.youtube.com/watch?v=2aS7aKoXn64&t=888s) It might check the AWS, uh, logs or the GitHub logs after the deployment happened to make sure, oh, th- are things actually working properly. So it's really important that the judge has access to the environment in the same way, uh, with some important safeguards, of course.

[15:00](https://www.youtube.com/watch?v=2aS7aKoXn64&t=900s) Uh, one is that we don't want the judge to make an accidental mutation in some way to the environment after the agent is done, so you wanna be very careful about that.

[15:06](https://www.youtube.com/watch?v=2aS7aKoXn64&t=906s) Maybe that means enforcing read-only permissions for a lot of this information. It can't actually kick off a deployment or anything like that. So those are things to be careful about.

[15:12](https://www.youtube.com/watch?v=2aS7aKoXn64&t=912s) Um, but I think this is really, really important, especially where there's a lot of open-ended approaches, and the only way we can really verify correctness is to actually look at the state itself.

[15:20](https://www.youtube.com/watch?v=2aS7aKoXn64&t=920s) Uh, you-- the answer isn't obvious of whether or not the agent completed the task just from looking at the trajectory. So I think that's one example where this approach is really, really important.

[15:27](https://www.youtube.com/watch?v=2aS7aKoXn64&t=927s) Um, I think, uh, the other thing to be notable of is, you know, as these environments get more complex, the agent trajectories get longer and longer, and part of the reason we also need the judge to be an agent is that you can't just use this really basic approach of taking the trajectory and stuffing it in the

[15:42](https://www.youtube.com/watch?v=2aS7aKoXn64&t=942s) context window of the judge and kinda have it be a basic LLM call. Uh, these trajectories can get really, really long and really, really complex, so we need to do a lot more thoughtful, uh, processing of the trajectory in some meaningful way.

[15:54](https://www.youtube.com/watch?v=2aS7aKoXn64&t=954s) So, you know, that might mean we put into some database, we use sub-agents to actually enrich certain information. Maybe we parse out, uh, specific phases that the agent was actually in.

[16:03](https://www.youtube.com/watch?v=2aS7aKoXn64&t=963s) Maybe the beginning part was it going through logs. The second part was a-actually writing code. The third part was actually it checking what happened after that. These are all different things we wanna, uh, we wanna do, and in that sense, we need to make the trajectory itself queryable.

[16:14](https://www.youtube.com/watch?v=2aS7aKoXn64&t=974s) So that might mean enriching of information, like I just said, or some other metadata we can kinda look at at certain steps. And it's really important so the agent can find critical steps, um, you know, like failure points, and, and verify whether or not those are actually failures.

[16:26](https://www.youtube.com/watch?v=2aS7aKoXn64&t=986s) Uh, making that, uh, kind of usable for the agent is really, really important.

[16:32](https://www.youtube.com/watch?v=2aS7aKoXn64&t=992s) Um, I think another important thing to consider is learnability of our environments. Um, you know, the most important thing here is just the density of the reward signal, and a lot of that comes from your rubric and kind of how the judge is defining that.

[16:43](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1003s) So I think you have to be very careful with just, uh, you know, overloading with density in your, in your rubric. Uh, a lot of times, especially for frontier problems that models aren't really capable of yet, uh, judges will really struggle to apply that rubric consistently.

[16:54](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1014s) So there's a lot of QA we kinda need to do to make sure judges are able to apply that information correctly. Um, you know, there's other learnability factors that we care about and we measure in environments, like the distribution of tasks and, and, and the actual underlying data there is.

[17:06](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1026s) Uh, and, and these are all kind of things we think about for learnability, and it's really important. Otherwise, you're just wasting a bunch of compute on, on problems where the model can't actually effectively learn.

[17:16](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1036s) Um, you know, these are some emerging rubric judge patterns we've seen. I'll quickly skim over this. Um, you know, oftentimes you-- deterministic verifiers aren't completely dead. Oftentimes, we use them in tandem with judges, maybe generating an artifact for the judge to actually look over, where you're maybe collecting metrics.

[17:31](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1051s) Or, uh, uh, an interesting thing that we could al-also use is dynamic evaluation time rubrics, where we're actually generating, um, you know-- We're maybe giving partial credit where we've baked in some assumptions that the model's made and assume they're correct.

[17:41](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1061s) It's like grading a test assuming, like, if you got the first part wrong, let's just assume it's correct. Did they get the rest of the part right? That can be really important as well, as well for kind of assigning credit there.

[17:53](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1073s) Skip over this part. Um, I will let Rayan just close things off with some things about QA for rubrics.

[17:59](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1079s) Yep. So for each rubric we produce, we run a couple different tests. Won't go into all of them. Some of them are pretty basic, right? Gold, no-op, variance. These are tests you wanna be considering regardless for your verifiers.

[18:08](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1088s) But I think increasingly, you know, as, as you involve AI in the process of even creating rubrics or verifying rubrics or aiding experts, you need to have more and more tests, especially as the tasks become more long horizon.

[18:18](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1098s) And so that really touches on the coverage and the expert agreement. But I think what we wanted to close off with today is why a lot of this stuff matters, right?

[18:25](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1105s) We spent a lot of time earlier in this presentation defining what long horizon means, and a huge reason we did that is because we feel like a lot of the literature and data-- that a lot of the literature shows that a lot of the data being produced right now and being used to train and evaluate models is

[18:38](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1118s) actually flawed. So we present three major benchmarks in the area of finance predominantly. And so this is GDPval, BankerToolBench, and APEX-Agents. There's a couple of notable issues here.

[18:49](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1129s) First, if you look at the average human hours per task, based on what METR has defined for a lot of the leading frontier models, a lot of these different average human hours per task fall far below that, and so they wouldn't actually be considered long-horizon tasks.

[19:01](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1141s) The second notable issue here, right, is that we see that these benchmarks are already reasonably saturated, and we think this is a downstream effect of the average human hours per task.

[19:10](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1150s) So it's really important to look at the metrics that are being used here. If you look at, you know, the APEX-Agents IB section of this benchmark that they put out, pass@1 effectively means that the-- for, like, fifty-seven percent of cases, the tasks are a hundred percent solved.

[19:24](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1164s) That is effectively telling us that, like, there's a large part of these tasks that models are solving similar to what we've seen already. But I think a third key important part here is the breadth.

[19:34](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1174s) For each of these different m-- uh, benchmarks, particularly GDPval, they have a very narrow set of Excel tasks they consider for finance. And for APEX-Agents, they're largely focused on IB.

[19:44](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1184s) What this means is that a lot of these more important areas for learnability like, you know, credit, debt, risk in the domain of finance don't really get covered. And then I think lastly, I'll, I'll, I'll note there, the reward signal, as Govrra mentioned, is really important.

[19:59](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1199s) And in regards to the reward signal here, we'll, we'll notice that there's, like, really...

[20:04](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1204s) You know, if you look at, if you look at what you need for a rubric, you need very granular, detailed reward signal. You need-- You know, we, we have 20 different sub-criteria and 10 different sub-criteria per criteria.

[20:17](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1217s) So I think there's a lot of room that's left when you read these benchmarks into how granular reward signal they're giving, which is really important for being able to go ahead and train your models.

[20:26](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1226s) With that, I think I wanted to round off with a couple of stats about the data we produce. Here we-- You know, you can look at some statistics for our finance data.

[20:33](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1233s) We can see that the human time to complete one task on average is 15 hours over a 50-task sample set. Furthermore, it takes models a pretty long time to work through these tasks, and after all of that, across all the domains we care about within finance, for example, they still struggle significantly.

[20:47](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1247s) And so here we provide mean@5, notably different than, you know, all of these previous scores, uh, we see here. So thanks for, for taking the time to talk with us today.

[20:55](https://www.youtube.com/watch?v=2aS7aKoXn64&t=1255s) Yeah. Thank you. [outro music]