Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic

Katelyn Lesse, who leads platform engineering at Anthropic, and Angela Jiang, who leads platform product, start with the familiar way teams improve an agent: give it more tokens or more expensive tokens. That treats budget as the main lever and assumes every token makes an interchangeable contribution to the result.

A conventional agent receives a task and spends its allowance continuing the work. “Tokens should have jobs” reframes that allowance as compute that can serve different functions. Some tokens can execute the task, while others check the approach, judge an attempt, or preserve a lesson for later. A particular allocation of those roles becomes a strategy.

This does not make individual tokens intrinsically different. The difference comes from the prompts, context, agent roles, and control flow that determine what the model does with them. The practical question changes from “How large is the budget?” to “What work should this budget buy?”

The three strategies intervene at different points in an agent’s lifecycle:

• Advising — during execution: An executor can ask a separate adviser whether its next step makes sense, then use that response to adjust its work. A sales agent, for example, might use advice while checking overdue follow-ups and stalled deals.
• Grading — after an attempt: A grader compares completed work with an explicit rubric. A passing attempt finishes; a failing attempt returns to the executor for another iteration.
• Dreaming — between runs: A dreamer inspects the executor’s work and transcript, extracts findings, and writes them to memory for the next run.

Grading works best when “good” can be stated clearly. Consider a customer asking a store for a refund. The executor drafts or selects a response; the rubric encodes the store’s refund criteria; the grader checks the proposed outcome against those criteria; and a failed check triggers another attempt. The loop makes policy the reference point, although its reliability still depends on whether the rubric captures the right rules and the grader applies them correctly.

Dreaming targets repeated work rather than the current answer. In the recruiting example, feedback about candidate fit accumulates across interactions. A dreamer turns findings from those interactions into persistent memory, and the next execution reads that memory before proceeding. The described improvement comes from changing future context, not from updating model weights. The talk does not specify how memories are selected, corrected, or retired, so those remain implementation problems.

The experiment moves these strategies into a benchmark of financial-analysis tasks intended to resemble work performed by an expert analyst. Execution alone serves as the control; advising, grading, and dreaming are alternative ways to organize the work.

The first one-shot comparison cannot isolate the value of those jobs. Execution scores 15% while using 39,000 tokens, whereas more complex strategies spend more and perform better. Dreaming reaches roughly 600,000 tokens. Strategy and compute have both changed, so the result cannot tell whether the improvement came from orchestration, additional spending, or both.

The next comparison fixes a common maximum allowance at roughly 600,000 tokens. More test-time compute improves every strategy: execution rises from 15 to 76, while advising and grading move from scores in the 60s toward the 90s. The revealing comparison is within the shared allowance: execution scores 76 and advising scores 89. The same budget produces a different outcome when some of it funds consultation rather than continuation.

That reported gap motivates strategy design, but its scope is uncertain: the presentation does not provide the number of tasks, model configurations, repeated-run variance, or statistical uncertainty needed to establish how broadly advising will outperform execution. The supported result is narrower—on this financial-analysis benchmark, allocation mattered under the fixed allowance.

The benchmark score still leaves a product question unanswered: can an analyst use the result? Suppose the agent produces a profit-and-loss statement that is 80% accurate. That sounds respectable as partial credit, but the analyst cannot safely accept invented or incorrect income and cost figures. They must recompute the statement or pay for another run. Observable progress on the benchmark has not produced a finished P&L.

The experiment is therefore rescored around usable completion. A perfectly scored task passes; anything below 100% fails. This converts the metric from average partial accuracy into the probability that one run returns a fully correct answer. The threshold fits the financial tasks being discussed and should not be assumed for applications where partial results retain value.

Under that criterion, execution passes about 42% of runs, while the more complex strategies reach as high as 75%. A failed run now has an explicit business consequence: its tokens count toward the cost of obtaining the answer, even though its output cannot be used.

The talk rounds execution’s pass rate to about 40% and budgets approximately three attempts. Assuming 600,000 tokens per attempt, that produces the headline estimate of 1.8 million tokens for a perfect answer: 3 × 600,000 = 1,800,000. This is a practical three-run estimate, not a guarantee or a precise expected-value calculation.

Applying retry cost across the strategies changes the decision. Advising and grading are described as comparatively token efficient because extra work inside a run can reduce the need to repeat the whole task. If cost per usable result is the priority, the recommendation for this domain is advising. If single-run reliability matters more, grading or dreaming becomes more attractive. Token efficiency and per-run reliability are related, but they are not the same objective.

The implementation separates two layers. An individual agent harness supports execution. Above it, a meta-harness coordinates the executor, adviser, grader, dreamer, and any other participating roles. The lower layer makes one agent capable of doing work; the upper layer decides how multiple kinds of work fit together.

The concrete lower layer in the presentation is Claude Managed Agents, Anthropic’s managed-agent offering on the Claude platform. Jiang says its architecture supplies the individual-agent harness, while orchestration above it coordinates the roles in a strategy. She also says some capabilities—including dreaming—are available there out of the box. The talk does not detail the harness’s internal components or provide an implementation guide for those managed capabilities.

What does the composed control flow look like? A task enters the executor, which can consult an adviser while working. The result goes to a grader. A failure returns to execution; a pass proceeds to dreaming, where findings are written to memory for the next run.

The relationship visible in this workflow is temporal: advice changes the current attempt, grading governs whether that attempt may finish, and dreaming changes future attempts. Combining them can cover all three moments, but the talk presents this composition as an architectural construction rather than a separately benchmarked result.

These roles are primitives, not a closed taxonomy. Builders can introduce new jobs and more complex coordination patterns for their own tasks. The longer-term goal is for models and the platform to construct strategies dynamically as work proceeds. That automatic construction is presented as future direction; meanwhile, builders can combine the available primitives and evaluate each strategy against the outcome, reliability target, and cost that matter for the application.