From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases

A leaderboard can show a convincing average while leaving an agent builder’s questions unanswered. Was this a lucky run? Does the result hold with the agent you actually use? What did it cost? Wolfram Ravenwolf, an AI evangelist at CoreWeave and Weights & Biases, introduces WolfBench through those practical concerns. His own agent choices have changed among OpenClaw, Hermes Agent, Claude Code and Codex; the useful comparison must account for those changes.

“A single run is benchmark lottery.” WolfBench repeats a configuration five times, spending more evaluation budget to expose variation. Five is a practical choice rather than a guarantee of certainty: more repetitions can strengthen a comparison, at additional cost. Publishing the underlying results lets someone else investigate why their experience differs from the headline score.

The object being evaluated is a configuration. It includes the dataset, model endpoint, agent harness, sandbox and settings. An inference-server update can change behavior without changing the model’s name. Updating an agent can change its prompts or execution behavior. A sandbox provider can change its backend. Recording these conditions is what makes a score interpretable; repeating a model name alone does not reproduce the experiment.

The workshop assembles six swappable layers: benchmark, evaluation runner, sandbox, agent harness, model service and settings. Python, uv, Git and a terminal support the local workflow. The demonstration uses W&B inference and sandbox access, but the method also allows other providers. The runner coordinates the experiment; the agent harness is the system actually attempting the task.

Terminal Bench fits the intended workload because its tasks extend beyond writing code: repair a Git repository, recover a password or repair a database. Benchmark choice starts with the work the agent should do. A GUI agent or a mathematics system would need a different task collection, and a team can create its own dataset when existing benchmarks miss its needs.

WolfBench retains Terminal Bench 2.0 to preserve its existing comparisons. Ravenwolf describes 2.1 as fixing 28 of the 89 tests: useful corrections, but enough to require retesting the configurations. This is the cost of a stable historical series. Keeping the old version preserves comparability within that series while also preserving its defects.

Harbor supplies the evaluation runner. Other choices include Inspect, the LM Evaluation Harness or a custom runner; choosing a runner first and then one of its supported benchmarks is also reasonable. The important decision is to pin the runner version. New releases may add needed agent or provider support, so Ravenwolf suggests maintaining a fork and backporting necessary changes for evaluations that need tighter control. Installing Harbor through uv tool install keeps the tool in a separate environment.

A fresh sandbox prevents one attempt from changing the starting conditions for the next. Running agents directly on a laptop risks both interference with the host and contamination between trials. Docker is an option, though concurrent tasks can exhaust local resources. Ravenwolf used Daytona for earlier evaluations and uses CoreWeave Serverless sandboxes in the workshop. Harbor’s provider client, an API key and YAML configuration connect the runner to the sandbox service.

The agent layer changes what the model can do and how it interacts with the environment. Harbor supports harnesses including Cursor, Claude Code, Cline, Codex, OpenClaw and Hermes Agent. The workshop chooses Terminus 2, the default described for Terminal Bench: a comparatively simple harness that presents a CLI environment to the model and executes its keystrokes. A score obtained through that harness answers a different question from a score obtained through a daily-use coding agent.

Context capacity belongs to the endpoint configuration. A model’s advertised maximum may exceed what a provider serves. In the demonstration, Ravenwolf describes an endpoint offering 256K tokens rather than a one-million-token maximum, trading capacity for serving speed and cost. He also reports quality degradation as context grows and favors earlier compaction. That is his operating judgment for these setups, rather than a measured universal optimum for context length.

Several settings change the behavior being tested in distinct ways:

• System instructions. An assistant prompted to help a person may ask a question and wait. An unattended benchmark has no person to answer. Ravenwolf has seen attempts fail this way; changing the instructions to fit autonomous work would change the evaluated configuration.
• Tool access. A web-enabled agent may find a published solution to a public benchmark. That can be sensible behavior for a useful assistant, while answering a different evaluation question from solving the task without retrieval. The trace reveals which happened.
• CPU and memory. A small sandbox may slow execution, run out of resources or crash. Its result measures that resource allocation together with the agent and model.

WolfBench initially allowed two hours per task to avoid penalizing open-weight models served by slower inference systems. The extra time exposed a costly failure mode: an agent unable to solve a task could keep circling through unsuccessful attempts. Across 89 tasks and repeated runs, those loops consumed substantial time and money. Ravenwolf settled on one hour as a practical limit. That decision makes execution speed part of the result, alongside the ability to find a solution.

Concurrency shortens elapsed evaluation time by running independent task trials together. Running all 89 tasks at once would be attractive only if the infrastructure could sustain it. An overloaded inference endpoint may rate-limit requests; sandbox allocations also accumulate, with eight gigabytes per sandbox quickly becoming a large resource demand. Raising concurrency can therefore damage completion rates instead of simply accelerating the same experiment.

Retry policy needs a causal distinction. A sandbox-service crash can justify retrying because the task did not receive the intended conditions. A crash caused by the agent’s own actions can be a genuine failure worth counting. Retrying every unsuccessful attempt would hide behavior the benchmark is supposed to measure; counting every infrastructure interruption as a model failure would distort it in the other direction.

Sampling settings need the same care. Temperature, top_p, output-token limits and seeds may come from provider recommendations, host limits or agent defaults. Temperature zero reduces one source of variation but does not guarantee identical trajectories. Ravenwolf reports changed results after a graphics-driver update and points to floating-point rounding and performance optimizations as sources of variation. A fixed seed can reduce randomness without making one run representative of all possible runs.

Provider routing is another easily hidden variable. The workshop loads an API key from an .env file and can use either CoreWeave Serverless Inference or OpenRouter. With OpenRouter, the provider must be explicitly selected for the intended comparison: otherwise different tasks in the same 89-task run may reach different providers. Fixing the model identifier without fixing its route still leaves the serving conditions free to change.

The live run uses one small task rather than the full benchmark: find lost material in a Git repository’s tree and restore it. Ravenwolf expects roughly three to five minutes. He installs Harbor and the W&B client, writes the supplied configuration to a file, loads the API key into the environment and starts Harbor with that configuration. The demonstration tests whether the assembled pipeline works end to end.

What survives after the temporary environment disappears? Harbor fetches the configured Terminal Bench 2.0 tasks, creates a sandbox for each active task, installs the agent and gives it the task request. A verifier checks the outcome. Results and trajectories must be collected before the sandbox is destroyed: the score records whether the task succeeded, while the trajectory records what the agent did. Verification also takes time beyond the agent’s own execution budget.

Each task receives a binary score, one for success or zero for failure. A complete run turns those outcomes across 89 tasks into a success percentage. Repeating the complete run five times supplies the familiar average, but retaining the task-level outcomes also enables the reliability measures examined later.

While the task runs, the workflow moves into WolfBench. Harbor performs the evaluation; WolfBench collects and analyzes its outputs. The toolkit is presented as Apache 2 licensed and currently oriented around Terminal Bench, with adaptation required for other benchmarks. The downloaded results occupy about 13 megabytes because the much larger trajectories live separately in Weave. Displayed costs use published prices, including for evaluations run on the presenter’s own inference infrastructure, so they should be read as price-based estimates rather than his actual bill.

A Marimo notebook provides the dashboard. It can discover evaluation VMs on exe.dev or Hetzner, or scan a local directory as in the workshop. Running Harbor on a VM ordinarily lets an evaluation continue when the laptop shuts down or loses Wi-Fi. The local scan finds 443 unique runs, which then need filtering before they become leaderboard data.

Which runs qualify, and what settings did they actually use? A valid full run must contain all required tests; a partial test should not compete with a complete benchmark. The dashboard also filters errors and timeouts, though the recording does not specify those thresholds. Labels need attention too: an agent may translate or fail to support a requested reasoning mode, so the displayed setting must describe the mode actually used rather than merely repeat the input parameter.

The dashboard saves a shareable JSON results file, generates the chart and can upload traces to Weave when credentials are configured. Its uploader can publish the generated page by replacing the site’s index HTML. Collection, analysis and publication are separate operations, making it possible to share a results snapshot without downloading every execution trace.

The Git recovery run now finishes: one trial, mean score 1.0. The repository was repaired. That perfect score remains outside the default leaderboard because the run contains only one task. It validates this demonstration’s outcome; it does not establish a 100% result across Terminal Bench.

The interactive leaderboard lets readers filter metrics and configurations, remove or reorder bars, and export a selected comparison. Its URL preserves the chosen view so another person can inspect the same arrangement. Those controls serve a useful question: are you choosing for the highest observed capability, a dependable base of solved tasks, or an acceptable spread between runs?

How can one bar answer those different questions? WolfBench combines five measures from the repeated runs:

• Consistency: the percentage of benchmark tasks solved in every run, shown as the dark base.
• Average: the mean run score, shown in yellow.
• Worst run: the lowest complete-run score, marked by the lower arrow.
• Best run: the highest complete-run score, marked by the upper arrow.
• Ceiling: the percentage of distinct tasks solved at least once, marked by a star.

The distinction is about task identity as well as totals. A 60% consistency base means the same 60% of tasks succeeded in all five observed runs. A ceiling counts the union of successful tasks across those runs; those successes need not have occurred together in any single run. Neither measure guarantees future behavior, but together they reveal how much observed capability survives repetition.

A run range from 60% to 79% exposes substantial variation that an average conceals. Another comparison has averages around 73% and 74%, yet a six-percentage-point difference in the consistency base. Ravenwolf favors the stronger base when average performance is nearly tied. In that example, higher reasoning effort improves consistency more visibly than it moves the headline average.

The next comparison narrows the chart to two model configurations using Terminus: Gemini 3.5 Flash and GPT 5.5. Their scores are close, with Gemini 3.5 Flash approaching the higher-scoring GPT 5.5 configuration. Adding token consumption changes the reading: the score alone hides how much generation was needed to reach it.

What did those nearby scores cost to obtain? GPT 5.5 consumed 136 million tokens, compared with 1.65 billion for Gemini 3.5 Flash. The displayed price estimates were $497 and $620 respectively. Lower token prices did not offset the much larger token volume. More generated tokens also require serving time, although this comparison does not establish an elapsed-time ratio.

Failure behavior can be expensive too. The tested Fable configuration recorded 13 refusals and a displayed evaluation cost of about $11,000. Refusals explain part of its disappointing task score despite substantial spending. The trace matters here because the same low score could otherwise be mistaken for an inability to solve the underlying problems.

The agent harness also changes the economics of a fixed model. The highest-scoring Codex configuration with GPT 5.5 cost almost $1,000 in the displayed comparison, while Cursor with GPT 5.5 cost $282. Terminus had the highest observed ceiling. Those are different reasons to choose a configuration: best average, lower cost or broader demonstrated capability. Close scores may need ten or more runs to distinguish confidently, but evaluation budgets eventually require a decision.

Increasing reasoning effort produced a surprising reversal in the Opus 4.6 evaluations: maximum thinking lowered the score, both through Terminus and through Claude Code. GPT evaluations showed gains as reasoning effort increased. The result therefore depends on the model and harness under the chosen limits; a setting named “max” does not promise the best task outcome.

The traces suggest a time-allocation problem. With one hour available, an agent may spend a minute or two deciding on a command, discover that the command is unavailable, then spend another two or three minutes deciding how to install it. A faster-acting configuration can try the command, read the error, install a dependency and correct the next failure through more interactions with the environment. Those errors provide information that deliberation alone cannot supply.

This does not make low reasoning effort the general winner. In the displayed comparison, Opus 4.6 already performed strongly with thinking off, exceeding a medium-reasoning GPT 5.5 configuration in both score and consistency. The useful observation is that additional deliberation can consume the time needed to exploit existing capability. The preferred setting must be tested with the intended execution deadline.

A separate failure came from departing from the task’s prescribed approach. Ravenwolf describes attempts in which a model pursued its own preferred method despite the benchmark’s instructions, and the final verifier failed the result. The trace distinguishes this instruction-following failure from an infrastructure interruption or an agent that simply ran out of time.

“A bar is a claim, a trace is evidence.” Clicking a WolfBench bar opens Weave, where the five runs behind that aggregate can be inspected. Selecting a run exposes its 89 task calls. Selecting a task reveals its name and reward. This navigation keeps the summary connected to the attempts that produced it.

What happened between the request and the reward? The task trajectory contains the initial user request, the model’s analysis, its commands and subsequent steps. This is where a reviewer can investigate the source of a surprising result rather than infer it from a bar’s height. Weave also offers an agent to help inspect traces; Ravenwolf’s useful condition is that its findings link back to the underlying evidence so a reviewer can check them.

The review continues after every run. Infrastructure and inference problems need to be separated from genuine task failures before results are interpreted or retries are chosen. Repeated characters and leaked prompt-template tokens are warning signs Ravenwolf checks for tokenizer or inference-server problems. They call for investigation of the serving setup, especially when a newly released model has immature inference support.

An endpoint can improve even when the visible experiment configuration stays the same. Ravenwolf reports rerunning a new model a week later and obtaining better results after inference-engine improvements. That experience limits what a published score can claim about model weights alone: the measured system includes the serving software at the time of the run.

The ending leaves the tools open to substitution. Choose the workload, runner, agent and model that answer your question; keep the conditions explicit; share the results so others can inspect and extend them. WolfBench is one implementation of that method, and Ravenwolf invites additional model and agent evaluations so the evidence can grow beyond what one person can afford to run.