From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases
Read the talk
From Zero to Leaderboard: Agent Evaluation
Wolfram Ravenwolf builds an evaluation workflow that connects benchmark conditions, repeated runs, costs and execution traces—then uses it to explain why a perfect score, a cheaper model and more reasoning can each mislead.
From a talk by Wolfram Ravenwolf
At a glance
Ideas worth remembering
Evaluate a complete configuration: dataset and runner versions, sandbox, agent, model endpoint, provider route and settings all affect the result.
Repeated runs reveal different things: average and range describe run scores, while consistency and ceiling track which tasks succeed every time or at least once.
Compare total token consumption and estimated cost. A lower token price can still produce a larger evaluation bill.
Reasoning effort competes with execution time. Under a fixed deadline, longer deliberation can reduce the number of useful command-and-feedback turns.
Keep task traces connected to aggregate scores, and distinguish agent failures from infrastructure or inference failures before choosing retries.
One average leaves the important questions open
A leaderboard can show a convincing average while leaving an agent builder’s questions unanswered. Was this a lucky run? Does the result hold with the agent you actually use? What did it cost? Wolfram Ravenwolf, an AI evangelist at CoreWeave and Weights & Biases, introduces WolfBench through those practical concerns. His own agent choices have changed among OpenClaw, Hermes Agent, Claude Code and Codex; the useful comparison must account for those changes.
“A single run is benchmark lottery.” WolfBench repeats a configuration five times, spending more evaluation budget to expose variation. Five is a practical choice rather than a guarantee of certainty: more repetitions can strengthen a comparison, at additional cost. Publishing the underlying results lets someone else investigate why their experience differs from the headline score.
The object being evaluated is a configuration. It includes the dataset, model endpoint, agent harness, sandbox and settings. An inference-server update can change behavior without changing the model’s name. Updating an agent can change its prompts or execution behavior. A sandbox provider can change its backend. Recording these conditions is what makes a score interpretable; repeating a model name alone does not reproduce the experiment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose six layers, then keep the comparison stable
The workshop assembles six swappable layers: benchmark, evaluation runner, sandbox, agent harness, model service and settings. Python, uv, Git and a terminal support the local workflow. The demonstration uses W&B inference and sandbox access, but the method also allows other providers. The runner coordinates the experiment; the agent harness is the system actually attempting the task.
Terminal Bench fits the intended workload because its tasks extend beyond writing code: repair a Git repository, recover a password or repair a database. Benchmark choice starts with the work the agent should do. A GUI agent or a mathematics system would need a different task collection, and a team can create its own dataset when existing benchmarks miss its needs.
WolfBench retains Terminal Bench 2.0 to preserve its existing comparisons. Ravenwolf describes 2.1 as fixing 28 of the 89 tests: useful corrections, but enough to require retesting the configurations. This is the cost of a stable historical series. Keeping the old version preserves comparability within that series while also preserving its defects.
Harbor supplies the evaluation runner. Other choices include Inspect, the LM Evaluation Harness or a custom runner; choosing a runner first and then one of its supported benchmarks is also reasonable. The important decision is to pin the runner version. New releases may add needed agent or provider support, so Ravenwolf suggests maintaining a fork and backporting necessary changes for evaluations that need tighter control. Installing Harbor through uv tool install keeps the tool in a separate environment.
A fresh sandbox prevents one attempt from changing the starting conditions for the next. Running agents directly on a laptop risks both interference with the host and contamination between trials. Docker is an option, though concurrent tasks can exhaust local resources. Ravenwolf used Daytona for earlier evaluations and uses CoreWeave Serverless sandboxes in the workshop. Harbor’s provider client, an API key and YAML configuration connect the runner to the sandbox service.
The agent layer changes what the model can do and how it interacts with the environment. Harbor supports harnesses including Cursor, Claude Code, Cline, Codex, OpenClaw and Hermes Agent. The workshop chooses Terminus 2, the default described for Terminal Bench: a comparatively simple harness that presents a CLI environment to the model and executes its keystrokes. A score obtained through that harness answers a different question from a score obtained through a daily-use coding agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompts, tools and context change what success measures
Context capacity belongs to the endpoint configuration. A model’s advertised maximum may exceed what a provider serves. In the demonstration, Ravenwolf describes an endpoint offering 256K tokens rather than a one-million-token maximum, trading capacity for serving speed and cost. He also reports quality degradation as context grows and favors earlier compaction. That is his operating judgment for these setups, rather than a measured universal optimum for context length.
Several settings change the behavior being tested in distinct ways:
- System instructions. An assistant prompted to help a person may ask a question and wait. An unattended benchmark has no person to answer. Ravenwolf has seen attempts fail this way; changing the instructions to fit autonomous work would change the evaluated configuration.
- Tool access. A web-enabled agent may find a published solution to a public benchmark. That can be sensible behavior for a useful assistant, while answering a different evaluation question from solving the task without retrieval. The trace reveals which happened.
- CPU and memory. A small sandbox may slow execution, run out of resources or crash. Its result measures that resource allocation together with the agent and model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Time limits, retries and provider routes belong in the experiment
WolfBench initially allowed two hours per task to avoid penalizing open-weight models served by slower inference systems. The extra time exposed a costly failure mode: an agent unable to solve a task could keep circling through unsuccessful attempts. Across 89 tasks and repeated runs, those loops consumed substantial time and money. Ravenwolf settled on one hour as a practical limit. That decision makes execution speed part of the result, alongside the ability to find a solution.
Concurrency shortens elapsed evaluation time by running independent task trials together. Running all 89 tasks at once would be attractive only if the infrastructure could sustain it. An overloaded inference endpoint may rate-limit requests; sandbox allocations also accumulate, with eight gigabytes per sandbox quickly becoming a large resource demand. Raising concurrency can therefore damage completion rates instead of simply accelerating the same experiment.
Retry policy needs a causal distinction. A sandbox-service crash can justify retrying because the task did not receive the intended conditions. A crash caused by the agent’s own actions can be a genuine failure worth counting. Retrying every unsuccessful attempt would hide behavior the benchmark is supposed to measure; counting every infrastructure interruption as a model failure would distort it in the other direction.
Sampling settings need the same care. Temperature, top_p, output-token limits and seeds may come from provider recommendations, host limits or agent defaults. Temperature zero reduces one source of variation but does not guarantee identical trajectories. Ravenwolf reports changed results after a graphics-driver update and points to floating-point rounding and performance optimizations as sources of variation. A fixed seed can reduce randomness without making one run representative of all possible runs.
Provider routing is another easily hidden variable. The workshop loads an API key from an .env file and can use either CoreWeave Serverless Inference or OpenRouter. With OpenRouter, the provider must be explicitly selected for the intended comparison: otherwise different tasks in the same 89-task run may reach different providers. Fixing the model identifier without fixing its route still leaves the serving conditions free to change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Launch one Git recovery task and preserve its evidence
The live run uses one small task rather than the full benchmark: find lost material in a Git repository’s tree and restore it. Ravenwolf expects roughly three to five minutes. He installs Harbor and the W&B client, writes the supplied configuration to a file, loads the API key into the environment and starts Harbor with that configuration. The demonstration tests whether the assembled pipeline works end to end.
What survives after the temporary environment disappears? Harbor fetches the configured Terminal Bench 2.0 tasks, creates a sandbox for each active task, installs the agent and gives it the task request. A verifier checks the outcome. Results and trajectories must be collected before the sandbox is destroyed: the score records whether the task succeeded, while the trajectory records what the agent did. Verification also takes time beyond the agent’s own execution budget.
Each task receives a binary score, one for success or zero for failure. A complete run turns those outcomes across 89 tasks into a success percentage. Repeating the complete run five times supplies the familiar average, but retaining the task-level outcomes also enables the reliability measures examined later.
One task produces both an outcome and a trajectory
Read the diagram as text
- Configured task. Harbor fetches the selected Terminal Bench 2.0 task.
- Agent in a fresh sandbox. Install the agent, provide the request and execute the task.
- Verify the outcome. evidence — Check whether the task succeeded; assign zero or one.
- Execution trajectory. evidence — Preserve what the agent did during the attempt.
- Collected run evidence. Retrieve results and trajectories before destroying the sandbox.
- Configured task → Agent in a fresh sandbox: Task request.
- Agent in a fresh sandbox → Verify the outcome: Outcome to check.
- Agent in a fresh sandbox → Execution trajectory: Execution history.
- Verify the outcome → Collected run evidence: Task result.
- Execution trajectory → Collected run evidence: Diagnostic history.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn run files into a valid comparison
While the task runs, the workflow moves into WolfBench. Harbor performs the evaluation; WolfBench collects and analyzes its outputs. The toolkit is presented as Apache 2 licensed and currently oriented around Terminal Bench, with adaptation required for other benchmarks. The downloaded results occupy about 13 megabytes because the much larger trajectories live separately in Weave. Displayed costs use published prices, including for evaluations run on the presenter’s own inference infrastructure, so they should be read as price-based estimates rather than his actual bill.
A Marimo notebook provides the dashboard. It can discover evaluation VMs on exe.dev or Hetzner, or scan a local directory as in the workshop. Running Harbor on a VM ordinarily lets an evaluation continue when the laptop shuts down or loses Wi-Fi. The local scan finds 443 unique runs, which then need filtering before they become leaderboard data.
Which runs qualify, and what settings did they actually use? A valid full run must contain all required tests; a partial test should not compete with a complete benchmark. The dashboard also filters errors and timeouts, though the recording does not specify those thresholds. Labels need attention too: an agent may translate or fail to support a requested reasoning mode, so the displayed setting must describe the mode actually used rather than merely repeat the input parameter.
The dashboard saves a shareable JSON results file, generates the chart and can upload traces to Weave when credentials are configured. Its uploader can publish the generated page by replacing the site’s index HTML. Collection, analysis and publication are separate operations, making it possible to share a results snapshot without downloading every execution trace.
The Git recovery run now finishes: one trial, mean score 1.0. The repository was repaired. That perfect score remains outside the default leaderboard because the run contains only one task. It validates this demonstration’s outcome; it does not establish a 100% result across Terminal Bench.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate observed capability from repeated success
The interactive leaderboard lets readers filter metrics and configurations, remove or reorder bars, and export a selected comparison. Its URL preserves the chosen view so another person can inspect the same arrangement. Those controls serve a useful question: are you choosing for the highest observed capability, a dependable base of solved tasks, or an acceptable spread between runs?
How can one bar answer those different questions? WolfBench combines five measures from the repeated runs:
- Consistency: the percentage of benchmark tasks solved in every run, shown as the dark base.
- Average: the mean run score, shown in yellow.
- Worst run: the lowest complete-run score, marked by the lower arrow.
- Best run: the highest complete-run score, marked by the upper arrow.
- Ceiling: the percentage of distinct tasks solved at least once, marked by a star.
The distinction is about task identity as well as totals. A 60% consistency base means the same 60% of tasks succeeded in all five observed runs. A ceiling counts the union of successful tasks across those runs; those successes need not have occurred together in any single run. Neither measure guarantees future behavior, but together they reveal how much observed capability survives repetition.
A run range from 60% to 79% exposes substantial variation that an average conceals. Another comparison has averages around 73% and 74%, yet a six-percentage-point difference in the consistency base. Ravenwolf favors the stronger base when average performance is nearly tied. In that example, higher reasoning effort improves consistency more visibly than it moves the headline average.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A cheaper token can still produce a bigger bill
The next comparison narrows the chart to two model configurations using Terminus: Gemini 3.5 Flash and GPT 5.5. Their scores are close, with Gemini 3.5 Flash approaching the higher-scoring GPT 5.5 configuration. Adding token consumption changes the reading: the score alone hides how much generation was needed to reach it.
What did those nearby scores cost to obtain? GPT 5.5 consumed 136 million tokens, compared with 1.65 billion for Gemini 3.5 Flash. The displayed price estimates were $497 and $620 respectively. Lower token prices did not offset the much larger token volume. More generated tokens also require serving time, although this comparison does not establish an elapsed-time ratio.
Failure behavior can be expensive too. The tested Fable configuration recorded 13 refusals and a displayed evaluation cost of about $11,000. Refusals explain part of its disappointing task score despite substantial spending. The trace matters here because the same low score could otherwise be mistaken for an inability to solve the underlying problems.
The agent harness also changes the economics of a fixed model. The highest-scoring Codex configuration with GPT 5.5 cost almost $1,000 in the displayed comparison, while Cursor with GPT 5.5 cost $282. Terminus had the highest observed ceiling. Those are different reasons to choose a configuration: best average, lower cost or broader demonstrated capability. Close scores may need ten or more runs to distinguish confidently, but evaluation budgets eventually require a decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More reasoning can leave less time to act
Increasing reasoning effort produced a surprising reversal in the Opus 4.6 evaluations: maximum thinking lowered the score, both through Terminus and through Claude Code. GPT evaluations showed gains as reasoning effort increased. The result therefore depends on the model and harness under the chosen limits; a setting named “max” does not promise the best task outcome.
The traces suggest a time-allocation problem. With one hour available, an agent may spend a minute or two deciding on a command, discover that the command is unavailable, then spend another two or three minutes deciding how to install it. A faster-acting configuration can try the command, read the error, install a dependency and correct the next failure through more interactions with the environment. Those errors provide information that deliberation alone cannot supply.
This does not make low reasoning effort the general winner. In the displayed comparison, Opus 4.6 already performed strongly with thinking off, exceeding a medium-reasoning GPT 5.5 configuration in both score and consistency. The useful observation is that additional deliberation can consume the time needed to exploit existing capability. The preferred setting must be tested with the intended execution deadline.
A separate failure came from departing from the task’s prescribed approach. Ravenwolf describes attempts in which a model pursued its own preferred method despite the benchmark’s instructions, and the final verifier failed the result. The trace distinguishes this instruction-following failure from an infrastructure interruption or an agent that simply ran out of time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow a bar down to commands, then classify the failure
“A bar is a claim, a trace is evidence.” Clicking a WolfBench bar opens Weave, where the five runs behind that aggregate can be inspected. Selecting a run exposes its 89 task calls. Selecting a task reveals its name and reward. This navigation keeps the summary connected to the attempts that produced it.
What happened between the request and the reward? The task trajectory contains the initial user request, the model’s analysis, its commands and subsequent steps. This is where a reviewer can investigate the source of a surprising result rather than infer it from a bar’s height. Weave also offers an agent to help inspect traces; Ravenwolf’s useful condition is that its findings link back to the underlying evidence so a reviewer can check them.
The review continues after every run. Infrastructure and inference problems need to be separated from genuine task failures before results are interpreted or retries are chosen. Repeated characters and leaked prompt-template tokens are warning signs Ravenwolf checks for tokenizer or inference-server problems. They call for investigation of the serving setup, especially when a newly released model has immature inference support.
An endpoint can improve even when the visible experiment configuration stays the same. Ravenwolf reports rerunning a new model a week later and obtaining better results after inference-engine improvements. That experience limits what a published score can claim about model weights alone: the measured system includes the serving software at the time of the run.
The ending leaves the tools open to substitution. Choose the workload, runner, agent and model that answer your question; keep the conditions explicit; share the results so others can inspect and extend them. WolfBench is one implementation of that method, and Ravenwolf invites additional model and agent evaluations so the evidence can grow beyond what one person can afford to run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Extends the value of execution traces from diagnosing evaluations to improving prompts, agent programs and repository skills through reflective optimization.
Read the complete timestamped transcript
- 0:12
Before we get started, if you want to follow along, though this is a practical workshop, so I see a lot of notebooks out there. Great. Um, if you, if you want to follow along exactly as I'm going, you should have a W&B account, a Weights & Biases account. If you don't have one, you can register and get a free one. And we also give you inference credits, so you can do an evaluation that way, and sandbox access is also included. And, um, if you want to use something else like O-like OpenRouter, you can do that as well, or
- 0:42
even a different provider, whatever you want to use. I will just use this as a example in the workshop. And if you have an AI agent, you can just tell it, change, uh, something, and make it work with my setup. So let's go. Quick about me. I'm Wolfram Ravenwolf, and my slogan is, "I have evaluate models for breakfast, build agents at night, and preach AI all day long." I do that as an AI evangelist at CoreWeave and Weights & Biases. And, um, I'm also a co-host on
- 1:12
Thursday AI, the podcast. Alex is also here. He's, uh, the main guy. And if you have technical issues during the workshop, you can contact him. He will help you out so we don't get interrupted. But he's your man . Thanks, Alex, for the help. And yeah, you can follow me on X or LinkedIn and Wolfbench's wolfbench.ai. Easy to remember. Um, but this workshop isn't just about Wolfbench. It's why and how I built it, so you can stand up your own model and agent eval from
- 1:42
zero to leaderboard. Um, yeah. My backstory basically I started as a little ch-child, uh, loved AI from the beginning. Then when ChatGPT happened, I decided I want to focus my career on this. Um, if anyone knows the old, good old times in local Llama when the block was quantizing models, I was testing the models. So I started that way. Was asked a lot about helping with inference and stuff, so I became a consultant. And, um, yeah, last year, I went full in,
- 2:12
in a G-German AI startup, and this year I joined CoreWeave. And I've been on the podcast for a long time, right, Alex? So but, uh, this is about you. So the workshop, who you are and why you are here. I think there are different kinds of people, and I'm interested in which category you think you belong to. So for instance, the agent builders. Are people building agents here that want to evaluate their agents? Someone here, someone here, a couple.
- 2:43
And eval newcomers. Uh, any people who want to do meaningful evaluations but have no idea what they are doing yet and want to learn this? Okay, those are more. And model evaluators. Who is interested in which model and agent should I trust, which is the one I want to use? Anyone interested in that specifically? Over there and over there. Great. Uh, platform and eval leads. The people who want to do comparable and reproducible evaluations. Someone here. Oh, a couple here. Great. Trace debuggers. Those are the people
- 3:13
who are looking at the actual traces and trying to find out what went wrong and how to improve this. Over here. Great. And skeptics. Who does not trust the leaderboards? I mean, now every hand should go up, right? But why is that, and how can we change that? And that is part of, uh, what we are talking about and how you can build your own. But first, my, my guiding principles are mindset is greater than skill set greater than tool set. So the tool set is what we are using, and that
- 3:43
changes in AI. It changes almost every week, every month. Something new comes out, and it changes all the time. Uh, so skill set is how we are using the tools that we have. That applies to different tools. So if you have a great skill, uh, how, you know how to use the tools, you can switch to other tools. But the mindset is the most important thing because that is the hardest to change or to improve. And for evaluation specifically, that is why we benchmark and what to watch out for. And skill set is just how to use the tools. And
- 4:12
tool sets, I will show you a couple of tool sets, and you can build your own that way, s-like I did with Wolfbench. And the premise is one score is not enough because performance is a distribution, not a point. The thing I disliked about the leaderboards and a lot of evaluations I have seen is you see some nice bars, and you see some numbers, and it's an average, an aggregate, and it doesn't really tell you very much. And that's why people always say or often say, "I don't trust this," and, "I have a
- 4:42
different experience." And there are things hidden behind the numbers that we can find out and look at. Because is it a really a good model? Did it have a lucky run? What are the hidden trade-offs? Stuff like that. And that's why I built Wolfbench because I wanted to know which models can I use with which agents. The agent changed over time. In January it was OpenClaw. In February it was Hermes agent. And, uh, I'm using a lot of Codex now. It was Claude Code before. And I still want to know which models work the best with which
- 5:12
settings. And yeah, like I said, most evaluation results collapse it to just one score. And most leaderboards rank the models. You don't even see the agentic systems behind it or the, yeah, different version settings, even if you even see what, uh, settings they have used for this. And they vary a lot, so you see some scores, and somebody else reports the same benchmark, but they use different settings. So can you really compare them? And is there a way to inspect the results those, those people got?
- 5:43
So Wolfbench is a framework. It has charts, dashboards, leaderboard. It's open source, by the way. And there's also methodology, and that is what I want to talk about most so you can use that methodology with your own tools. And for Wolfbench, um, one thing that is, that is applicable to all the benchmarks is you have to decide which metrics are interested to you. Do you want to look at an average? Do you want to look at the worst and best scores? Do you want to look at how many percent of the benchmarks, uh, it can solve at all,
- 6:12
and how much does it consistently? And also cost. That is also something that not everybody takes into account. How many tokens has it burned? How much did it cost? And it can happen that a cheaper and faster model takes longer and costs more than one of the bigger models because it just used so many tokens. Also very important are uniform conditions, so you have the same setup every time. Because if you change just the variable, one of the many that are there, and we will look at a bunch of those, if you change it, it is basically a new benchmark
- 6:42
now. And also multi-agent, if you want to do different agents, which ones do you look at? And always do multiple runs. A single run is benchmark lottery. It is a random result, and it is just a ballpark number. So you have to do multiple runs. I do five with Wolfbench, but you could do more if you have the money, of course. It's always a trade-off. Uh, transparency is also super important to me because I publish all the results and people can look at that. So I don't just show a score and say, "This is why this is my number one
- 7:12
model, number two model." I upload everything, and you can inspect it and check it out, and have your agent check it out, which makes a lot of sense with all the data we have collected. So basically, get the thinking right, and then the tools become interchangeable. And the most important thing I th- I think that people need to realize is you're not ranking a model. If you are benchmarking a model, it's this model, but you are actually benchmarking a specific configuration. So you use a data set, and if you change the data set, it's a new benchmark. You have the model,
- 7:42
and behind the model you have the inference engine and whatever is happening on the endpoint that you often can't control. But if they do an update for the inference software, it can change the results again, and y- you have no way to actually see that unless it's your own inference on your own system. Agent harness. Also, the agent harness, if I test Claude Code or Codex, it is a different version if I test it next week and I do the update. So that is also important to keep in mind. And same thing with the sandbox. Even if you are using a sandbox provider and they
- 8:12
change something on their back end, you may not even notice, and it could have an effect. And all the settings you choose yourself. So change any part, and it's a different experiment. You have to keep that in mind. Um, okay, here's a glossary. You will get the presentation, so I'm not going through all of this. Most of you are very familiar with this, and I will explain whatever, uh, is needed at the moment. So we are looking at a specific tool set because it's a practical workshop, and this is one way to build it, not the only way, and I will show you the different ways, and you can pick
- 8:42
your own stuff. So to follow along, you should have your laptop and your terminal. Python, UV, and G- Git are, uh, recommended. We have files to download, so if you are not following, you can download the results and import them and use the tool set anyway, so you don't have to follow along everything. But if you want to, you can do it and use your own results that way. Um, yeah, you should have the Wi-Fi, and an API key is optional. I showed it in the beginning. I will show it in a minute again, so you can get this. So here
- 9:12
again, take a screenshot, scan the QR code. This is, um, where you can get, uh, you get to a form where you can get free inference credits for the duration of the AI engineer conference. And to do that, you need a free W&B account. There, you can sign up if you don't have already, if you don't already have one. Who has W&B account already? Does anyone here use W&B? Bunch of people. Great. Okay, and, um, also the sandboxes, I'm using the CoreWeave serverless sandboxes, a newer offering,
- 9:42
where you have an isolated box for each task, which is super important because, um, otherwise you would change your own system with every evaluation run, and then it would be even more random. Okay, we are looking at six swappable layers, and we'll choose each one, and you can do the same. So this is, uh, the skill set, how to do it, and the mindset in the background. And now we choose the different tools for this toolkit that we are building. So we need a benchmark, which is a data set. We need an evaluation harness to run the data set and the
- 10:12
evaluation and, yeah, to orchestrate all of this and to run it in sandboxes so it doesn't run on your system. And also, we need an agent harness. Which, which agent do we want to use? And the model and service. That is, of course, usually the thing that people are most interested in. How does this model do? And the thing is, you are not just using a model, you are always using an endpoint. It's always on a provider. If it's Open Router, they are routing to other providers. There are a lot, and if it's
- 10:41
OpenAI or Google or xAI, whatever it is, they have their own setup, and you are also evaluating that in a way. And all the settings you can choose. So the first thing is, which benchmark do we use? You can start with the benchmark. I did it that way because I wanted to test agentic use with different harnesses. So I decided I want a benchmark that does that. So I looked at all the benchmarks that have been used by the Frontier Labs, and I saw Terminal Bench was on pretty much every one. And it's an agentic benchmark. It's not a
- 11:11
coding benchmark, not just. It's an agentic benchmark where there are tasks like, uh, like fix a Git repository, track a password, um, repair a database, stuff you would ask your u- agent in normal use, as an example. And that's why I picked Terminal Bench. But you could, if you are interested in multilingual stuff, MMLU is one thing, or if you are interested in multimodal stuff, MMLU Pro is something. If you are us- uh, interested in GUI agent use, then you would p- pick a different benchmark.
- 11:42
If you are interested in mathematics, coding, whatever it is, there's a benchmark for it. And the thing is, find the benchmark that is, yeah, most interesting to you and that is comparable. You could also make your own, of course. You can make your own data set. And, um, regarding Terminal Bench, that is agentic, so that's why I chose it to benchmark agents. There are versions two point oh and two point one, and three point oh is in the making right now. I chose two point oh because two point one wasn't available when I started. Um, if you are starting now, choose the
- 12:12
latest version, two point one. But I'm using two point oh because ch- choosing a new version would be a new benchmark, and I could not compare the scores anymore. Um, it actually fixes only twenty-eight of the eighty-nine tests, but it would be a different benchmark for all the models. I would have to retest everything I already did. And I did a lot, and never enough, but I did a lot, so I don't want to make a new version of Wolfbench right now. I will do that at a later point, maybe with three point oh, or if a big new jump in capabilities
- 12:41
happens and we need a different benchmark. Evaluation harness. This is new. You have chosen which, uh, benchmark you want to use, which data set to use. Now you need something to actually run the evaluation. And Harbor is a default framework that is for terminal bench. There are others. Inspect is a great framework by the UK, um, AI Security Institute, which has a lot of different benchmarks you can use. It's also very popular. The LM Evaluation Harness, I used that years ago, so it's been around forever. And OpenAI has their own
- 13:11
eval stuff. You can build your own runner or one of the others. You pick what you want. Some people can also-- Some people start with picking their runner, and then they pick one of the benchmarks it supports. That is also a valid way. And now for the first practical step, let's install Harbor. In this case, I'm using the current version. It was current yesterday . I'm not sure if it's current right now. It changes a lot. But I pin it to the ac-- to, to the version. And I highly recommend people stick to one version because
- 13:41
any time you change one of the variables, if it has an effect on the evaluation, and many have, then yeah, you are changing already a variable and the compares-- comparability is suffering. Even if it has a bug, you are at least testing all the models with the same bug. But it's, uh, visual thinking, of course, because sometimes you have to update if a new version of the, uh, harness, of the evaluation harness, um, supports a different agent or provider, then you may be forced to update. What I recommend for the people who are really serious about this, fork
- 14:11
the repository, maintain your own fork, and just, uh, backport whatever you need. And if you install it, this is, uh, there's a line break missing. It's UV tool install Harbor, so it's installing in a UV environment, so it doesn't affect your system, and you can cleanly uninstall it again. Then you should have this version. The next step, agents need a sandbox because if you just run it locally on your own system, the second agent running, it would find a system that has already been changed,
- 14:41
and it would interfere with your system, so I would not recommend that. You can use Docker. That is very common. But, uh, it's very hard to parallelize that depending on what system you are using. So you need a lot of resources. So that's why there are cloud providers. And I have to give a big shout-out to Daytona. They supported me in the beginning before CoreWeave had their own sandboxes, and I wanted to start with this. So they really supported me a lot. And, um, I've done a lot of the evaluations I have, uh, with Daytona. Now we have our own offerings, so I'm dogfooding it. And,
- 15:11
um, that's why there are two options here that I recommend. The CoreWeave sandboxes are for those using the Kubernetes service. And the, the golden one here is the CoreWeave Serverless sandboxes. You just need your W&B account. Do you wanna show them the download PDF of the slides so some people take it home? Ah, okay. We also have a-- Um, I, I-- Can you-- Well, I, I turned off Slack, so it's not shared here. If anyone wants to get the PDF right away, just go to Alex.
- 15:41
It's, it's probably the easiest way right now. Um, okay. So after you have installed this, uh, Harbor is installed. Now we need a sandbox client that is part of Harbor. And in this case, they support a couple of things. Here's the QR code for the presentation.
- 16:03
It works with Zoom as well.
- 16:08
Yeah. Good. Um, UV tool install Harbor in square brackets Wantbee installs a provider to use the sandboxes. Then you don't have to do anything. Once it's installed, you just have to have your API key. And you configure this in a YAML file. And here's just the next step. You don't have to copy this. There is a link and an easy URL you can just enter with the full configuration a couple of slides ahead. So just so you-- This is just so you see it.
- 16:38
But if you are following along, you should just type the UV tool install command to install the Harbor Wantbee provider if you are using this. Let's continue. Next step, agent. This is the most, most fun part for me because there are so many agents and I want to test, uh, how does this agent do with this model. And, um, here's a list of some of the agents. These are all the agents, and there are even more, that are supported by the Harbor framework. So we have Cursor, we have Claude, we have Cline, we have
- 17:07
Codex, OpenClaw Hermes agent. Um, and Terminus 2 is the default agent for the Terminal Bench and Harbor framework. And this is not a real agent. You can't use it for your coding stuff, or you shouldn't. It's more like simulating a CLI environment to the actual model, and it sends keystrokes, and the agent executes them. It's one way to do this. So an simple agent, which is the default, if you are seeing Terminal Bench scores somewhere and nobody said it's tested with
- 17:37
a different agent, that is the agent they used. And then which model do you want to test? You have to pick a model, and you have to pick a provider. If you are us-using GPT 5.5, you will be using OpenAI. If you are using Claude Fable, lucky if you are to use it, then, uh, you will be using Anthropic. Also, Google also provides, uh, some of the Anthropic models. And these are just some of those. Um, Open Router provides almost all of them, probably all of them,
- 18:07
even with DDI enabled, so you are not sending any data to, uh, the model vendors. And we also have our own inference, CoreWeave Serverless Inference, where you also get a selection of open source models. And in this case, what I wanted to test is GLM 5.2, which was the latest, uh, model I tested and, uh, also the best open source model, open weights model. So I just picked this for the demonstration. You could choose any other, of course.
- 18:36
And now the thing is the settings. There are a lot of settings. Some are very familiar to people, some maybe not as much. So the context window size is also very important Because you always see this model has this context window size, but every provider has the, yeah, some providers choose to deliver a different size, a smaller size for performance opt-- performance reasons. Like our own inference also doesn't offer the full one million of G, uh, GLM five point two, only two hundred and fifty-six k. But that way we can
- 19:07
serve it faster and, um, the quality of the model dec-dec-decreates after a certain amount of tokens. So basically half context is already very, um, very much noticeable. And the thing is, it costs a lot if you always go to the million and then you compact. So I personally think two hundred and fifty-six k, that's why I recommended it to our inference team to use this in that situation. Um, I also use the same with Codex. Um, basically right now that
- 19:37
gives you the best quality at the best price performance and speed. Thinking and reasoning effort. This is a big factor and I have done different evaluations with different thinking modes and they really have a big, uh, effect on the model and sometimes a very surprising effect where if you raise the thinking it goes down, but I will tell about this when we get to the slide. So system prompt. You don't always have the ability to change the system prompt or even inspect it, but it is a very important factor. And if you think about an agent like
- 20:07
OpenClaw or Hermes agent, you have the SOUL.md where you can put a lot of instructions. And now imagine you are using this with the benchmark and if you give the instruction, you are a helpful assistant, it will be very different if you give it an instruction it is an autonomous agent doing stuff. I have seen results where the model or the benchmark delivered a bad result because the model tried to ask the user, but there was no user and it didn't know. So if I change the system prompt and tell it you can ask the user, it would
- 20:37
behave differently and it would get a better score, I'm pretty sure. And that would be completely changing what I'm testing. So there are so many variables here. Also tool config. If you give the agent web access, I've seen where a model went out and looked up the solution for the benchmark because Terminal Bench is a public benchmark and you have to look in the traces to see what happened here. It, it's not really-- it's not even a bad thing. If you want an agent, it should be smart and if there's a solution for the problem, why should I try to think of
- 21:07
it if it, there's a published solution? So it depends what you really want to measure and if you want to give it access or not. And not all of those do that, or not every time. Sandbox resources, other thing. If you give it less CPU or memory, it may run out or it be maybe, uh, slower so it takes longer to do something or even crash and then you get a bad result. And it's not the model's fault, it's because of the sandbox configuration. Task timeout. Every task gets a timeout. By default, they range from I think five
- 21:37
minute, fifteen minutes to over three hours. One is three hours something and another one is two hours or so. And that means you have to decide how much time you want to invest. When I started with Wolfbench, I gave every task a fixed two hours because I didn't want to, um, basically handicap open weights models which are often running on less powerful inference endpoints than the big SOTA frontier lab models. So if you are running DeepSeek on a local
- 22:07
system, it's usually not as fast as GPT on optimized systems. And that would skew the score if you don't give it unlimited time. But I noticed when I was going with two hours that, uh, if the model can't do the test, it sometimes just goes in circles and tries and tries and tries and fails and fails and fails. So it was very expensive and took a long time because if I do two hours per task and there are eighty-nine tests and I parallelize them depending on how much concurrency I'm using, which is the next point, it takes so long. It takes so long and it
- 22:37
costs so much and you don't get a result because it just can't do the test. So I noticed one hour is a good mark. If you use the default Terminal Bench, the, the, um, test duration will rise. Concurrency. That is what I meant. You don't want to wait ninety-eight, uh, eighty-nine hours or even twice the amount of time. You want to do concurrency, so you have multiple trials happening at the same time. But then you have to decide how high can I go. If I go concurrency eighty-nine, I can do
- 23:07
all the eighty-nine tasks of Terminal Bench at once. But if my inference endpoint is not fast enough and I get rate limited, then I will get bad scores or even no scores at the, the end. The same with the sandboxes. If I have sandbox provider, I get resources and if I use eight gigabyte per sandbox, yeah, it sums up quickly and maybe you get over the limit you are allowed. Retry policy, how often to retry something. That is also something if I don't retry a task and it fails because the sandbox crashed, I get a
- 23:37
bad score for the model. But it wasn't the model's fault. So there are errors that you should retry and there are errors you should not retry. Like if the model causes a crash on its own by doing something stupid, that is actually a fail you want to measure. Temperature, the usual stuff. Um, that is a difficult thing for a benchmark because which temperature do you use? Some model providers, they give you the recommended temperature to use with their model. Some don't. And some agents have a default. Some agents
- 24:06
de-decide based on the model. And there are so many combinations that you have to decide. Do you want to look for every model? What is the recommended one? If there is none, which do you use? How do you make sure that people who are doing the same benchmark are using the same setting you chose? It's difficult. And top P as well. Do you use ni-- zero point nine five or do you use one or do you use any other value? It's just one setting and there are so many others. And max new tokens. How many tokens is the model allowed to generate? You have to
- 24:36
decide on this because, um, there are recommendations by the providers. There are limitations by the ho-- by the, the hosters. You have to choose something and usually you have to configure it. And if you can, you can also set a seed so the randomness is reduced. We-- I-- When I started benchmarking I used temperature zero so there was less randomness. There was never no randomness. And I've seen things as small as a graphic driver update to change the actual results I got, even if everything was, uh, deterministic or looked like that. Because we have
- 25:06
floating point rounding errors, we have performance optimization, so many effects that change the result even if you use, uh, temperature zero. And nowadays, when the model is recommending temperature one, then you should not set zero, of course, otherwise you would be going against the recommendation. In that case, you could set a seed. But even if you choose a seed like forty-two, uh, is very popular. You know why. And the thing is, some model may be more, uh, have more luck with that seed than others have less luck with that seed. It's all
- 25:36
choices you have to make. Now, for this choice, which, uh, inference we want to use, I have on the left side the normal, uh, inference, CoreWeave Serverless Inference. If you want to follow along and you want to use OpenRouter because you don't-- didn't go through the, um, getting your own key here, you could use OpenRouter as well. In either case, you just use an .env file and put in the variable at the bottom here. You just put the variable, the key in the .env file, and then we will, uh, use that with the agent.
- 26:07
Um, if you use OpenRouter, you have to set some special settings to use our provider, the W&B provider. This is very important if you are using OpenRouter and benchmarking a model. There are so many providers, and by default, you get one of these. They decide which is the best at the moment. So if you are running a single benchmark run with eighty-nine tasks, you may get half the tasks with this provider and half the ta-tasks with the other provider. So you definitely have to set which one you want to use. And
- 26:37
after all this, you have chosen one part for each of the six layers we went through. We have chosen Terminal Bench for our evaluation. We have chosen Harbor and a specific version of that. We have cho-chosen our sandboxes. We have chosen Terminus two as the agent, GLM five point two, and we have specific settings. If you are doing evaluation, you will do the same. Whatever framework you use, whatever model you use, you have to make choices, and you should be very clear what you are using. This is important when you are
- 27:07
publishing your results, so others know what you have actually used and measured. So here is another QR code. This gives you the full configuration, so you don't have to copy stuff from the slides. And here, the formatting has been messed up a little. Excuse this. Um, if you download this configuration, I will show you in the next slide how to run it, and it will take about five minutes to actually get the results in this case. And I will also do the same. So if you want to download the config, here's
- 27:37
the QR code. And I will go to the next one. So let me just not just show it, but actually run it. So take a look at this. What we are doing, we are sourcing the .env file to get the API key we defined two slides back. And then we are running it with the UV run Harbor run config command and giving it the configuration you downloaded or created. And then we can actually get the results. I will do this right now in the shell. So the
- 28:07
first thing, installing the Harbor framework, executing the command. I will zoom in a bit.
- 28:17
So if my Wi-Fi is working the way it should. It's full. Okay, we are letting this run. I will go back in a minute. So continuing with the presentation. So we are running the thing, and it will do just one test. I didn't want to use eighty-nine tests for this, otherwise we would be sitting an hour overtime. And that is only the time I give every test. So afterwards, the verifier has
- 28:47
to run and check if it's correct. That takes also some time, so it takes even longer. But, um, Harbor pulls the tests, the Terminal Bench two point zero data set. It uses that from the configuration. So this is the part where it has, uh, Terminal Bench version two point zero in the config. Then it starts the sandboxes using a provider we are using. So it creates a sandbox per test, depending on the concurrency, how many it uses. Then it runs the agent. It actually
- 29:17
goes in the sandbox, installs the agent, and executes the test. Give it-- gives it a system, uh, user prompt, basically saying, "Here, that is what you have to do." The agent does it. And afterwards, it cr-- uh, collects the results from the sandbox before it is destroyed. And it gets the trajectories, really, what is the agent doing. You get a lot of information that way. And this small task, it's one of the smallest. It's just a g-Git repository where something has been, uh, lost, and it should re-- find
- 29:47
it in the Git tree and restore it. This takes about three to five minutes. Let's see how far we are here. It's installed now. Then we install our W&B client. And the next step, in this case, I'm just getting my API key from the 1Password.
- 30:11
And then I'm writing the configuration here. This is the one you can download. I'm just cutting it into a file
- 30:21
and loading the .env file. Now the API key is in my environment variables. And I will just-- This is a run command. Run the evaluation. So now it's starting to go out. Starts the sandboxes, install the agent, run it, and this is all happening here.
- 30:41
And while it's doing that, I will go back to the presentation.
- 30:48
Next part. Run as evidence, not only a score. The formatting is totally messed up here. I will fix the presentation, and we will upload it. This is just the directory of what is in the, in the evaluation folder. We can create this live as soon as it's done. Of course, it will take about five minutes, should it take. Right now, it's starting the environment, which is creating the sandboxes. So just letting it run. Oh, this is too early
- 31:18
And when we have that, we get one task and it is scored. It either, it gets a score of one or zero. Did it succeed or did it fail? If you do that for the eighty-nine tests, you get an amount how many tests it did, and then you do that over five runs, for instance, and then you get a percentage, and you know how many percent of Terminal Bench did the model get. And that is usually the score that the labs report. Most people just report the average across five runs. And this is the
- 31:48
installation of the Wolfbench toolkit. What we have done before, it was all the Harbor framework and Terminal Bench, and this is what I programmed, what I created. The Wolfbench toolkit is, uh, a way to work with this data because otherwise you now have a couple of files and you have some score, but how do you analyze it? So this is a collector, what Wolfbench does, and I will-- It's best if I show it when I open the program. So let me go back here. While this is happening, I will just start
- 32:18
it in a new shell.
- 32:23
And we just clone it and go in there. So it's open source, Apache two license. You can use it for your own projects. Of course, you have to adapt it if you are using something else than Terminal Bench, but it's easy to adapt nowadays. Just tell your agent. Quick question, who's using agents for stuff like that?
- 32:44
Not everybody? Oh, I'm surprised. I still, I still see a lot of notebooks, by the way, because some people now just pull out their phone, take a screenshot and say, "Okay, agent, go. Do whatever he's showing on the stage." Next year. Okay, so it's downloading. And I'm also downloading all the results I did with all the runs I did so far. So quick spoiler, I did, uh, four hundred and thirty runs right now across,
- 33:15
um... Where is it? Uh, it was four hundred and thirty runs. But anyway, here are the agents I tested so far. Claude Code, Hermes Agent, OpenClaw, Codex. Cline is not published yet. That is on my list and some other agents. If you are making an agent, contact me if you want to appear as well. And some models I tested, many on those W&B that was on our own inference. So the, the final cost was
- 33:44
unfortunately not as high as it is displayed for almost forty K here. So this is, uh, some of this is on our own inference, but I just use the prices that have been published. So let's see where we are here. It's downloaded. So th-the thirteen megabytes, those are the results. If you want to see all the trajectories, those have been uploaded to Weave and you can just inspect them. You don't have to, um, download gigabytes of data. Now, if you want to use, uh, Wolfbench, you can set some variables,
- 34:14
so it is also an uploader. But I will show you if I just start it.
- 34:19
And this uses a marimo notebook to, um, provide a dashboard, an easy way to handle all the evaluation data. So any moment now. There it is. So this is a dashboard. You can enter your information, where you want to upload the traces if you have the Weave stuff. Then you can discover VMs. If you are using exe.dev or Hetzner, it will find all your VMs on there, and you can filter which VM names are evaluation VMs. Because I don't run the Harbor framework on my
- 34:49
own laptop, usually. I run it in a VM, so I can turn off my laptop or my Wi-Fi doesn't go off and the connection to the sandboxes fails because that could also affect something. Um, in this case, everything is local. As soon as I download the stuff, which I did, so I can just scan my local directory and it will find all the run data I already did. So here are the four hundred and forty-three unique runs. And it shows you here a filter. So I-- every run that, uh, has a score is,
- 35:19
uh, is measured. It has to have all the tests. If some tests have not been run, it's not a valid rest-- run. It was just a test run. It shouldn't have too many errors or timeouts. And then I get a list of all of this stuff, and I can choose which models I want to create an, a nice chart for. In this case, I will just use all of them. Then you can also just change the names of the models, and sometimes you also have to change the thinking because the parameter you give can be changed by the agent. Like if you are
- 35:49
using a thinking extra high or if you want to use a thinking max mode and your agent doesn't support it, maybe it is, uh, called a different way, and then you have to change it so it's really reflecting what has been used actually. Now, after all of this is done, I can just create my results file. You can upload to Weave here if you have the key. I do not have set the key, so I right now don't have the button. Now I create a save file, and this is
- 36:19
one JSON file that contains all the results. You can share that with other people. And in the end, I generate a chart, which is the artifact I generate. So now from all the results I did, I get a new chart, and this is the same that I-- Then in the next step, I have an uploader as well. After the chart is generated, I just upload it to wolfbench.ai, and it's updated. If I activate this option, it replaces the index HTML, and it is online immediately.
- 36:50
So going back to the present. Oh, let's just check how far the benchmark run. Oh, it's done. See, it's done. Trials one. Mean score one point zero. So it had one task. It solved the task, so it worked.
- 37:04
It finished, and it fixed the Git repository. And this task is now in my f- directory, and I could import it, but it's not a full run, so it is not, uh, displayed by default. And once we have done that and uploaded this, we can look at what Wolfbench provides. This is also the mindset thing again. What do we want to look at?
- 37:27
And in this case, this is what we just did. I showed you the dashboard, and, uh, now we have to take a look at the leaderboard and what it means because the leaderboard itself, whatever it generates, it is not just an answer. It helps you to find a better question. Like, what are you looking at? Do you care about the capability of a model or the reliability of a model? And I will show you what that means by going to wolfbench.ai.
- 38:00
And the thing is, I will just make it a little smaller. So they are sorted from left to right, and I have all of the information here. Most benchmarks you look at, they just show the terminus two score, and they only show the average. So this is an interactive page, which is used not only to display results, but to create charts that you can then copy and paste and put somewhere else. There's a screenshot symbol, so I can show stuff on X, for instance. So if I want to show how did this model do, I can just mark it. I can remove some of
- 38:30
the bars or reorder the bars. So it's interactive. So I can create my report that way. And what-- It also has it in the URL, so everybody can... If you give the URL to this, they see the same thing you sh-had on your screen. So I'm switching over here.
- 38:52
And it looks a bit overwhelming sometimes because there are so many information, so, so many information on the same bar. Like I said, you can filter to different information to make it a simple bar. But I like to see this because the darker part at the bottom, that percentage is how many percent of the whole benchmark has a model solved in every run of the five runs, runs I did. So that is what I call consistency. This model and this agent, they have solved the tasks sixty percent of the whole benchmark
- 39:22
every time. And this means if that is a very high score, the model is very reliable because if it solves the test, you can expect it to solve the test all the time. While the yellow score is the average, the one score that usually is reported, and the up and down arrows around it, those are the best and the worst scores in one of the five runs. So if those are very close to each other, then there's not a high variance, so you can expect the same score. If they are very far apart, like we see here, from
- 39:51
seventy-nine percent to sixty percent, that means there's a lot of variance, and that is, uh, a sign that the model, uh, there's a lot of randomness involved here. You can't really trust it to execute the, the test. Even if it has a high score, it's a bit random if it does it. And the highest marks, the star, that is the ceiling. How many percent of the benchmark has it solved at all, at least once, the, the amount of tests? That tells you how, yeah, in an ideal case, that would be the score the model
- 40:21
should get all the time if it's perfect and always doing it. Although that would also be an indicator that it is probably benchmarks for this. Um, but it shows you how far it can go. So what you want is a very, very high ceiling, a very stable average, and a very high solid base. And if a model has the same score, and they are almost the same, you can still see that, uh, the solid base is different. And then in that case, I would pick one that has a higher solid base. Like here, it's
- 40:51
seventy-three percent to seventy-four percent. It's almost the same. If you just look at the average score, it's almost exactly the same thing. Could be, uh, variance, just variance. But this is a bigger difference. It's six percent points, and that is a noticeable gap. So you see the extra high thinking is better than the medium thinking, as we expected. But if we are looking at, um, that is also very interesting. I will show it in the presentation. Let me switch to the presentation. So these
- 41:21
were the different ways to imagine the model. Ah, risk and cost tolerance. I have to show you this because this is something. If you look at the second best model on my evaluation, it was Gemini three point five Flash. So this model, a Flash model, is almost as good as GPT five point five. So what gives? If I just look at Terminus, so we don't have so many bars, and I can also say I just want to compare these two. And yeah, it's a great model, Gemini three point five Flash. But if I look at how did it achieve that, I have a three-D
- 41:51
view which shows you how many tokens it used to achieve that. And it has been billions of tokens to get that score, while GPT five point five used much less. Let me zoom in.
- 42:06
So a hundred and thirty-six million tokens for GPT five point five compared to one point six five billion tokens for Gemini three point five Flash. And the thing is, if we also look at the price, which is another three-D view, this is the cost. And I can even show both at the same time. The cost is now a shadow. You see that the cost is over here. It's overlapped because it used so many tokens. But this shows that Gemini three point five Flash is very close to GPT five point five, but it used so many tokens
- 42:36
that it costs even more than GPT five point five would have cost. So it costs, in this case, six hundred and twenty dollar compared to four hundred and ninety-seven. So five hundred to over six hundred. So it's not that you... Yeah, you have to think about this. Do you want to use a fast and cheap model? But you have to compare, is it really faster and cheaper? Because all those tokens also take time. That is one thing. Going back to the presentation. And
- 43:06
it's a decision surface. It's not just a chart. So you load your s-snapshot, and you can add your own if you did it. If you execute the code, you would see them. And then you can filter, like I showed you, different me-metrics and models. You can reorder the bars and export your own. And then you can show this and make a social post about this or write an article, whatever you do. Um, you can do this. In this case, I was looking at which is the best model on this benchmark. So my number one right now is, uh, GPT 5.5
- 43:36
with Codex. It is even better than Fable because if you are asking what happened to Fable, that is also very interesting to take a look at. If I put Fable in here, it had thirteen refusals. So it refused, and if you take a look at the cost, it goes so far it was my most expensive benchmark. Don't tell anyone. Because it cost eleven thousand just to test Fable, and it failed because it refused so many tests. Whoops. And that is also
- 44:06
something, of course, you have to decide. Um, that is when people say, "Oh, I don't trust this benchmark. The, this model is in the first place." Look at what happened here. And the benchmark has to be transparent, so you can look at the traces and see what has happened. Then it makes sense suddenly. Okay. Going back. And yeah, like I said, the best model was Codex with GPT 5.5. But as you can see, it was pretty very expensive. Let's zoom in. It cost almost a thousand bucks.
- 44:36
While Cursor, with the same model, cost only two hundred and eighty-two. So that was very cheap compared to the other one. And it used a little less tokens here. But, um, yeah, that's-- You could say, "Okay, this is the best combination," but maybe the other one is from a price and performance perspective, the better one. And Terminus also had the highest ceiling. But if the scores are very close to each other, that could be runs. You have, would
- 45:06
have to do ten runs or even more to be very sure. But you have to make a, um, a choice sometime and pick one and go with it. This was also the thing that I just showed, Gemini 3.5 Flash compared to GPT 5.5. Lower score, bigger bill. So in that case, it would make more sense to use the other model. And this was also super interesting and surprised me a lot. When I tested Opus 4.6 at the time, I raised the
- 45:36
thinking to max and the score decreased. It really got worse. And I did the same with Claude Code because I used Terminus as well. And there I saw the same effect. When I do that with GPT, if I increase from thinking off to extra high or from medium to extra high, if I raise the thinking level, the model gets smarter, as you would expect. So why did it get worse with, uh, Opus? That was surprising to me. And the difference, if you look at the traces, is that you see the model, it thinks too much. It's overthinking. And that
- 46:06
means, uh, I give it one hour. And if the model thinks for a minute or two and then decides I want to execute this command, and the command doesn't exist on the machine yet, then it thinks another two or three minutes, and how do I install this? While an agent that is not even thinking, it's just doing stuff. So the first impulse is I run this command, get an error message. I run this command, get an error message. I install it, get an error message, fix it, and it gets more, uh, turns. So it can do more. That is the difference between thinking for an hour and not doing
- 46:35
anything or just trying, trying, trying, trial and error. And the interesting stuff is that the GPT models, they are, by default, they appeared not as smart as Opus. Opus is very capable out of the box. Even with thinking off, it already got a great score if you compare that here. It is, um, already with off better than medium here on, uh, GPT 5.5. And even the solid base was higher, so it did more tasks, uh, in every run. Yeah. So that
- 47:05
is stuff you have to look at. And if it's surprising, take a look at the traces to find out what is happening here. Model overthinking. Also, so the same with Fable, by the way, where the model was very smart, and it seemed to know it was so smart. So what it did is it decided to go this direction. And even if the benchmark said it should be done that way, it said, "No, I know better. I will do it my way." And then the verifier at the end, it ran, tested, and failed. So the model was too smart for its own good in some cases.
- 47:36
And this is also an important feature I have to show you. A bar is a claim, a trace is evidence. I'm talking about traces all the time. So we have the bar, which is what we see on Wolfbench. You have the nice bars. And the thing is, you can click any of the bars, any of these. You just click it, and you get taken to our W&B page where we have Weave, which allows you to inspect all the traces. In this case, I clicked one bar, and I'm seeing all the five runs that composed
- 48:06
that score. And I can now pick any of the runs. Go in here. And this I should remove. I'll show this later. Uh, go away. So we have eighty-nine calls. That are the eighty-nine tasks that I have been doing. And you can see all of the traces here. It's a lot, so it's loading a bit. And then you can go into any of these, and you see the task name, and you see the reward. Did it succeed or not?
- 48:36
And you see all the steps. You see the initial request, the user message that says, "You are an assistant. You are tasked to do this." Then you see what the LLM did as a first command, what analysis it did, and what it wants to do. And then the command it sends, and so on. And you can really inspect all of this, what has happened, and look at it, or use your agent and have this done. And we have a new feature, which is also an agent built into the platform. So you have an agent inside of Weave, and you can just tell
- 49:06
it to do this. There's a talk about this by my colleague. Tim Sweeney is doing it on July 1. There you can check it out. I use agents a lot because reading all the traces takes a long time. And if the agent gives you a link to Weave and you see that it's true what it says, that is also very good. So to recap, there are so many variables relevant to benchmarks. You have the benchmark version, Terminal Bench two, oh, two point one, the agent version, the model service and route, even updates to the infer-inference endpoint
- 49:37
Can change the results and the settings. Sandbox resources, like I said, if the m- if, if the agent runs out of resources, it will fail, and then you get a bad score for the model, although it wasn't the model's fault. Sampling, temperature, all things we discussed. Thinking defaults, like maximum not being higher if it thinks too long, and a smaller, faster agent, uh, reaction could work better. Timeout policy, retry policy. If you don't retry and there has been infrastructure error, you would get a bad score. So after every run, I
- 50:07
go through all of the errors and mistakes and check have there been inference issues. You can find that out if there are repeated characters or if prompt template, um, tokens leak into the actual response, then you know there's a tokenizer issue, an inference server issue. And especially with new models, that is a big thing. If you are benchmarking models as they come out, they are so new that the inference engine maybe has not the best support yet. So I have done evaluations where the new model wasn't that great. A week later, I did
- 50:37
the same. Everything was the same, and it was better because the inference engine has been improved. That is also a part of this. So use the method, bring your own everything. You have the building blocks. Check out the presentation. Think about what you want to do, which evaluation to use, which agent to use, which model to test. You have the mindset now to know what to look out for and apply this to make, uh, leaderboards and charts and evaluations that are more reliable than if you just run something and publish a score.
- 51:07
So on your own reproducible benchmark configurations. Share the results. If you want to work with Wolfbench, contact me. If you want to run some models, that would be great. Um, we could collaborate and publish the results so I don't have to run everything. And, uh, if you want to make an off spin, that's also great. Use another framework, whatever you want. And if you are building models or agents, feel free to contact me so I can benchmark them. I worked with some of the agent, uh, creators already and will publish results
- 51:37
soon. Yeah, that's basically the end of the presentation. I hope you could get something out of it. And if you have some questions, now is the time. Otherwise, you can catch me at the booth. And, uh, big shout-out to Alex for the help. The podcast, Thursday AI, it's every Thursday. And in the morning, it's the time zone. I'm from another time zone. 8:30, um, Pacific Time every week because it's so hard to keep track of what's happening, and it's hard for us as well, so we
- 52:07
are forced to do it, or we can report it to you. Okay. Thank you.