Wolfram Ravenwolf is an AI Evangelist at Weights & Biases by CoreWeave and the creator of WolfBench, a framework for comparing AI agents across repeated task attempts. His evaluations examine how model behavior changes with compression, output limits and execution settings—and distinguish a task solved once from a task solved consistently.
Local models and personally owned AI
Based in Germany, Wolfram developed his public evaluation work around local language models. His April 2024 comparison of Llama 3 tested 20 model versions across different formats and compression levels. He examined question answering and instruction following, separating tests with supplied background information from those without it. These comparisons explored how much useful behavior survived the choices required to fit a model onto available hardware.
Wolfram also published Miquliz-120B v2, a model constructed by interleaving layers from two existing 70-billion-parameter models with MergeKit. He adapted a merging recipe from Eric Hartford and credited the underlying models and community contributors. The contribution was model composition, rather than training a foundation model from scratch.
The model card illustrated Amy, his personal AI assistant, while distinguishing the underlying model from the prompting that shaped her personality. Wolfram has developed Amy since 2023, making personally owned AI an ongoing interest alongside his comparative testing.
What a benchmark result measures
By December 2024, Wolfram had expanded his testing into a comparison of 25 models through 59 benchmark runs. He chose MMLU-Pro’s computer-science questions because they related to his daily work and kept testing feasible on consumer hardware. Repeated runs exposed variation that a single result would conceal.
One consequential finding concerned QwQ’s output allowance. The reasoning model performed worse when its answers were cut short by a token limit; allowing longer output changed the comparison. The result depended on whether the model had enough room to finish its answer. His January 2025 follow-up also distinguished benchmark performance from everyday usefulness: a computer-science multiple-choice test could miss a model’s conversational manner or the visual capabilities it was designed to exercise.
WolfBench’s open-source toolkit extends this work from models answering questions to agents executing tasks. It uses Terminal-Bench 2.0 tasks and Harbor for orchestration; Wolfram’s contribution is the evaluation framework and tooling for collecting, curating and presenting the runs.
Repeated success: WolfBench reports average performance, best and worst individual runs, tasks solved at least once, and tasks solved every time. Together, these measures distinguish the capability an agent has demonstrated from the consistency with which it delivers that capability under the tested conditions.
Comparable conditions: Wolfram treats the evaluated system as six layers: benchmark, runner, sandbox, agent, model service and settings. Versions, provider routing, system prompts, resource limits and timeouts can all change the result. Recording those choices makes comparisons interpretable and helps identify what changed between runs.
Inspectable results: The toolkit links its leaderboard to W&B Weave evaluations and execution traces, allowing readers to inspect commands, messages and failures behind a score. Its collection workflow filters incomplete or invalid runs and distinguishes infrastructure failures from genuine task failures.
Wolfram’s agent-evaluation workshop makes the consequences concrete. An agent repairs a broken Git repository and earns a perfect score, but that run stays out of the default leaderboard because it covers only one task. He also examines why a model with a lower token price can cost more overall if it consumes enough tokens, and why greater reasoning effort can lower performance when it leaves too little time to execute commands. More output helped QwQ complete benchmark answers; more reasoning can hinder an agent working against an execution deadline. The useful allowance depends on the task and its constraints.
Alongside this work, Wolfram co-hosts ThursdAI, the weekly AI-news podcast, combining developer education with hands-on model and agent evaluation.
Wolfram Ravenwolf builds an evaluation workflow that connects benchmark conditions, repeated runs, costs and execution traces—then uses it to explain why a perfect score, a cheaper model and more reasoning can each mislead.
Evaluate a complete configuration: dataset and runner versions, sandbox, agent, model endpoint, provider route and settings all affect the result.
Repeated runs reveal different things: average and range describe run scores, while consistency and ceiling track which tasks succeed every time or at least once.
▶ Watch ↗