The Cheaper AI Model That Costs You More - Rayan Krishnan, Vals AI

Vals AI builds benchmarks that frontier labs cite in their model releases. Its Vals Index brings together results from finance, law, coding, healthcare, and other domains to estimate how capable models are at economically valuable work. But a buyer trying to equip a team has a narrower question. Which model should an engineer use on this repository? Even knowing which model leads on coding leaves that decision open.

Krishnan calls this the jagged frontier. Break the index into its constituent benchmarks and different models take first place. Build a benchmark for a particular repository, then break out performance by language and library, and the rankings fragment further. Those comparisons still leave out the harness the agent runs in, the time it takes, and the cost of getting an answer.

The gap matters as companies expand access. Krishnan cited figures showing enterprise AI spending roughly doubling year over year, with regular AI use at 90% and established ROI at just 7%. Groups that could track spending were better able to identify where returns came from. For others, the difficulty of connecting the bill to the work was itself a reason to hold back on a wider rollout.

Anthropic’s Opus 5.5 looked like the economical choice for tax work. Its quoted input and output rates were $4 and $20, compared with $10 and $50 for Fable 5.1. Both Opus rates were 60% lower. But the bill depends on how many tokens the model consumes at each rate, and Opus used more tokens trying to complete the benchmark.

In Vals AI’s tax evaluation, Opus ended up costing $14 more than Fable and scoring 3% lower. A team choosing Fable for that workload could get better answers while spending less. The harder question was how far that result traveled, even within finance.

Vals AI evaluates financial research, Excel modeling, and tax work separately. For Excel, Krishnan plotted performance against cost per task. The Pareto frontier consists of the options for which no competitor does at least as well on both measures and strictly better on one. It gives a buyer a way to see where paying more buys a better result.

Krishnan overlaid two sets of results on the same plot. Green represented all the Excel questions. Red covered only leveraged buyout (LBO) models. The entire LBO frontier sat lower, so the overall Excel score overstated what a firm specializing in buyouts could expect. The membership of the frontier changed, too. Fable 5.1 looked like the choice for peak performance across Excel, yet dropped off the frontier entirely on LBOs. A buyout team paying its premium could do better on the cost-performance tradeoff with another model.

Partial credit asks what fraction of the available rubric points a model earns on an average task. That matters when a professional can take a useful draft and finish it. A team sending an answer straight to a client needs a different denominator. Of all the tasks assigned, what fraction came back with every criterion satisfied? A model can reliably produce a useful starting point while leaving a direct handoff a coin toss.

Krishnan applies three further checks. Validity asks whether the tasks resemble actual work and the rubric captures the standard people apply to it. Reliability asks whether there are enough tasks and samples for repeated runs to yield consistent results. Trying a new model on five or six examples usually leaves that question open. Discriminability asks whether the test separates the candidates. If every model scores zero or every model scores 100%, the result offers little help choosing one. Including some work beyond today’s capabilities also leaves room to measure progress.

In finance, a rubric might check for items that should appear in a response. An agent can improve its chances of hitting them by offering three possible answers. The person who asked for a decision now has to make it themselves. To reward the intended behavior, the evaluation has to capture the deliverable the person wanted, including the requirement to commit to one answer.

The execution environment creates another opportunity to game a test. Vals AI has seen sandboxed agents try to call other models through its model proxy or seek internet access. The container and proxy have to enforce the tool access the benchmark assumes. Otherwise, a score attributed to one model can reflect help from another.

Vals AI had crossed six trillion tokens in evaluation alone, running tasks with as much parallelism as possible. For a long-running agent, the infrastructure has to maintain consistent conditions over time and resume an interrupted run. Krishnan expects teams to encounter those requirements at much smaller volumes.

There is also a missing step in many single-prompt benchmarks. A person gives an agent what seems like a complete spec, the agent asks a question, and the first version draws feedback that changes the next one. Measuring the initial response omits the work of getting from that spec to an acceptable final product. A multi-turn eval can include the clarification and revision in the task itself.

When Vals AI got free access to some coding tools, it gave its engineers unlimited tokens. In one month, a team Krishnan recalled as 10 or 12 people used $1.5 million worth, roughly ten times its salary spend for that month. The team liked the output. Its next experiment was to see how much of the usage it could cut while producing as much work.

GitHub already held both sides of a coding evaluation. An issue recorded what someone had asked for, and a pull request showed how an engineer had resolved it. By winding the repository back to the state before the fix, Vals AI could give agents the original task and judge whether they could do equivalent work. The internal tools became a product called Val Smith.

One result challenged the assumption that moving to self-hosted open models would save money. On the internal eval, GPT 5.6 Sol scored 2% better than Kimi K3 and cost $2.50 versus $11, even with Kimi running on an inference-optimized stack. Krishnan also reported that Sol was faster half the time. The team used its results to select tools, set usage limits, and recommend a specific tool and model for an individual GitHub issue. Krishnan estimated that spending fell about 90% while the team maintained its output.

A century-old law firm may have a rich record of what its lawyers consider excellent work. Finding the original request alongside the accepted response takes more digging than it does in GitHub. A client email thread may preserve both. Other pieces may live in SharePoint or a Databricks export. The value of those records is that they show how the firm handled a real request and what it was willing to deliver.

Once that standard is expressed in a rubric or a script, it can inform several decisions. Procurement can set limits against measured usage and bring credible alternatives into bulk-pricing negotiations. A router can select a model for the task at hand. A post-training effort can draw tasks from the same history while reserving a separate evaluation set to check whether the trained model improved. In each case, the firm’s own record supplies a definition of good work that a public model card could never specify for it.