October 13, 2026Main stage4 min read

Beyond the Model Card: The Jagged Frontier of AI in Finance

A model can earn strong partial credit and still produce work that cannot be sent to a client. Rayan Krishnan shows how evaluations built from a firm’s own work can guide model choice, spending, and deployment.

Rayan KrishnanVals AICo-Founder & CEO

Watch "Beyond the Model Card: The Jagged Frontier of AI in Finance — conference livestream" on YouTube

From the conference livestream · 5:33:35–5:43:30 on YouTube. The player starts and stops at this session’s boundaries.

5:33:35 ↗

A useful draft and a finished answer are different outcomes

A model can satisfy most of a financial task's rubric and still produce an answer you cannot send to a client. Rayan Krishnan, cofounder of Vals AI, starts with the evaluation choice behind that gap: are you measuring useful assistance, or finished work?

Partial credit asks how much of a task’s rubric the model satisfies on average. Fully correct answers ask how often the entire result meets the standard. A model can be a strong collaborator under the first measure while still producing too many incomplete answers to send directly to a client. The evaluation should reflect what happens to the output next.

Cost also belongs at the task level. Token prices alone do not tell you what a completed piece of work costs. A model may use more steps, need more attempts, or fail tasks another model completes. Krishnan’s concern is the combination of quality and cost that the organization actually experiences.

Three properties determine whether the benchmark helps you choose:

Validity: The tasks resemble real work, and the rubric captures the standard that work must meet.
Reliability: Enough tasks and repeated samples distinguish a stable result from a handful of lucky examples.
Discriminability: The test separates models. If everything scores perfectly—or nothing succeeds—the scores offer little help choosing among them.

5:36:48 ↗

Evaluate the work, including the ways an agent can game it

An agent can solve the test without doing the intended work. Krishnan describes agents attempting to call other models through an evaluation proxy or gain internet access from a sandbox. The environment must make clear which capabilities are actually being measured.

A subtler failure appears in financial answers. If a rubric rewards the presence of expected items, a model can hedge by offering several answers. That may improve its chance of satisfying the rubric while making the response less useful to the person who needs one correct answer. Evaluation design has to capture the desired work product, including what should be excluded.

Long-running agents make the infrastructure harder too. An evaluation environment must remain consistent and resume after interruptions. Krishnan says Vals had crossed six trillion evaluation tokens, running with substantial parallelism; his point is that these operational problems appear well before that scale.

Finally, the task often does not end after one prompt and one response. People clarify a specification, answer questions, and revise an output after seeing it. A benchmark that omits those turns may measure a different process from the one users rely on.

5:39:06 ↗

Turn past pull requests into future purchasing decisions

Vals tried giving its engineers access to effectively unlimited tokens. Krishnan says a team of roughly ten to twelve people used $1.5 million worth in one month—about ten times its salary spending for that period. The tools were supplied with free access, so this is the token value he reports, not a claim that the team paid a $1.5 million invoice.

The output was useful. The next question was how to preserve it while spending less. The team built internal evaluation tools, later offered as Vals Smith, and used its GitHub history as a record of completed work: take a prior issue and its resolution, rewind the repository to the earlier state, and ask an agent to perform the task again. That gives the comparison a concrete destination instead of an impression that one assistant feels smarter.

The comparison below puts success, cost, and speed beside one another for the replayed work.

Source slide at 5:40:50. The point is to compare completed tasks across all three measures: a model’s token price alone leaves out both its success rate and how much work it takes to finish.

These evaluations identified models that were unnecessarily expensive for particular work and models that could do it at lower cost. In one internal comparison, Krishnan reports that a closed model both outperformed a self-hosted open model and cost less per task. For that workload, owning the weights did not make the completed task cheaper.

The results fed into two decisions: which tools to procure and what limits to set, and which tool/model combination to recommend for a particular GitHub issue. Krishnan reports a roughly 90% spending reduction while maintaining the team’s output.

5:41:29 ↗

Find where your organization stores its standard of good work

A software team has pull requests. A law firm or investment business needs to find its equivalent: the record of work already performed and the evidence of what counted as a good answer. Krishnan points to email exchanges, SharePoint, and data-platform exports as possible places where that history lives.

The valuable asset is the organization’s accumulated judgment. An evaluation can express some of that judgment as tasks and rubrics, making it usable for decisions beyond the first rollout. It can help right-size usage limits, support pricing negotiations, route a task to a suitable model, and test whether fine-tuning or post-training has actually improved the work.

That last use requires an external evaluation set: improving a system against its training tasks does not establish that it generalizes. The organization needs a standard it can keep testing against as the implementation changes.

Employees already carry much of this standard in their heads. Turn their judgment into tasks and rubrics, and the next model release becomes something the company can test against its own work.

A useful draft and a finished answer are different outcomes

Evaluate the work, including the ways an agent can game it

Turn past pull requests into future purchasing decisions

Find where your organization stores its standard of good work

Further reading