Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics

A prompt change made a coding-agent run 28% cheaper. It did not make the agent use the code-search tool the experiment was meant to test. That discrepancy opens Stephanie Jarmak's practical case for treating agent evaluations as experiments. An AI engineer at Omni Analytics at recording time, she previously worked on Amp and evaluation infrastructure at Sourcegraph, including CodeScaleBench. Before that, she was an astronomer. The scientific habit she brings to agents is to question what a result actually establishes.

The Sourcegraph experiment concerned a code-search MCP tool. The intervention was a change to the agent's initial prompt, intended to encourage tool use; the run retained traces and produced several metrics. The question to inspect in the comparison is simple: did the intended behavior change alongside the cost? It did not. The 28% reduction was an observation about spending, while the agent continued to skip the tool.

The cheaper run could justify a separate investigation into cost variation. It could not establish that the code-search integration was working. Keeping those questions separate prevents an attractive secondary metric from deciding an experiment whose primary objective remains unmet.

A second comparison asks what happens when the model stays in the experiment but its surrounding harness changes. Jarmak describes a third-party test on roughly 63 Terminal-Bench tasks: one model performed better under OpenHands, while another performed better after a harness swap. She reports a difference of more than 30% in configuration effectiveness. The account does not specify whether that figure is a relative change or a percentage-point difference, so it supports the ranking reversal rather than a more precise numerical interpretation.

The harness is the surrounding setup through which the model receives instructions, accesses tools, and carries out work. A benchmark result therefore describes a configuration, not a model in isolation. If a harness change can reverse the ordering, choosing a model from a leaderboard without testing the intended setup can select the wrong combination for the actual job.

This gives public benchmarks and product evaluations different roles:

• Public benchmarks narrow the search. They provide a standardized reference set and a view of the wider landscape, helping a team shortlist configurations worth investigating.
• Internal evaluations support a decision. They measure changes to a product or process using tasks and environments that reflect its own work. A standardized internal benchmark can then make successive configurations comparable.

The law-firm example makes the relevance test concrete. Terminal-Bench results might help the firm's developers choose a coding agent; they do not automatically tell the firm how an agent will perform its other work. Match the benchmark to the task, then test the configuration as close as possible to the environment where it will run.

Once a shortlist exists, experimental design starts with the work the system must cover. Grouping tasks into families makes it harder for one convenient task type to stand in for the whole product. Jarmak built a broader coding benchmark because, at the time, the bug-fixing emphasis of SWE-bench did not cover the range of developer work she wanted to study. Understanding a codebase and fixing a bug are different tasks; a model can be effective at one without being equally effective at the other.

Three design choices deserve separate attention:

• Workload coverage: Select representative tasks and organize them into meaningful families. Tooling may need to work differently across those families.
• Repeated runs: Decide how many runs to perform because the same setup can produce different results from one attempt to the next. A single run cannot reveal that variation.
• Interaction effects: Test relevant combinations of changes. The model-and-harness comparison shows why the effect of one component can depend on which other component accompanies it.

These choices determine what a result can mean. A configuration that improves bug fixing may leave code understanding unchanged; a strong run may not repeat; a model that wins in one harness may lose in another. The experiment needs enough coverage to distinguish those possibilities before the result becomes a product decision.

Returning to the unused search tool, the next question is where the information path broke. Retain the traces, then begin with the context actually available to the agent. Was the tool available at all? If not, stop the investigation there and repair the setup. If it was available, inspect whether the agent called it and what values came back. Tool availability, tool invocation, and useful returned information are separate facts.

The diagnostic map below answers a narrower question than a final score: which prerequisite failed before the information could help? Its branches distinguish a missing tool from an available tool the agent ignored. Following the successful-call path still leaves another question: did the agent use the returned information in a way that changed the task outcome?

Even a successful tool correction can leave the final result unchanged. Jarmak describes the scenario of an agent finally using a much better tool, yet failing to benefit from its information. That moves the investigation downstream: the integration now works far enough to return useful material, but the agent or configuration still does not turn it into a better result. Keep non-use and failed attempts in view rather than examining only successful calls.

The search-tool comparison also exposes a measurement choice. In Jarmak's example, Claude used grep and returned 295 files when the task needed only six—the “entire kitchen sink” approach. The useful comparison is between the breadth of the returned set and the small set the task actually required. Counting whether relevant information appeared somewhere in the results would miss how much unnecessary material accompanied it.

Two retrieval measures answer different questions:

• Recall: How much of the needed material did the search retrieve? If the objective is to miss no relevant information, recall of one may be sufficient.
• Precision: How much of the returned material was actually needed? This distinguishes a focused result from one that includes the right files among many irrelevant ones.

Jarmak wanted a balance of precision and recall, expressed through an F1-style score, rather than rewarding retrieval breadth alone.

The business objective decides which tradeoff matters. A search that returns every possibly relevant item can be appropriate under a recall-focused objective and poor under a precision-sensitive one. Before adopting a single benchmark score, ask what its validator measures and why that criterion was chosen. A higher score answers the validator's question; it only answers yours if those questions align.

Evaluation overlaps with operational safety when a detected failure is supposed to block further action. In Jarmak's experimental software factory, work requests passed through approval queues and required checks, with regular audits. The agent could notice that something was wrong. The missing mechanism was a way to enforce a stop.

Separate the responsibilities:

• Detection: An agent or another checking system flags a problem.
• Enforcement: Code implements the gate that prevents work from continuing when required conditions are not met.
• Escalation: A defined path brings the blocked decision to a human when necessary.

A checker that produces the right warning has only performed the first job. Without enforcement, the workflow can continue through a failure it has already recognized.

Sending every alarm to a person creates a second engineering problem: review capacity. In a separate LLM-as-a-judge system Jarmak cites, a 1% false-alarm rate would produce 10,000 human reviews at the scale under discussion. The percentage sounds small; the queue does not. The design must account for whether domain experts can handle that load and how consequential the flagged cases are.

Alarm quality and downstream operations therefore belong in the same evaluation. A threshold does not become useful merely because it catches problems: its false alarms consume scarce attention, while the consequences of missed problems affect how much risk the organization can accept. The bottleneck may move from automated execution to expert review.

Retaining traces is necessary for diagnosis, but the record itself can be unreliable. Jarmak describes a security experiment in which researchers seeded issues into an agent setup and used a hash check intended to detect tampering. That check passed without noticing the injected issues. An independent comparison of records—including traffic history and model changes—detected all the seeded issues in the described experiment.

The comparison asks what each checking method can establish. Here, a passing hash check did not establish that the security-relevant history was sound, while the independent record comparison exposed discrepancies. The talk does not detail the hash coverage or the injection mechanisms, so this is a warning about relying on that particular check, not a general claim that hashes cannot protect records.

Observability needs a trust model of its own. Decide whether the retained record is reliable enough for the system's security needs before using it to explain behavior or certify a result. More traces help only if the relevant events are recorded in a form the investigation can trust.

At Omni, the workload design begins with internal production data: how users actually interact with agents. Those interactions are classified into task types, including Q&A ranging from easy to hard questions, dashboard building, and app building. These categories give the evaluation harness a way to compare changes against recognizable kinds of user work rather than an undifferentiated average.

The purpose is to identify where a system change adds value and where failures appear downstream. Even a carefully constructed internal suite remains a proxy for actual workloads. “Production is truth” is Jarmak's shorthand for the next obligation: after deployment, keep measuring the system in its real environment.

Shadow operation offers a way to gather information when directly changing live behavior would carry a larger blast radius. The choice is risk-dependent: obtain observations from realistic conditions while limiting the effects of an insufficiently trusted change. Offline results guide the deployment decision; measurements in the deployed environment test whether those results carried over.

The final decision is how much risk to accept before a human must intervene. In her software factory, Jarmak allows some autonomy for lower-risk decisions while making the proposed next action and its scope explicit. When human input is needed, the system identifies why a person should weigh in. That makes escalation a bounded decision about an action, rather than an unspecified request to inspect everything the agent has done.

The distinction to encode is when autonomous deployment is acceptable and when work must stop and escalate. It connects back to the earlier gates: risk criteria decide when human involvement is required, and enforcement must make that requirement real. More autonomy is a choice about which actions the system may take under which conditions.

Jarmak closes by offering reusable skills for evaluation design, harness traceability, and reliability procedures, alongside a roughly 300-page archive that agents can consult. These materials supply procedures to bring into a system. The decisions they support remain concrete: what work is representative, what result matters, what record can be trusted, and when the system must hand the next action to a human.