How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

An AI engineer can build an agent without knowing enough to grade its work. That is the predicament Nathanael Weill describes at Air Canada: his team delivers AI solutions across operations, corporate, commercial, and loyalty, but each business unit has its own specialized tasks. A public benchmark cannot tell the team whether an agent handles those tasks correctly. The people who can tell are busy doing the work themselves.

Weill opens with a mock confession: “I'm a non-evaluator.” The joke has a practical target. Without a way to evaluate changes, a team can keep shipping agents while learning surprisingly little about whether they are getting better. His route out began with an earlier telecommunications project, where the gap between making a change and judging it became impossible to ignore.

In 2024, the assignment was a question-answering assistant for telecommunications engineers. Its information base included about 10,000 documents, with vendor documentation on top. Finding an answer required knowing which documents to consult, not merely searching a small collection of interchangeable pages. Retrieval-augmented generation, or RAG, looked like the obvious approach. As Weill puts it, “on LinkedIn, it seems super easy. It's not.”

The system took a question, retrieved relevant documents, and used them to formulate an answer. For this task, Weill estimated that finding the right page led to the right answer about 90% of the time. That observation made retrieval the main target for improvement; it was not a claim that the entire assistant had 90% accuracy.

The pipeline grew as the team tried techniques from publications. Those additions did not reliably help on its own documents. Worse, a change that addressed one problem could break a question that previously worked. Adding query rewriting or another reasoning step took minutes. Establishing whether the addition was an improvement took much longer.

Evaluation meant a weekly meeting: put five chatbots on a large screen, give each the same question, and ask experts which answers they preferred. The experts sometimes disagreed. Sometimes they first had to look up the answer themselves. The effective budget was one candidate solution a week, so engineering speed mostly produced a queue of changes waiting for judgment.

The eventual assistant delivered value: people liked it and used it. Other teams had tackled the same task with the same documents in successive attempts. Weill does not claim to know why his team's solution succeeded where theirs did not. What he can explain is the change that let his team test its own ideas much faster.

The reversal was simple: choose a page before asking a question. An LLM writes a question about that page. The agent receives the generated question and searches for the answer. Because the test began with a known source page, code can compare the agent's retrieved result with that page instead of asking an expert to reconstruct the answer from scratch.

Where does the expected result come from? The same page supplies both the material for question generation and the reference for grading. The generated question travels through the agent; the reference page stays outside that path and meets the retrieved result at the comparison. This checks whether retrieval finds its way back to the source. It does not, by itself, grade every aspect of the final written answer.

Automating that check changed the pace from one evaluation a week to dozens a day. Expert grades also improved over the following weeks. The faster tests helped the team remove unnecessary pipeline complexity and concentrate on the embedding model, chunking, and the parameter Weill calls the BM25 “alpha.” The recording does not define that parameter, so it should not be read as a specific formula or tuning prescription.

The important change was that simplification became testable. A new layer no longer earned its place merely because a publication made it sound promising; the team could check whether it helped its own retrieval task and whether it broke existing cases. Weill credits the test set as an important difference in how the team worked, while leaving the comparison with the other teams as his hypothesis rather than a controlled result.

At Air Canada in 2026, the new task was supplier-contract review. Experts read contracts clause by clause, identify noncompliance, and propose rewrites. Slow review can delay a needed supplier; missing a problematic clause can create costs later. The team again faced a specialized task, scarce expert time, and no ready-made benchmark that matched the work.

The simplified agent had four stages:

1. Spot the issue: Classify contract sections into domains, then identify defects or needed changes within them.
2. Find supporting documents: Retrieve material that supports a proposed amendment.
3. Write the amendment: Produce the proposed change.
4. Decide whether to keep it: Classify the amendment for retention or rejection.

The earlier RAG technique supplied a way to test document retrieval. It did not automatically supply expected answers for spotting defects, drafting amendments, or deciding which amendments to keep. The team needed additional ways to construct cases where it already knew what the agent should find. Weill introduces five approaches, including one that failed because the required historical data did not exist.

Approach 1: Compare original and signed contracts. The natural starting point was existing expert work. Obtain the original contract and its signed revision, compute a diff as with source code, and use the changes to identify flags the agent should have raised. But the business retained the final contracts rather than the complete version history. Negotiation further complicated that history. Without original-and-final pairs, the proposed test set could not be built.

Approach 2: Start from a rule and plant a violation. Instead of extracting the answer from an old review, define the problem during test construction. Choose a known rule, create a clause that violates it, and generate a contract around that clause. The expected result is the corresponding flag. Weill's example involves a cap above $1 million in applicable contract domains; the full policy conditions are not specified, so this is an illustration of test construction rather than a statement of a universal contract requirement.

The relationship to inspect is the planted defect and its expected flag: the contract can vary around the clause while the known problem remains the test's reference. Rules came from experts and existing documentation, and the team also devised rules for its tests. This produced 275 tests with known answers. The limit is coverage: generating many contracts from known rules does not establish that the rule collection includes everything a real reviewer needs to catch.

Approach 3: Break a signed contract by removing a clause. Begin with an accepted contract, delete a clause, and ask the agent to review the modified document. Now the expected finding is an absence: something that should be there is missing. That differs from finding objectionable wording already present on the page. Supplier contracts use different templates and contain different material, so the agent has to infer what belongs in this kind of contract.

What should change between the accepted contract and the version with a deletion? The modified version should trigger a missing-clause finding, while intact contracts included as controls should not acquire flags. Mixing those cases tests whether the agent distinguishes a real omission from an acceptable document. This benchmark exposed a substantial weakness: the agents missed most of the removed clauses. Weill gives no numerical detection rate, but the reported result is clear enough to redirect development toward a failure that ordinary review examples had not made so visible.

Approach 4: Test changes that should leave the result alone. Some tests do not require a new expert answer for each document. They require a known relationship between results. For the changes Weill describes, the contract's relevant substance stays the same, so the agent should not produce a different review merely because incidental details or presentation changed.

• Supplier details: Change the supplier's name or address where those changes should not affect the review. The number of flags should remain unchanged.
• Clause order: Reorder clauses without changing their content. The result should remain consistent despite the new arrangement.
• Repeated execution: Submit the same contract repeatedly and check whether the output stays consistent.

These checks ask whether the agent respects an expected invariance, rather than asking another model to decide whether each answer sounds good. Code can check the specified relationship between outputs. They test consistency under the chosen conditions; an unchanged answer can still be wrong, so these cases complement the planted violations and removed clauses rather than replacing them.

Approach 5: Fill template blanks from arrays of values. The experts already used contract templates with blanks they would normally fill themselves. For testing, the team instead supplied an array of possible values for each blank. Combining those values produced many completed contracts from a familiar starting structure.

Which completed contracts should trigger a breach flag? The answer depends on the selected combination of values—the tuple—not simply on whether a blank was filled. Some combinations break a known rule; others do not. Because the test builder knows that distinction, each generated contract can carry an expected flag or no-flag result that code can check. The useful comparison is between combinations with different expected outcomes, not merely between differently worded documents.

Reusing the experts' own templates gives this method a practical foothold: the team does not have to invent every contract's structure. It still needs to know which value combinations violate which rules. The templates multiply examples whose outcomes are understood; they do not remove the need for that initial domain knowledge.

For the contract project, Weill contrasts a four-week feedback cycle with a script that returns an evaluation in five minutes. Those timings describe access to development feedback, not the time needed to establish that the product is ready for deployment. Known answers make exact-match and code-based checks attractive because they can run quickly without another round of subjective assessment.

An LLM judge can be useful, but it introduced too much noise for this project. Its prompt and surrounding evaluation setup became another difficult system to design. If the judge changes its assessment unpredictably, developers have to disentangle changes in the agent from changes in the grader. Weill describes finding a suitable judge as almost as hard as solving the original task, and says the team did not find the right one here.

The preference is therefore conditional: when code can check the answer you actually care about, use it. The earlier examples create that opportunity deliberately. A retained source page provides a retrieval target; a planted defect provides an expected flag; a controlled document change provides an expected relationship between results. The work shifts from asking a model to judge an open-ended answer toward constructing a question whose relevant outcome is already known.

The same reversal can target agent behavior rather than document findings. In another project, the team knew which endpoints an agent should call and in what order. It could generate a story that called for that sequence, run the agent, and inspect the execution trace against the expected trajectory. Here the known answer is a pattern of actions, and the generated input is the situation that should cause those actions.

There is a design tradeoff. The method does not fit every task, and the team sometimes re-engineered a solution to make it evaluable, moving slightly away from the original design. That decision gives up some attachment to the initial approach in exchange for an observable target. A design the team can evaluate offers a path for comparing changes; one it cannot evaluate leaves that path uncertain.

Experts still matter. Once a better-tested agent receives real feedback, Weill's recommendation is to use that feedback to improve the tests rather than treating every comment as a one-off instruction to patch the agent. A missed issue can suggest a better generation prompt, a new kind of clause removal, or another test case. The expert's scarce attention then improves the collection of checks developers can run repeatedly.

“Write the answer first” is the closing instruction, but it comes with an important limit. Generated tests are a starting point and a tool for developers. Passing them is not the product's go/no-go decision. They let a team make and compare changes without waiting weeks for each judgment; they do not establish, on their own, that a contract-review agent is safe or complete enough to ship.