# How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

Nathanael Weill, Air Canada

[Talk page](https://ai.engineer/talks/c_SPW__jF9w-test-ai-agents-when-no-benchmark-exists) · [Video](https://www.youtube.com/watch?v=c_SPW__jF9w)

## How to Test AI Agents When No Benchmark Exists

Nathanael Weill explains how starting with a known answer made a telecom RAG system easier to improve, then applies the same idea to supplier-contract review at Air Canada. The useful trick is to generate inputs whose expected results can be checked with code—and to keep those development tests separate from the decision to ship.

### No benchmark—and little time from the people who know

[0:12](https://www.youtube.com/watch?v=c_SPW__jF9w&t=12s)

An AI engineer can build an agent without knowing enough to grade its work. That is the predicament Nathanael Weill describes at Air Canada: his team delivers AI solutions across operations, corporate, commercial, and loyalty, but each business unit has its own specialized tasks. A public benchmark cannot tell the team whether an agent handles those tasks correctly. The people who can tell are busy doing the work themselves.

Weill opens with a mock confession: “I'm a non-evaluator.” The joke has a practical target. Without a way to evaluate changes, a team can keep shipping agents while learning surprisingly little about whether they are getting better. His route out began with an earlier telecommunications project, where the gap between making a change and judging it became impossible to ignore.

### A RAG system that changed in minutes and learned once a week

[1:39](https://www.youtube.com/watch?v=c_SPW__jF9w&t=99s)

In 2024, the assignment was a question-answering assistant for telecommunications engineers. Its information base included about 10,000 documents, with vendor documentation on top. Finding an answer required knowing which documents to consult, not merely searching a small collection of interchangeable pages. Retrieval-augmented generation, or RAG, looked like the obvious approach. As Weill puts it, “on LinkedIn, it seems super easy. It's not.”

The system took a question, retrieved relevant documents, and used them to formulate an answer. For this task, Weill estimated that finding the right page led to the right answer about 90% of the time. That observation made retrieval the main target for improvement; it was not a claim that the entire assistant had 90% accuracy.

The pipeline grew as the team tried techniques from publications. Those additions did not reliably help on its own documents. Worse, a change that addressed one problem could break a question that previously worked. Adding query rewriting or another reasoning step took minutes. Establishing whether the addition was an improvement took much longer.

Evaluation meant a weekly meeting: put five chatbots on a large screen, give each the same question, and ask experts which answers they preferred. The experts sometimes disagreed. Sometimes they first had to look up the answer themselves. The effective budget was one candidate solution a week, so engineering speed mostly produced a queue of changes waiting for judgment.

The eventual assistant delivered value: people liked it and used it. Other teams had tackled the same task with the same documents in successive attempts. Weill does not claim to know why his team's solution succeeded where theirs did not. What he can explain is the change that let his team test its own ideas much faster.

### Start with the page, then generate the question

[5:51](https://www.youtube.com/watch?v=c_SPW__jF9w&t=351s)

The reversal was simple: choose a page before asking a question. An LLM writes a question about that page. The agent receives the generated question and searches for the answer. Because the test began with a known source page, code can compare the agent's retrieved result with that page instead of asking an expert to reconstruct the answer from scratch.

![Two flows contrast waiting a week for an expert to grade an answer with selecting the right page, generating a question and checking retrieval in code.](https://ai.engineer/api/talk-essay-media/df18835e49212a52d1cc881aa3081b0077df82bbc4540d3fc10a73f1f10587d0)

Choose the source page, generate a question about it, then check whether the agent retrieves that page. Starting with the page gives the test an answer key.

Where does the expected result come from? The same page supplies both the material for question generation and the reference for grading. The generated question travels through the agent; the reference page stays outside that path and meets the retrieved result at the comparison. This checks whether retrieval finds its way back to the source. It does not, by itself, grade every aspect of the final written answer.

Automating that check changed the pace from one evaluation a week to dozens a day. Expert grades also improved over the following weeks. The faster tests helped the team remove unnecessary pipeline complexity and concentrate on the embedding model, chunking, and the parameter Weill calls the BM25 “alpha.” The recording does not define that parameter, so it should not be read as a specific formula or tuning prescription.

The important change was that simplification became testable. A new layer no longer earned its place merely because a publication made it sound promising; the team could check whether it helped its own retrieval task and whether it broke existing cases. Weill credits the test set as an important difference in how the team worked, while leaving the comparison with the other teams as his hypothesis rather than a controlled result.

### Supplier contracts need more than a retrieval test

[7:31](https://www.youtube.com/watch?v=c_SPW__jF9w&t=451s)

At Air Canada in 2026, the new task was supplier-contract review. Experts read contracts clause by clause, identify noncompliance, and propose rewrites. Slow review can delay a needed supplier; missing a problematic clause can create costs later. The team again faced a specialized task, scarce expert time, and no ready-made benchmark that matched the work.

The simplified agent had four stages:

1. **Spot the issue:** Classify contract sections into domains, then identify defects or needed changes within them.
2. **Find supporting documents:** Retrieve material that supports a proposed amendment.
3. **Write the amendment:** Produce the proposed change.
4. **Decide whether to keep it:** Classify the amendment for retention or rejection.

The earlier RAG technique supplied a way to test document retrieval. It did not automatically supply expected answers for spotting defects, drafting amendments, or deciding which amendments to keep. The team needed additional ways to construct cases where it already knew what the agent should find. Weill introduces five approaches, including one that failed because the required historical data did not exist.

### First recover past decisions; failing that, plant a known problem

[12:00](https://www.youtube.com/watch?v=c_SPW__jF9w&t=720s)

**Approach 1: Compare original and signed contracts.** The natural starting point was existing expert work. Obtain the original contract and its signed revision, compute a diff as with source code, and use the changes to identify flags the agent should have raised. But the business retained the final contracts rather than the complete version history. Negotiation further complicated that history. Without original-and-final pairs, the proposed test set could not be built.

**Approach 2: Start from a rule and plant a violation.** Instead of extracting the answer from an old review, define the problem during test construction. Choose a known rule, create a clause that violates it, and generate a contract around that clause. The expected result is the corresponding flag. Weill's example involves a cap above $1 million in applicable contract domains; the full policy conditions are not specified, so this is an illustration of test construction rather than a statement of a universal contract requirement.

The relationship to inspect is the planted defect and its expected flag: the contract can vary around the clause while the known problem remains the test's reference. Rules came from experts and existing documentation, and the team also devised rules for its tests. This produced **275 tests with known answers**. The limit is coverage: generating many contracts from known rules does not establish that the rule collection includes everything a real reviewer needs to catch.

### A missing clause exposed a different failure

[13:59](https://www.youtube.com/watch?v=c_SPW__jF9w&t=839s)

**Approach 3: Break a signed contract by removing a clause.** Begin with an accepted contract, delete a clause, and ask the agent to review the modified document. Now the expected finding is an absence: something that should be there is missing. That differs from finding objectionable wording already present on the page. Supplier contracts use different templates and contain different material, so the agent has to infer what belongs in this kind of contract.

What should change between the accepted contract and the version with a deletion? The modified version should trigger a missing-clause finding, while intact contracts included as controls should not acquire flags. Mixing those cases tests whether the agent distinguishes a real omission from an acceptable document. Weill’s slide reports that the agents noticed only one in three deleted sections. The test gave the team a specific kind of omission to work on.

![A diagram shows deletion of a clause from a good contract, followed by a comparison against untouched contracts.](https://ai.engineer/api/talk-essay-media/7f3961d74c17983a0df9da6371a1e7a50adc71a913dedf4690bf18e0e5c0e454)

The team mixed intact contracts with copies missing a known clause. The agent flagged the deletion only one time in three.

### Change the document without changing the expected judgment

[14:46](https://www.youtube.com/watch?v=c_SPW__jF9w&t=886s)

**Approach 4: Test changes that should leave the result alone.** Some tests do not require a new expert answer for each document. They require a known relationship between results. For the changes Weill describes, the contract's relevant substance stays the same, so the agent should not produce a different review merely because incidental details or presentation changed.

- **Supplier details:** Change the supplier's name or address where those changes should not affect the review. The number of flags should remain unchanged.
- **Clause order:** Reorder clauses without changing their content. The result should remain consistent despite the new arrangement.
- **Repeated execution:** Submit the same contract repeatedly and check whether the output stays consistent.

These checks ask whether the agent respects an expected invariance, rather than asking another model to decide whether each answer sounds good. Code can check the specified relationship between outputs. They test consistency under the chosen conditions; an unchanged answer can still be wrong, so these cases complement the planted violations and removed clauses rather than replacing them.

### Turn familiar templates into many labeled contracts

[15:21](https://www.youtube.com/watch?v=c_SPW__jF9w&t=921s)

**Approach 5: Fill template blanks from arrays of values.** The experts already used contract templates with blanks they would normally fill themselves. For testing, the team instead supplied an array of possible values for each blank. Combining those values produced many completed contracts from a familiar starting structure.

Which completed contracts should trigger a breach flag? The answer depends on the selected combination of values—the tuple—not simply on whether a blank was filled. Some combinations break a known rule; others do not. Because the test builder knows that distinction, each generated contract can carry an expected flag or no-flag result that code can check. The useful comparison is between combinations with different expected outcomes, not merely between differently worded documents.

![Notice periods and payment terms become fill-in-the-blank variables. Code can check the resulting decision against those variables.](https://ai.engineer/api/talk-essay-media/08c24e3442590c7f2b8c5fc14cb30f096c3ee4765a5c32b8aff26d0ec6e137d3)

One contract template produces different test cases as the notice period and payment terms change. In the slide’s example, code flags a notice period under 60 days.

Reusing the experts' own templates gives this method a practical foothold: the team does not have to invent every contract's structure. It still needs to know which value combinations violate which rules. The templates multiply examples whose outcomes are understood; they do not remove the need for that initial domain knowledge.

### Do not build a second uncertain agent just to grade the first

[16:29](https://www.youtube.com/watch?v=c_SPW__jF9w&t=989s)

For the contract project, Weill contrasts a four-week feedback cycle with a script that returns an evaluation in five minutes. Those timings describe access to development feedback, not the time needed to establish that the product is ready for deployment. Known answers make exact-match and code-based checks attractive because they can run quickly without another round of subjective assessment.

An LLM judge can be useful, but it introduced too much noise for this project. Its prompt and surrounding evaluation setup became another difficult system to design. If the judge changes its assessment unpredictably, developers have to disentangle changes in the agent from changes in the grader. Weill describes finding a suitable judge as almost as hard as solving the original task, and says the team did not find the right one here.

The preference is therefore conditional: when code can check the answer you actually care about, use it. The earlier examples create that opportunity deliberately. A retained source page provides a retrieval target; a planted defect provides an expected flag; a controlled document change provides an expected relationship between results. The work shifts from asking a model to judge an open-ended answer toward constructing a question whose relevant outcome is already known.

### Use expert feedback to improve the tests—and keep a separate shipping decision

[17:29](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1049s)

The same reversal can target agent behavior rather than document findings. In another project, the team knew which endpoints an agent should call and in what order. It could generate a story that called for that sequence, run the agent, and inspect the execution trace against the expected trajectory. Here the known answer is a pattern of actions, and the generated input is the situation that should cause those actions.

There is a design tradeoff. The method does not fit every task, and the team sometimes re-engineered a solution to make it evaluable, moving slightly away from the original design. That decision gives up some attachment to the initial approach in exchange for an observable target. A design the team can evaluate offers a path for comparing changes; one it cannot evaluate leaves that path uncertain.

Experts still matter. Once a better-tested agent receives real feedback, Weill's recommendation is to use that feedback to improve the tests rather than treating every comment as a one-off instruction to patch the agent. A missed issue can suggest a better generation prompt, a new kind of clause removal, or another test case. The expert's scarce attention then improves the collection of checks developers can run repeatedly.

“Write the answer first” is the closing instruction, but it comes with an important limit. Generated tests are a starting point and a tool for developers. Passing them is not the product's go/no-go decision. They let a team make and compare changes without waiting weeks for each judgment; they do not establish, on their own, that a contract-review agent is safe or complete enough to ship.

## Key takeaways

- Generate a question from a known source page, then check whether retrieval returns that page. In the telecom project, this changed evaluation frequency from once a week to dozens of times a day.
- Known-answer contract tests can come from historical revisions, planted rule violations, removed clauses, changes that should preserve results, and filled-in templates. Historical revisions were unusable here because original-and-final contract pairs were not retained.
- Test omissions separately from bad wording. Removing clauses from accepted contracts exposed agents missing most of the deleted clauses; intact contracts supplied no-flag controls.
- Prefer code-based grading when the expected outcome is known. For this contract project, the team could not find an LLM judge whose noise and setup cost made it a suitable alternative.
- Use expert feedback to improve reusable tests, but keep development evaluation separate from release approval. Generated tests help improve an agent; passing them does not by itself establish readiness to ship.

## Slides

### Air Canada’s agents have no benchmarks

[1:19](https://www.youtube.com/watch?v=c_SPW__jF9w&t=79s) [Four boxes identify operations, corporate, commercial and loyalty teams. The question below asks how a nonexpert can test an agent when experts lack time.](https://assets.ai.engineer/sites/aiecode2025/sha256/09ef6bc2f3dfdb3369615362832abb29199095e751120ba55143eb610a544c56.webp)

Air Canada has lots of agents and no benchmarks
Operations · Corporate · Commercial · Loyalty
How do you test an agent when you’re not the expert, and the experts have no time?

### A telecom question-answering task

[1:51](https://www.youtube.com/watch?v=c_SPW__jF9w&t=111s) [Four cards lay out a Q&A assistant, the standards and vendor documents it searches, technical questions and the page that supports each answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/1cd5c5f94e88ff9c0d7b17e4ab087d8e07d5b5a1dfdfda5d8dac5d645b866d42.webp)

I hit this problem once before, in 2024
The agent: A Q&A assistant
The documents: Telecom standards and vendor documents
The task: Answer a technical question
A good result: The right answer, from the right page

### Four teams attempted the same task

[3:01](https://www.youtube.com/watch?v=c_SPW__jF9w&t=181s) [A four-column comparison shows the speaker’s team as the only one reporting success with the same documents and task.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee9ef6fea82522a2e11dd487bfbbff74799b24fa720a31a62981cee40f93011c.webp)

Four teams, same challenge. Only one made it work
Team A: didn’t work
Team B: didn’t work
Us: worked
Team C: didn’t work

### Weekly retrieval changes and flat answer quality

[4:45](https://www.youtube.com/watch?v=c_SPW__jF9w&t=285s) [A flat quality curve stretches across eight weeks despite changes to parsing, chunking, reranking and reasoning.](https://assets.ai.engineer/sites/aiecode2025/sha256/81972d11f1b06c91231add7de7c7a78a25f90f513a515cd0858e4f52ba42fe34.webp)

Every week we added a part. Quality didn’t move
Question → Retrieve → Answer
Better parsing · Chunker · Re-ranker · HyPE · Query rewriter · Reverse HyPE · Reasoning step 1 · Reasoning step 2
still flat

### Write the answer before the question

[5:51](https://www.youtube.com/watch?v=c_SPW__jF9w&t=351s) [Two flows contrast waiting a week for an expert to grade an answer with selecting the right page, generating a question and checking retrieval in code.](https://assets.ai.engineer/sites/aiecode2025/sha256/df18835e49212a52d1cc881aa3081b0077df82bbc4540d3fc10a73f1f10587d0.webp)

So we wrote the answer first
BEFORE: the expert grades the answer
NOW: we pick the answer, then write the question
Pick a page → LLM writes a question → Agent searches → Right page?

### Daily automated experiments

[6:44](https://www.youtube.com/watch?v=c_SPW__jF9w&t=404s) [One expert-reviewed experiment per week is contrasted with dozens of generated-question trials per day. A second chart shows an upward quality trend.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b230b2f0a5dbdeadc160b96b3b1ff85ea68d87f9b850acf42ec7ca674ce312a.webp)

Then we could test every change, every day
1 try per week · ~dozens tries per day
Every change got a score. Keep it if the score goes up, drop it if it goes down.

### What the test set made possible

[7:29](https://www.youtube.com/watch?v=c_SPW__jF9w&t=449s) [The successful team is the sole row with a test set. A highlighted statement connects measuring a change with improving the agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d805400c0048a979940c7937e2ebcfb234cd09965915f00618135d37b815ad6.webp)

Why the difference? I treat it as an axiom
If you can’t tell whether a change helped, you can’t make the agent better.

### From supplier contracts to suggested edits

[7:44](https://www.youtube.com/watch?v=c_SPW__jF9w&t=464s) [A flowchart follows a supplier contract through rule checks, suggested changes with reasons, and the human reviewer’s decision.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ea0bdfd4d74b672e3f333cd8229cbaeebf4658151a50c49952c9496bd8f1e6f.webp)

Now, in 2026 at Air Canada: a different challenge
Supplier contract → Agent checks each section against our rules → Suggested changes, each with a reason → A reviewer accepts or rejects each one

### Each contract-review task needs its own test

[10:45](https://www.youtube.com/watch?v=c_SPW__jF9w&t=645s) [Four stages split contract review into spotting an issue, finding a rule, writing a change and deciding whether to flag it. Search tests cover only the second stage.](https://assets.ai.engineer/sites/aiecode2025/sha256/374f893ad4bf1a7835f5789763a38ad7158f077a369e3b029e03a19761e4163f.webp)

We were stuck because each job needs its own test
Spot · Find · Write · Decide

### Five ways to construct a test

[10:58](https://www.youtube.com/watch?v=c_SPW__jF9w&t=658s) [An overview lists five sources for synthetic tests, from past decisions and rules to existing contracts and examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf510556ec6d3e1b423af4e1a467aed7065a303e45039b6f4565d05866f62963.webp)

Working the steps: five ways, no expert needed
Use past decisions as the answers
Choose the answer, then write the contract
Take a good contract and break it
Check how the answer moves
Turn one example into many

### Choose the answer, then generate the contract

[12:40](https://www.youtube.com/watch?v=c_SPW__jF9w&t=760s) [A four-step example starts with a contract cap, chooses an expected flag, generates a contract, and checks for that flag.](https://assets.ai.engineer/sites/aiecode2025/sha256/856ee5bb9faa26cd75aab2ab6f95fbc6a82a6c9f827038d88fad91f804003f17.webp)

2. Choose the answer, then write the contract
Take a rule · Pick the answer · Generate the contract · Does it flag $8?
43 rules → 275 answers, known before any test ran.

### Remove a clause to create a known defect

[13:57](https://www.youtube.com/watch?v=c_SPW__jF9w&t=837s) [A diagram shows deletion of a clause from a good contract, followed by a comparison against untouched contracts.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f3961d74c17983a0df9da6371a1e7a50adc71a913dedf4690bf18e0e5c0e454.webp)

3. Take a good contract and break it
A good contract · Delete one clause · Now you know the answer · Mix in clean ones
deleted sections were noticed 1 time in 3.

### Check whether an answer stays consistent

[14:45](https://www.youtube.com/watch?v=c_SPW__jF9w&t=885s) [Rows compare the agent’s flags after renaming a party, reordering clauses and rerunning the same contract.](https://assets.ai.engineer/sites/aiecode2025/sha256/09b519c9c54bcf2da1560675a962a42403fb70fc90bbd7bfe5c5cc4b8bf60041.webp)

4. No right answer? Check how the answer moves
Rename a party = same flags
Reorder the clauses = same flags
Run it again = same flags

### Generate variants from one example

[15:36](https://www.youtube.com/watch?v=c_SPW__jF9w&t=936s) [Notice periods and payment terms become fill-in-the-blank variables. Code can check the resulting decision against those variables.](https://assets.ai.engineer/sites/aiecode2025/sha256/08c24e3442590c7f2b8c5fc14cb30f096c3ee4765a5c32b8aff26d0ec6e137d3.webp)

5. Turn one example into many
Notice: 30d · Pay: net 60
notice < 60d → flag

### Four weeks of feedback versus five minutes

[16:27](https://www.youtube.com/watch?v=c_SPW__jF9w&t=987s) [Boxes compare delayed beta feedback with a scripted five-minute run. A second row orders checks from exact matching through human review.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ae81dede995321248327dd262a5107d103460355b79e4eea27b1f3e19765004.webp)

From four weeks to five minutes
BEFORE: 4 weeks
NOW: 5 min
Cheapest check first: Exact match → Code → AI judge → Human

### Applying answer-first tests beyond contracts

[17:53](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1073s) [A table applies the same three steps to tool-call trajectories and text-to-SQL, then invites the reader to apply them to another agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc5a7e6b83c3d5321690855ead3ea51fc2fae1a8978f61ce0907945de4d83529.webp)

The same trick works far beyond contracts
1. Write the answer
2. Generate the input
3. Run and check
Trajectory · Text to SQL · Your agent

### Real feedback improves generated tests

[18:25](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1105s) [A loop connects generated tests to the agent, real feedback and new real tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/76a5d66b43cb5de0bbb057d06856d69215b561ed4f37dd93581a1908243536b3.webp)

Real feedback makes the tests better over time
Generated tests → Agent → Real feedback → New real tests

### Generated tests as a starting point

[18:58](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1138s) [Two closing rules distinguish tests that let engineers measure changes from the human checks needed for shipping.](https://assets.ai.engineer/sites/aiecode2025/sha256/2bf4b0e45841f1692090b31e8df70fc14ab526ca69f3d1e63e7604aaec1b9ab7.webp)

No benchmark? Write the answer first
Write the answer first.
Generated tests are a starting point, not a ship decision.

## Full transcript

[0:12](https://www.youtube.com/watch?v=c_SPW__jF9w&t=12s) Thank you for joining this session. Uh, it's a bit of an emotional one. Um, my name is Nathanael, and I'm a non-evaluator. Uh, it's been a lot of time since the last, uh, agent I've been shipping without any evaluation. And today, I want to tell you the story, how I quit this addiction of shipping an agent without any evaluation.

[0:39](https://www.youtube.com/watch?v=c_SPW__jF9w&t=39s) So I'm working at Air Canada, so compared to the previous talks, more like focused on specific business. Uh, I joined Air Canada in June, uh, of this year. And Air Canada is a huge company. We work in operation, corporate, commercial, and loyalty. Each of those segments have their own challenges. So my team is in, in charge of delivering, uh, AI solution for all of those. And the problem when we tackle s-- uh, a, a task or we create an

[1:09](https://www.youtube.com/watch?v=c_SPW__jF9w&t=69s) agent for a task that is so specific, we don't have any public benchmark we can rely on. And that's the main, main problem we-- I'm trying to address in this presentation. More importantly, it's not that we don't have a benchmark, is we are data scientists that can build agents or AI engineers that can build agents, but we are not domain expert. And those domain experts sits in different business units, and they are very busy in their own task,

[1:39](https://www.youtube.com/watch?v=c_SPW__jF9w&t=99s) and we have a limited access to that. So both of those things went together in a situation where it's very hard for us to build any evaluation or any, any, uh, benchmark for our task. So b-before going to Air Canada, I was working for a different company. Um, it was a company th-- And, and the task was to build a, um, a Q&A assistant for telecommunication engineer. So I don't know if you're familiar with the domain. Um, telecommunication is a huge

[2:09](https://www.youtube.com/watch?v=c_SPW__jF9w&t=129s) domain. There is about ten thousand documents, a lot of vendor documents on top of that. There's a logic to know which document you should, you should found in order to answer the question. So it's, it's a bit of a complex task. And in twenty twenty-four, everybody wanted to build RAGs, so we started to do that. Uh, on LinkedIn, it seems super easy. It's not. So, um, the thing is, when we tackle this project, the same problem as at Air Canada, we don't have

[2:39](https://www.youtube.com/watch?v=c_SPW__jF9w&t=159s) access to expert. We have limited access to experts. And we, we could not actually grade the answer ourself. So that's the, the problem I, I want to talk today. Funny enough, uh, we were in competition with other teams, um, two external, one internal. It's not like in parallel, but in sequence. And our solution actually worked. What mean worked is actually delivering the value. The people liked the solution, actually used it.

[3:09](https://www.youtube.com/watch?v=c_SPW__jF9w&t=189s) Uh, but to be honest, I don't know why we actually made it work compared to the other, the other team. So today, I'm just gonna show what I actually did or my team actually did to make it work. Um, so they, they all have the same document, same task. Everything is the same, but our solution somehow worked. So for those that are familiar with RAGs, um, the principle is very-- is, is to ask a question to the system. The system found the,

[3:39](https://www.youtube.com/watch?v=c_SPW__jF9w&t=219s) the relevant documents. Based on this relevant document, formulate an answer. It's something that exists for a long time now. And for those that actually tried it, if you found the right page or the right documents, ninety, ninety percent of the time, you have actually the right answer. So the challenge to find the right documents, at least in this task.

[4:00](https://www.youtube.com/watch?v=c_SPW__jF9w&t=240s) So we created a very complex pipeline that gets complexified over time. If you, uh, publication-- If you found publication back in the time, there's everybody has an amazing technique to improve the, the solution. We try them. It actually didn't improve the solution. So on paper, it worked for them, for the publication, but for us it didn't really help. And more importantly, when we add complexity, we found that, like, we break something else. So something that used to work now is not working anymore. And

[4:30](https://www.youtube.com/watch?v=c_SPW__jF9w&t=270s) that's actually the, the, the real problem. So we can add and more i-- we can get more ideas on-- into solving these, uh, different problems, but we, we keep continu-- we keep breaking things over and over. So complexity doesn't always help. And the other problem is this one. Uh, we can change an agent in few minutes, add a certain layer of, uh, query writing or whatever, reasoning step. But to grade that, we had

[5:00](https://www.youtube.com/watch?v=c_SPW__jF9w&t=300s) this crazy meeting once a week with experts. We put a gi-giant screen with five different chatbot, ask the same question to all of them, and ask the, the expert to grade them. "I prefer this one. I prefer this one." And funny enough, they're not always agreeing. It's very hard  when you, when you put expert-- two expert in the room, you get three opinions. So that's the, the, the problem we were facing. And so our budget was we can try one solution a week, and this was, uh,

[5:29](https://www.youtube.com/watch?v=c_SPW__jF9w&t=329s) not enough to go fast.

[5:33](https://www.youtube.com/watch?v=c_SPW__jF9w&t=333s) So the-- Before, uh, going further, so the, the, the, the process was the following. We ask, we ask a question, the agent answer. We ask the expert to read. Sometimes they don't know the answer. They have to look up. It takes a lot of time. Um, and we had the budget of one trial a week.

[5:51](https://www.youtube.com/watch?v=c_SPW__jF9w&t=351s) Um, so there is nothing new here. This binding that is well known is we reverse the problem. So we start with a page. We ask an LLM to write a question about this page, which is, uh, an easy task. We ask the agent to search for this answer and compare to what the agent found, compare to the, to the page that has been used for-- to generate the question. We can actually use code to grade that. And no longer opinions, no longer

[6:21](https://www.youtube.com/watch?v=c_SPW__jF9w&t=381s) anything. We can automate this task, and that allow us to go much, much faster. So for those that work in RAGs, there is plenty of implementation of this thing. I didn't invent it. I just found it very, very, uh, very smart, and it stick to me. Like why this principle cannot be applied to other tasks? So the impact of that is we went from one evaluation a week to dozens a day, and we start to see the,

[6:51](https://www.youtube.com/watch?v=c_SPW__jF9w&t=411s) the accuracy or the, the, the grading from the expert going up and up and actually removing a lot of complexity help. Like, we didn't need a lot of those. And focusing on the right, uh, embedding model, the right chunking, the right, uh, alpha in the BM25 are, is the right thing to do. And we, we saw the, we saw the, the accuracy increasing over, over weeks. So I don't know why we succeeded, but I'm pretty sure the other team didn't have any test set. So they were using this kind of long

[7:21](https://www.youtube.com/watch?v=c_SPW__jF9w&t=441s) loop with expert, maybe a much more expert than we had access to, but we had, uh, we had this test set.

[7:31](https://www.youtube.com/watch?v=c_SPW__jF9w&t=451s) And my learning from this, uh, back in 2024 is if you cannot grade your agent, uh, you cannot make it better. So that stick to me until this day, has been two years now, and I want to go back to Air Canada now, where we have a different project. So in 2026, the project is about, uh, analyzing, um, supplier contract. So you can imagine Air Canada has a lot of

[8:00](https://www.youtube.com/watch?v=c_SPW__jF9w&t=480s) suppliers. We get a lot of this contract. There's experts looking at them one by one, checking each of those clauses, and they have to analyze it and provide the feedback. Like, this clause is not compliant. This clause needs to be rewritten. Like, there is a lot of work to do that. It's a very long task and very costly because if we need the, if we need some supplier to deliver right now, and we don't, then it cause a lot of delay in the, in the chain. And if we miss a

[8:30](https://www.youtube.com/watch?v=c_SPW__jF9w&t=510s) clause, it's also very costly because at, at the end of the day can backfire to us. So it's a tricky problem. Um, and they came to me, uh, I mean to the, to the data science team, to build an agent that can actually do that for them. So can we automatize this task? And obviously, there is a lot of documents, a lot of rules, a lot of things to consider when you do that. Uh, experts are super busy. You can imagine the price of, of a lawyer

[9:00](https://www.youtube.com/watch?v=c_SPW__jF9w&t=540s) by hour, and there is no benchmark for that. We cannot go on the web and find documents, uh, uh, find an existing benchmark we can apply. So go back to the same problem as I was in 2024. Uh, cycle were super slow, and we knew that we want to be there in a s-- in a, in a, in a shape where we can go fast and try and try and try. Um, and I knew that if we, if we learn to this

[9:30](https://www.youtube.com/watch?v=c_SPW__jF9w&t=570s) situation, we can have a better trajectory in term of accuracy, and that was the challenge for this, uh, for this project. So the, the problem is creating the answer from documents is a well-known and defined things in the RAGs, but for this specific project, there is no guy in the world that say, "Hey, you should use this technique. You should use this technique." So we try to think how we can reapply the same mechanism from what I learned on RAG to this specific project, and we have

[10:00](https://www.youtube.com/watch?v=c_SPW__jF9w&t=600s) plenty of project at Air Canada. So we're using the same concept of getting the answer first and then try the agent to find it back. So going into the details, uh, it's a bit simplified, but actually the, the agent has four steps. Uh, we need to spot the clause in the contract. We need to find the relevant documents to support an amendment. Uh, we need to write the, the amendment. And then at the end, there is a classification system that tells you should we keep

[10:30](https://www.youtube.com/watch?v=c_SPW__jF9w&t=630s) it or not, uh, this amendment. So the first step, for example, spot is classifying all the section of the contract into specific domain, domains, and then find if there's a defect or there's a change to be, to be done in this section. So we did have this trick from the RAG, so we know how to find the document. So this is covered. Uh, but for the other, other section of the, of the agent, we didn't have anything.

[10:59](https://www.youtube.com/watch?v=c_SPW__jF9w&t=659s) So I'm gonna present five different techniques that we use in this project, or we try to use in this project that is trying to address this specific problem. My goal is not to show off. My goal is to get you inspired in how to reapply those techniques in your own, uh, challenges.

[11:20](https://www.youtube.com/watch?v=c_SPW__jF9w&t=680s) So the first one is to use past decision and answer. So we, we-- you can retrieve past contract, compare with the one that has been, uh, changed. The second one is to choose something you're looking for, write the contract around or generate the contract. The third one is, uh, take a good contract that has been signed and break it. Um, the fourth one is about things that should not change. No matter what you do, you should not get more flags. And

[11:50](https://www.youtube.com/watch?v=c_SPW__jF9w&t=710s) the fifth one is actually the most powerful. It's very hard, is to turn an example into many using the concept of templates.

[12:00](https://www.youtube.com/watch?v=c_SPW__jF9w&t=720s) So immediately, we jump into this approach. Uh, we ask the expert to provide us the original contract, the one that has been signed We're gonna take that, do a diff like you do in the code, compare what has been changed, and the agent should, should provide the same exact flags. That would be perfect. However, for many reasons, our, our team do not keep all the contract. Um, they don't keep the original one, they just keep the last one. They don't have all the versioning of the

[12:30](https://www.youtube.com/watch?v=c_SPW__jF9w&t=750s) contracts. It sometimes goes into negotiation. There is a lot of things. So we could not have any of those pairs of contracts, so this thing doesn't work.

[12:42](https://www.youtube.com/watch?v=c_SPW__jF9w&t=762s) Okay, second one, we took a rule. So that's changing a bit the topic. Uh, instead of creating, um, uh, flags or creating an agent that can provide flags, we start from rules. And the rule is, for example, we have the, the cap above one million dollars. So that's every contract at Air Canada should follow this rule or ma-many of those in specific domains. So you know the rule. You create, uh, a clause

[13:12](https://www.youtube.com/watch?v=c_SPW__jF9w&t=792s) without, uh, breaking this rule and generate the contract around, and you should find, um, the rule back-- the, the flag, which is, uh, breaking the rule. So that's simple. It doesn't require a lot of, uh, of, of work. A lot of things can be generated. The problem is you need to know the rule. So lucky enough, we could actually get some of those from either the expert or the documentation they had, uh, but cannot guarantee the coverage of everything. So we came up, we also with our own rules,

[13:43](https://www.youtube.com/watch?v=c_SPW__jF9w&t=823s) et cetera. So we generate at the end, uh, two hundred seventy-five tests that we know can run, and we know the answer. So similar to the RAG, we invested-- we invented the problem, we planted the problem, and we tried to find it back.

[13:59](https://www.youtube.com/watch?v=c_SPW__jF9w&t=839s) Another one, which is very hard actually, uh, we took an example of a contract that has been signed. We remove a clause, and the agent is supposed to say, "Hey, there's something missing here." Um, it's a very hard task because some supplier has different templates, have different kind of content. Knowing what should be in what contract require a b- a bit of thinking. Uh, and we mix those with existing contract that should not show any

[14:28](https://www.youtube.com/watch?v=c_SPW__jF9w&t=868s) flags. So that's the most powerful approach with that we have today. Um, using this approach, we can generate a lot of contract, and at the end of the day, we found that our agents are missing, uh, most of the clause that has been removed. So that's a very, very strong benchmark.

[14:46](https://www.youtube.com/watch?v=c_SPW__jF9w&t=886s) Okay, uh, fourth technique. Um,

[14:51](https://www.youtube.com/watch?v=c_SPW__jF9w&t=891s) you don't change-- You change something that should not affect the agent or shouldn't-- should not change anything in the result. For example, the name of the supplier, the address of the supplier, something that should not have any impact on the number of flags. Um, you can reorder the clause, so some randomness should also lead to the same results. Uh, and you can run the same, uh, contract again and again. All of those are things that we check to make sure that the, the output is

[15:21](https://www.youtube.com/watch?v=c_SPW__jF9w&t=921s) consistent over all of those. In this example too, everything can be checked using code. You don't need to do any LLM-as-a-judge. The previous speaker, I'm sure, like, it's a very hard task. So we stay away from, uh, LLM-as-a-judge as much as possible. The fifth one, um, it's the concept of template. Our expert have different kind of templates with blank. So I don't know if you ever write a contract. Those blanks should be filled

[15:51](https://www.youtube.com/watch?v=c_SPW__jF9w&t=951s) by the expert. And instead of trying to fill them, we generate, um, an array for each of those blanks to be filled with. So that allow us to generate plenty of contracts. Some tuples break some rules, some tuple doesn't. We know that. And so we know what actually should be flagged as a, as a, as a breach or not. Again, this is a code check. We don't need to have any LLM involved. Um, what, what-- The good

[16:21](https://www.youtube.com/watch?v=c_SPW__jF9w&t=981s) thing with this one is our expert already used templates, so we can reuse them to generate some contracts.

[16:29](https://www.youtube.com/watch?v=c_SPW__jF9w&t=989s) So the cycle was a bit slower in this case. So four weeks to get a, uh, a feedback, and we can get a script to run and get the, the evaluation back in five minutes. So I just want to highlight one point, and we went through all of those, um, kind of categories of grading at the, at the bottom. Exact match or code are very fast. As soon as we step into LLM-as-a-judge, uh, it's can be very fancy, it can be very useful. It's so hard. Like, we got

[16:59](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1019s) so much noise from the LLM-as-a-judge compared to the noise with the, with the agent that we're building. Solving this problem is actually almost as hard as solving the, the, the right problem. So finding the right prompt or the right harness for the LLM-as-a-judge is a very hard task. There is some padding you can do, task, uh, you can do, uh, fact-checking, you can do a bunch of stuff, but actually it requires so much work, and we could not find the right one for this project.

[17:29](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1049s) There is plenty of other techniques. I won't go over all of them. For example, trajectory. We have another project that require the agent to hit some endpoints. So trajectory, you can, um... You, you know what the endpoint should be in some order. You generate a story about it. You ask the LLM to fill, uh, to use the agent and look at the trace to, to see if the, the trajectory is correct. Text-to-SQL, you might already know it. Doesn't matter what your agent is, if you know how to pick the answer, if you

[17:59](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1079s) know what you're looking for, and I showed a couple of examples, um, generate the input and then try, try to find it. It doesn't apply to everything, and sometime we had to re-engineer our solution to fit this paradigm. Even if we move away slightly from the original design of the solution, we could, uh, with the original design, we could not evaluate it. We switch to something that we can actually evaluate because of that.

[18:27](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1107s) Um, one thing to highlight is not, uh, at the end of the day, you generate a test, you get a better agent, you get some real feedback, so expert can provide some feedback, and we don't use this feedback to fix the agent. We use this feedback to generate better test, better prompt, better ideas how to remove clauses, et cetera, et cetera, and one loop at a time. Uh, uh, that can be done by AI Rise. There is a lot of tool there.

[18:58](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1138s) Thanks for the invitation. Um, and that's my conclusion slide. Write the answer first. Um, I think I, I explained that. Um, when you generate those tests, it's not enough to know if you can ship the product. It's just a starting point. It's a tool to help and support the developer. It's not the go/no-go for the, for the solution. So I hope you like this, um, this presentation. I know

[19:28](https://www.youtube.com/watch?v=c_SPW__jF9w&t=1168s) that the twenty-four hour ship is the hardest one if you want to quit, uh, uh, releasing agent without evaluation. And, um, I would be happy to support you and have your own feedback. Thank you very much.
