# Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest

Jonathan Osborne, NatWest

[Talk page](https://ai.engineer/talks/5UgnatzpZ9E-building-ai-agent-is-easy-proving-its) · [Video](https://www.youtube.com/watch?v=5UgnatzpZ9E)

## How NatWest Tests Whether Cora Can Safely Answer a Customer

A mortgage question, 111 payment conversations and 14,000 complaint investigations show how NatWest decides which requests an agent can handle and how much confidence to place in the scores.

### A new answer at the scale of 100,000 conversations a day

[0:12](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=12s)

Jonathan Osborne, a lead AI engineer at NatWest Group, could build an agent that answered banking questions. The harder engineering problem was deciding what evidence would justify letting it answer customers. NatWest, the UK’s second-largest retail bank, had roughly 19 million retail customers. Its digital assistant, Cora, was their primary point of contact and handled around 100,000 conversations a day. At that volume, even an infrequent mistake could affect someone making a decision about real money.

Most of Cora’s answers were scripted. If the team had anticipated a question, it could write and test the path to an answer. A customer whose question fell outside those paths might have to rephrase it or queue for a human. NatWest was working to bring generated answers and agents to approximately a third of Cora’s journeys. Retrieval-augmented generation (RAG) would draw on the bank’s knowledge bases. Agents could also take actions on a customer’s behalf, such as helping make a payment or change an address.

That combination widened what Cora could do and the consequences of getting it wrong. A plausible answer could cost a customer money, leave the bank answering difficult questions from its regulator and damage its reputation. To seek approval for the customer-facing service, the engineers first had to establish which of their own measurements they could trust.

### One mortgage question exposes four ways an agent can fail

[2:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=134s)

“Can I overpay my mortgage without being charged?” Scripted Cora recognizes the word “mortgage” and offers choices such as applying, checking rates or going to the customer’s mortgage. The customer can select “none of the above,” but the system has made little progress. Asking the customer to phrase the question differently is another turn spent trying to reach the answer. Queuing for a bank employee is the likely outcome.

In Osborne’s agent example, Cora retrieves guidance and says the customer can overpay up to 10% of their balance in a year without an early repayment charge. It offers to check their deal. That is a much more direct response, but the order matters. The customer has already heard a specific allowance before Cora has established that it applies to their mortgage. Then the customer asks whether they can close the account instead. A change of subject is ordinary customer behavior, even if it breaks the sequence the engineer had in mind.

![Two illustrative chats start with the same mortgage overpayment question. Watson offers a menu and the customer rejects the choices. The generated answer mentions a usual 10% allowance, then the customer asks about closing the account.](https://ai.engineer/api/talk-essay-media/7edccf30a915aba0c384354c4e415e7f77973f3572f0086c7e2df6711841ea3f)

We are changing what happens next

Osborne identified four failure modes. First, an agent can sound certain about a fabricated answer and even cite a fictitious source. Second, the input can become adversarial through prompt injection, with someone trying to talk the agent into doing something it should resist. Third, an answer can be accurate and within policy while the conversation goes in circles. A customer who still cannot get their question answered experiences a failure that a factuality check could miss.

Fourth, the same input can produce a different output on the next run. The Cora team had accumulated years of regression tests for a decision tree whose possible paths it could enumerate. An assertion about the response expected at a particular point worked for that system. With an agent choosing a course of action and generating its reply, passing that assertion on one run offers limited assurance about the next.

### Cora filters known events before it starts a model

[4:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=284s)

Every incoming conversation enters a routing funnel. The first gate is a contextual filter for events such as button clicks. If a customer has just selected a known option, the event itself conveys their choice. This filter handles about 20% of incoming traffic. The second gate applies deterministic preclassification rules to the customer’s utterance and accompanying metadata. When those rules identify a destination, Cora can route the request directly. Up through this point, routing requires no model inference.

The remaining utterances can enter a gated relevance and rephrasing step, useful for detailed questions that depend on several pieces of context. Next comes the intent classifier, a neural network over embeddings that predicts which backend should handle the utterance. This is a traditional machine-learning component. A final set of policy rules checks whether this particular customer should have access to the predicted backend. A classifier can correctly identify what someone wants to do while the access decision still has to be made.

![Horizontal bars narrow from 100% of customer turns to 80% at the contextual filter and about 45% at each of four routing stages. A side panel explains how the rules support access control, auditability, resilience, and governance.](https://ai.engineer/api/talk-essay-media/8c8524855918fbaca2a7ad67605ee099f548656a2a00b68846ccb08fd4b645c3)

The routing funnel

Four operational requirements shaped the funnel. The policy gate enforces access control. For auditability, every routing decision carries a code so the bank can reproduce and account for the route it took. For resilience, the scripted service gives Cora a way to answer when a model is unavailable. For governance, the team keeps routing rules in versioned configuration. It can change a rule in response to changing conditions without redeploying the whole application.

### Routing accuracy leaves three questions about the answer

[6:30](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=390s)

Suppose the intent router is 94% accurate. That tells an engineer how often it picked the right destination. It says little about whether the eventual answer was grounded in a real source, whether the customer’s problem was resolved or whether Cora recognized when to bring in a human. Osborne contrasted a router that reports 100% confidence even on the 6% of requests it misroutes with one whose 85% confidence is well calibrated. The latter gives Cora a signal it can use when deciding whether to proceed or hand off.

NatWest evaluates each answer across three categories. Safety asks whether the response could harm the customer. Task completion asks whether it accomplished the job. Experience asks what it felt like to deal with Cora. Each can have its own monitoring and alerts. An invented mortgage term merits investigation into the source and the answer. Repeatedly returning accurate guidance to a customer who is trying to do something else calls for a different investigation. An overall quality score could send both incidents to an engineer looking for the same defect.

### Payment reviews uncover a 54-point gap on changes of subject

[7:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=464s)

Before automating the scoring, NatWest ran a human validation exercise in June. Three people independently reviewed each of 111 conversations covering 37 payment journeys across four brands. The payment agent passed 83% overall. Split the conversations by what the customer did, however, and the result changes substantially. Happy-path scenarios passed 97% of the time. Edge cases passed 76%. When the customer changed topic, the pass rate fell to 43%.

The gap between following the expected path and changing the subject was 54 percentage points. The customer who asks about closing an account partway through discussing a mortgage illustrates the kind of digression at issue. Testing only whether an agent could carry a cooperative customer through a payment would have missed much of the difficulty. Human review also helped locate faults outside the model. The two biggest causes of failure in the exercise turned out to be engineering issues.

Independent scoring exposed another variable in the evaluation: how consistently people could apply each criterion. Reviewers agreed 95% of the time on bias and fairness. They agreed only 64% of the time on intent understanding, the judgment about whether the agent understood what the customer wanted. That complicates the instruction to make an automated judge match a human. On intent, choosing a different human as the reference could change whether the very same judgment counted as a match.

![The June baseline shows pass rates of 97% for happy paths, 76% for edge cases, and 43% for topic changes. A separate panel shows pairwise scorer agreement ranging from 95% on bias and fairness to 64% on intent understanding.](https://ai.engineer/api/talk-essay-media/e0f459768a9eed220277f9198e095cd2286ce6c38580d2d5d22f5d5834dcaf9d)

We had humans score it first

### A judge averaging 0.97 could barely distinguish good cases from bad

[9:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=554s)

Cora Plus had yet to go live, so Osborne drew the production evidence for automated judging from a separate system. NatWest’s internal complaints agent, live since February, receives a complaint, uses tools to investigate and writes a summary for a complaint handler. The summary should help the handler understand why the customer complained and how the bank can help. Around 14,000 cases had scores from both a human validator and an LLM judge, allowing the team to compare the assessments case by case.

On faithfulness, the judge gave a mean score of 0.97, with scores packed near the top of the scale. Humans averaged 0.83 and used a much wider range. Osborne reported an average error against the humans of about 0.2, a number that might look tolerable in isolation. The correlation was almost zero. When handlers marked some summaries down and gave others high scores, the judge scarcely distinguished between them. An engineer using its score to find summaries worth investigating would have little to go on.

![Two bars compare faithfulness scores for the same outputs. The LLM judge’s narrow red bar sits above the 0.89 pass mark. The wider green bar for human scores crosses it. The reported correlation is 0.04.](https://ai.engineer/api/talk-essay-media/7eca23df7c29210b0831eefa6523e93433421a583069bcdae12c89787e48b1cb)

A broken judge looks like a good judge

A model change exposed a second problem. The team kept the input and the agent’s output fixed and changed only the model doing the judging. In Osborne’s threshold comparison, the reported margin above the cutoff fell from 0.8 to 0.2. The work being assessed was identical, yet the bank’s apparent headroom had narrowed. The judge version therefore belongs in the configuration alongside the threshold. NatWest also moved the judge to a different model family from the agent being evaluated. Osborne said the different blind spots helped bring judge and human scores closer together.

### Trace each conversation, score samples every four hours and gate prompt changes

[11:05](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=665s)

Cora’s evaluation pipeline captures activity from the agent and its guardrails using OpenTelemetry and OpenInference. It takes a 5% random sample, then also selects conversations that hand off to a human, fail, run particularly long or show signs of an angry customer. The two selection methods serve different purposes. The random sample includes ordinary interactions. The targeted selection gives engineers more chances to examine behaviors that would appear infrequently in that sample. The LLM judge scores the selected cases across the quality dimensions, and traces, scores and experiments go into Comet Opik.

![A four-stage flow follows a conversation through sampling and LLM scoring into Comet Opik. Panels below describe private connections, PII masking, and the timing of traces, offline scoring and regression checks.](https://ai.engineer/api/talk-essay-media/f515bdf44a21599c4e2dca62b6842b6217574a51c416224f5afc3b8b565203f8)

How the scoring runs

The bank runs in an air-gapped environment without internet access. Evaluation reaches the judge over a private path through an internal GenAI gateway. NatWest consumes Comet, a SaaS product, as a signed partner app through AWS. The bank also has zero tolerance for personally identifiable information leaving its network, so much of the evaluation takes place with that information absent.

Trace capture, scoring and release decisions operate on different clocks. Cora records traces as events happen. Offline scoring runs every four hours. A change to a prompt or the context supplied to the agent triggers a comparison against a known-good baseline in CI/CD. A hard regression gate holds the merge until the team understands the effect. This matters even for a small text edit because its effect on the agent’s responses can be disproportionately large.

### Trace spans locate agent failures while feature drift warns of classifier decay

[12:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=764s)

In a trace from the payment agent, each LangGraph node appears as its own span. If the conversation is slow or breaks at a particular step, the engineer can open that part of the trace and investigate what happened there. The conversation-level score identifies an interaction worth inspecting. The spans give the investigator a place to start inside the agent.

The intent classifier needs another instrument. Customers’ language evolves after training, so the distribution of inputs can shift even while the classifier stays fixed. NatWest monitors input feature drift alongside F1 calculated from human labels, with thresholds for triggering retraining. Osborne found feature drift particularly valuable because a shift in the inputs could show up before a decline in F1. The team uses Comet for both kinds of work, with Opik for generative AI and model performance monitoring for traditional ML. Engineers can observe both parts of a journey while measuring each according to how it fails.

![On the left, an Opik trace and node diagram show a LangGraph payment agent spending 61.6 seconds in its outbound guardrail. On the right, a multicolored area chart shows classifier feature drift rising and falling during 21–28 Sep.](https://ai.engineer/api/talk-essay-media/5beabeaaee72c0ac1b415294672c88591b204ec93ca95dcd388cb85c1428753d)

Two models, two failure modes

### Shadow routing gives risk reviewers production evidence before customers get generated replies

[14:07](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=847s)

Model-risk colleagues wanted to know how Cora would handle actual customer language before approving a customer-facing release. Yet the team needed access to production requests to gather that evidence. Shadow deployment separated two decisions that would otherwise have had to happen together. The candidate classified live utterances while existing scripted Cora supplied the replies customers actually saw. Reviewers could inspect the real range of queries and where the classifier would send them. Whether a generated reply would resolve a customer’s problem remained a question for the next stage.

After shadow routing, the team planned to roll out customer-facing replies brand by brand, gradually increasing the percentage of traffic exposed within each brand. At each stage, performance indicators would help determine whether to expand further. Some decisions could be automated and others required manual approval. Feature flags gave operators three levels of intervention. They could isolate a single backend for one brand, isolate that backend globally or use the global flag to send every request back to scripted Cora. Each change could take effect within seconds.

![Four boxes show a gated rollout from shadow classification, while Watson answers, to customer-facing routing at successively larger brands. Three panels list the requirements to advance, live alerts and rollback options.](https://ai.engineer/api/talk-essay-media/034514d2368d3a3f832bf5adc081ec7763d6010b945c3b420737c417a41b1f0c)

Shadow first, then brand by brand

### A required reason for reruns turns complaint handling into evaluation data

[15:43](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=943s)

The complaints team collected feedback at a point where a colleague had a specific judgment to offer. A handler who wanted the agent to rerun an investigation first had to explain why the original attempt needed another pass. Capturing that reason before the rerun helped build the set of roughly 14,000 colleague labels. Osborne said other projects struggled to collect even a tenth as many. Those labels made it possible to discover how generously the production judge was grading the work.

The two projects exchanged what they learned. Cora incorporated judge calibration and regression checks after the complaints team found the scoring problem. The complaints team, in turn, planned to adopt Cora’s Opik-based production monitoring. Each agent addressed a different banking task, but the machinery for recording what it did, comparing assessments and detecting regressions could be reused. At the time Osborne spoke, the complaints agent had supplied the live evidence. Cora Plus was due to face its own customer rollout the following month.

## Key takeaways

- Route button clicks and recognizable requests deterministically. For requests that need a classifier, check the customer’s permission after it predicts a backend.
- Report results by scenario and by scoring criterion. NatWest’s 83% overall payment pass rate included a 43% pass rate on topic changes, and reviewers agreed much less often on intent than on fairness.
- Calibrate a judge on paired human and model scores for individual cases. Check the spread, the correlation and the margin above the threshold whenever the judge model changes.
- Use shadow classification to inspect routing on real requests, then expand customer exposure by brand and percentage with a tested route back to the scripted service.

## Slides

### Agents for 19 million customers

[0:39](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=39s) [Two large figures show 19 million UK customers and 100,000 daily Cora interactions. Three cards describe the move from scripts to agents, the risk of wrong answers, and the evidence needed to go live.](https://assets.ai.engineer/sites/aiecode2025/sha256/583668aba3c1837076cdad938df1e36913008585702e36c14466a19289d9dc1d.webp)

WHY NOW
NatWest Group
Agents for 19 million customers

19 million
UK customers

100,000
Cora interactions a day

NatWest is the UK's second-largest retail bank. Cora is our customers' primary point of contact.
Until now, every answer was scripted. Next month we go live in shadow, then customers follow.

THE SHIFT
Scripts to agents
A scripted journey only answers what someone thought to script. We are moving about a third of Cora to AI-generated answers and an agent that can act.

THE HARD PART
Regulated, at scale
A wrong answer to 19 million people is not a bug ticket. It is a customer harmed, and a regulator asking why.

THE TALK
Earning permission
This is the evidence we had to build to be allowed to go live, and what it took to trust our own measurements.

Building the agent was not the hard part. It was proving it was safe.
2

### We are changing what happens next

[2:22](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=142s) [Two illustrative chats start with the same mortgage overpayment question. Watson offers a menu and the customer rejects the choices. The generated answer mentions a usual 10% allowance, then the customer asks about closing the account.](https://assets.ai.engineer/sites/aiecode2025/sha256/7edccf30a915aba0c384354c4e415e7f77973f3572f0086c7e2df6711841ea3f.webp)

THE STAKES
NatWest Group
We are changing what happens next
Same question. Two very different conversations.

BEFORE - WATSON, THE SCRIPTED ASSISTANT
Can I overpay my mortgage without being charged?
I can help with mortgages. Which do you need? Apply | Rates | My mortgage
None of those

NOW - A GENERATED ANSWER
Can I overpay my mortgage without being charged?
You can usually overpay up to 10% of your balance a year with no early repayment charge. Shall I check your deal?
Actually, can I close the account instead?

Illustrative example, not a real customer conversation

The moment it can answer anything, it can be wrong about anything
3

### Accuracy cannot tell you if it is safe

[3:38](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=218s) [Four numbered cards describe confident errors, prompt injection, unhelpful answers, and varying outputs for the same input. A green callout proposes shrinking the failure surface before testing it.](https://assets.ai.engineer/sites/aiecode2025/sha256/d63d3914d3093abb759f136bb1cf0237bfc44e535e27d87715ea5d61418549d8.webp)

THE PROBLEM
NatWest Group
Accuracy cannot tell you if it is safe
It reasons, decides and generates. Four failure modes a scripted bot cannot have.

1  Confidently wrong
Hallucinates against a source, and sounds certain doing it

2  Talked into things
Prompt injection. The input itself is adversarial

3  Right but unhelpful
Accurate, on-policy, and still loops without resolving

4  Non-deterministic
Same input, different output. No assertion can regression-test it

OUR ANSWER
Shrink the surface before you harden it. Do not try to test an infinite surface - make it smaller first.

A scripted system's failure surface is bounded. An agent's is not.
4

### The routing funnel

[4:59](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=299s) [Horizontal bars narrow from 100% of customer turns to 80% at the contextual filter and about 45% at each of four routing stages. A side panel explains how the rules support access control, auditability, resilience, and governance.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c8524855918fbaca2a7ad67605ee099f548656a2a00b68846ccb08fd4b645c3.webp)

APPENDIX
NatWest Group
The routing funnel
Deterministic rules bracket the model. Fewer than half of all turns ever reach it.

All customer turns  100%
Contextual filter (button presses)  80%
Stage 1  Pre-classification rules (no ML)  ~45%
Stage 2  Relevancy and rephrasing (LLM, gated)  ~45%
Stage 3  ML classifier (an MLP, not an LLM)  ~45%
Stage 4  Post-classification rules  ~45%

WHY
Access control
Enforced before any model runs
Auditability
Every decision carries a code
Resilience
Rules survive an ML outage
Governance
Versioned config, no redeploy

We did not point a language model at 19 million customers
A1

### Quality is a vector, not a score

[6:32](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=392s) [A crossed-out “accuracy 94%” sits beside questions about grounding, resolution, and human handoff. Three columns list the safety, task, and experience metrics scored on every generated answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa880caf65674145ee7ecc9fba9b669e21ebe370499d9c047a99261fd3e2909a.webp)

DEFINING QUALITY
NatWest Group
Quality is a vector, not a score
We could not answer “is it working?” with one number

accuracy 94%
Was the answer grounded in a real source? | Did the customer actually get resolved?
Did it know when to stop and get a human?

WHAT WE SCORE ON EVERY GENERATED ANSWER

SAFETY
Could it harm the customer?
groundedness
bias

TASK
Did it actually do the job?
task_completion
kb_hit_rate
resolution_focus
conversation_completeness

EXPERIENCE
How did it feel to deal with?
tov_adherence
knowledge_retention
user_satisfaction
frustration_signal_count

A router that is 94% accurate and 100% confident when wrong is more dangerous than one that is 85% accurate and knows it
5

### We had humans score it first

[7:42](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=462s) [The June baseline shows pass rates of 97% for happy paths, 76% for edge cases, and 43% for topic changes. A separate panel shows pairwise scorer agreement ranging from 95% on bias and fairness to 64% on intent understanding.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0f459768a9eed220277f9198e095cd2286ce6c38580d2d5d22f5d5834dcaf9d.webp)

WHAT WE LEARNED
NatWest Group
We had humans score it first
June baseline: 111 conversations, 37 journeys, four brands, three independent scorers

ONE PASS RATE, THREE DIFFERENT STORIES
97%  happy path
76%  edge cases
43%  topic changes

DID OUR HUMANS AGREE WITH EACH OTHER?
Bias and fairness - 95%
Overall pass / fail verdict - 78%
Final response quality - 66%
Intent understanding - 64%
Pairwise agreement, three independent scorers

A single 83% pass rate hid a 54-point spread
Objective dimensions are easy. Subjective ones are not.

If your humans agree 64% of the time, “match the human” is not a target

These are the June baseline. The two biggest failure causes were fixed as a result - measuring is what found them.
6

### A broken judge looks like a good judge

[9:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=554s) [Two bars compare faithfulness scores for the same outputs. The LLM judge’s narrow red bar sits above the 0.89 pass mark. The wider green bar for human scores crosses it. The reported correlation is 0.04.](https://assets.ai.engineer/sites/aiecode2025/sha256/7eca23df7c29210b0831eefa6523e93433421a583069bcdae12c89787e48b1cb.webp)

WHAT WE LEARNED NEXT
NatWest Group

A broken judge looks like a good judge
Cora+ is not live yet, so this is our complaints project - live since February

FAITHFULNESS SCORE, SAME OUTPUTS SCORED BY BOTH
LLM JUDGE  mean ~0.97, spread ~0.04
HUMAN  mean ~0.83, spread ~0.26
0.00     0.25     0.50     0.75     1.00
pass 0.89

WHY THE HEADLINE LIED
Error vs human  ~0.2
looked close
Correlation  0.04
was not close at all
A judge pinned near the top cannot discriminate, so correlation collapses whatever the human does.

Same outputs, same 0.89 pass mark: 0.08 headroom under one judge, 0.02 under another

Your judge model is part of your threshold configuration. Choose a judge from a different model family.
7

### How the scoring runs

[11:05](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=665s) [A four-stage flow follows a conversation through sampling and LLM scoring into Comet Opik. Panels below describe private connections, PII masking, and the timing of traces, offline scoring and regression checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/f515bdf44a21599c4e2dca62b6842b6217574a51c416224f5afc3b8b565203f8.webp)

THE MACHINERY
NatWest Group

How the scoring runs
Sampled, scored and landed in one platform. Running pre-production now, on live traffic from launch.

Live conversation
agent + guardrails
→ Sampled
5% + all handoffs, fails, long, angry
→ Scored
LLM-as-judge on each dimension
→ Comet Opik
traces + scores + experiments

NO INTERNET FROM THE PODS
The judge model is reached over a private path through an internal gateway. Comet, a SaaS product, is reached through a signed partner-app channel with no API keys in the application.

PII NEVER LEAVES
Masked before anything exits our network. The governed system of record is our own warehouse; the vendor UI is for deep dives, not the dashboard of record.

Live traces  as it happens
Offline scoring  every 4 hours
Regression gate  every merge request

Continuous evaluation is production infrastructure, not a notebook you run once
8

### Two models, two failure modes

[12:42](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=762s) [On the left, an Opik trace and node diagram show a LangGraph payment agent spending 61.6 seconds in its outbound guardrail. On the right, a multicolored area chart shows classifier feature drift rising and falling during 21–28 Sep.](https://assets.ai.engineer/sites/aiecode2025/sha256/5beabeaaee72c0ac1b415294672c88591b204ec93ca95dcd388cb85c1428753d.webp)

ONE PLATFORM
NatWest Group

Two models, two failure modes
A classifier decays. An agent regresses. Both on one platform.

GENAI - COMET OPIK
The agent regresses when someone edits a prompt.
Every run traced and judged.
opik

LangGraph  66.6s
start_node  1.6s
route_from_start  0.001s
orchestrator  1.3s
GenAiLLM  1.2s
route_from_orchestrator  0.001s
extraction_node  0.9s
GenAiLLM  0.9s
route_from_extraction  0.001s
routing_node  0.002s
response_node  1s
GenAiLLM  1s
outbound_guardrail  61.6s

start_node
extraction_node
orchestrator
END     routing_node
END
END     response_node
outbound_guardrail
__end__

Real trace: LangGraph payment agent, 12 spans, guardrail is the costly step

CLASSIC ML - COMET MPM
The intent classifier decays as language drifts. Feature drift, weekly F1, gates at 0.87 / 0.85.

DATA DRIFT BY FEATURE - Earth Mover's Distance, 21-28 Sep
Drift climbing away from the training distribution is the signal to retrain.
This is the decay, watched live.
9

### Shadow first, then brand by brand

[14:04](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=844s) [Four boxes show a gated rollout from shadow classification, while Watson answers, to customer-facing routing at successively larger brands. Three panels list the requirements to advance, live alerts and rollback options.](https://assets.ai.engineer/sites/aiecode2025/sha256/034514d2368d3a3f832bf5adc081ec7763d6010b945c3b420737c417a41b1f0c.webp)

GOING LIVE
NatWest Group

Shadow first, then brand by brand
One launch, split into two much smaller decisions.

SHADOW
Classify all real traffic.
Watson answers.
GATE
SMALLEST BRAND
First customer-facing traffic, lowest exposure.
GATE
LARGER
Percentage ramp within brand.
GATE
LARGEST
Full customer-facing routing.

GATE TO ADVANCE A STEP
Error rate below 1%
p95 latency below 500ms
Zero circuit-breaker trips

ALERTS WHILE LIVE
groundedness below 0.90
handoff rate above baseline
drift or confidence shift

ROLL BACK, THREE LEVELS
Global - all to Watson
Per backend - RAG or agent
Per brand - isolated

Advance on a gate, watch continuously, roll back in seconds. Zero customer risk in shadow.
10

### The expensive part was never the agent

[15:41](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=941s) [Two panels describe a complaints-summary agent and an exchange between teams. Handlers must score an investigation before rerunning it. The complaints team identified a judge problem and is adopting the other team’s Opik evaluation layer.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf8b946394236d718b458b0d88354052db9cb09aa4fcc2832ecc5ad15d8a7647.webp)

APPENDIX
NatWest Group

The expensive part was never the agent
Same eval spine, pointed at other problems

THE COMPLAINTS PROJECT
A colleague-facing agent. Writes an investigation summary for the handler.
17 complaint categories, fraud over half the volume.
Every investigation scored on coherence, completeness, faithfulness, relevancy, bias.
Handler feedback is a GATE, not a thumbs-up. You cannot re-run an investigation until you have scored the last one.

A TWO-WAY STREET
The lesson came to us
They found the judge problem in production. Our judge-calibration check exists because of it.

The architecture went to them
We built the eval layer on Opik. They are adopting it for production monitoring.

The agent was the cheap part. The eval spine is the part you get to reuse.
A2

### What I would tell you

[16:46](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=1006s) [Six numbered rows pair lessons with observations about traffic reaching a model, quality dimensions, judge score spread, threshold settings, quality gates and reuse of the bank’s evaluation system.](https://assets.ai.engineer/sites/aiecode2025/sha256/4d0b10724fed5de635df44190fd454ee672b3f59cdcdcf6231c8515c93aa0f90.webp)

TAKEAWAYS
NatWest Group

What I would tell you
If you are shipping agents where a bad answer lands on a real person

1  Shrink the surface before you harden it.
More than half our traffic never reaches a model.

2  One score cannot tell you if an agent is safe.
Split quality into dimensions you can alarm on.

3  A judge pinned near the top is a broken instrument.
Check its spread against human spread, not its average.

4  Your judge model is part of your threshold config.
Swapping it silently moves your quality gate.

5  Ship quality gates soft, then hard.
A gate people do not trust gets disabled.

6  The expensive part was never the agent.
It was the eval spine - now reused across the bank.

One platform for both halves of that spine. Next month we start finding out how much of it was right.
11

## Full transcript

[0:12](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=12s) Hi guys. So my name's Jonathan Osborne. I am a lead AI engineer from NatWest Group. I'll tell you more about what that is in a second. Uh, and today I'm gonna be talking about taking agents from prototype to production, customer-facing that is, in a regulated banking environment. And I guess maybe some spoilers. With the access to coding assistance and tools that we have today, the actual creation of the agent, that's kind of the easy part. The much harder part, I think, is proving that it's safe. So let me tell you a

[0:42](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=42s) bit about Nafwe-- NatWest Group. So NatWest is the second-largest retail bank in the UK, and currently we have somewhere in the range of nineteen million retail customers. Nineteen million, hold onto that number because it's really what makes everything else in this talk matter. Cora is our customer's primary point of contact, so it's our digital assistant, and it serves somewhere in the realm of a hundred thousand conversations a day. And until now, the vast, vast bulk of those answers were scripted. So if a customer comes to

[1:12](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=72s) us and they ask a conversation, then if we haven't thought to script that, then, uh, you know, we're not gonna be able to answer. Customer's gonna get frustrated. They're probably gonna have to queue to speak to a human agent. That's changing. So a-as of this year, we are now working to bring approximately a third of the journeys within Cora to AI-generated answers backed by RAG and our knowledge bases and agents. Agents that are able to take action on the behalf of customers. So for example, helping them make a payment or change their address.

[1:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=104s) I guess the, the scary part here really isn't the model. Like by and large, models are quite good. They provide good answers now. Uh, it's the blast radius. So with nineteen million customers, uh, if we get a few of these utterances wrong, uh, that isn't a bug ticket. That isn't kind of a maybe a funny headline. It's a real customer, and they're real money. It's likely very difficult questions from the regulator, and it's reputational damage for us as a company. So this talk is about what we had to go through to get the approval to go live. A-and

[2:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=134s) I guess what it took us to maybe trust some of our own metrics as well because as I say, the interesting part is not the agent, it's proving and making it safe. Let's maybe make that more concrete by considering an example here. So kind of same question, two very different conversations. So customer comes, they ask, "Can I overpay my mortgage without being charged?" Today, in the scripted solution, that's gonna recognize that keyword mortgage, and we're gonna present the customer with a menu. So apply rates, my mortgage. Customer

[2:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=164s) is gonna have to select none of the above. Maybe we'll ask them to rephrase their question, but more than likely, game over. This customer is gonna have to queue, speak to a human agent. They're gonna be frustrated. On the right-hand side, bit of a different story. So AI has been able to answer that question using our knowledge bases. So you can overpay up to ten percent of your balance in a year and no early repayment charges. Shall I check your deal? But then the customer does something which is very normal but is can be frustrating, so they change the topic. So actually, can I close the account instead?

[3:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=194s) This happens all the time, and a traditional decision tree is really, really bad at this type of digression. It's worth mentioning that digressions are gonna come back, so we'll hold on to that point. I guess the key point though is if we build an agent that can answer any question, it can also answer any question wrong. So that ten percent, if that doesn't apply to this particular customer's deal, it's a real customer, it's their real mortgage, and it's their real money at stake. So why is it so much harder to make an agentic-backed so-solution, you know,

[3:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=224s) more, uh, you know, how do we protect against it as opposed to that scripted bot? Well, it's because it reasons, it decides, it generates. And there are at least kind of four failure mode here, which we wouldn't have in that scripted solution. One, it can be confidently wrong. Hallucination against a fictitious source and sound absolutely sure of itself while it does it. Two, it can be talked into things. Prompt injection. The input itself can become adversarial. Three, right but unhelpful. So it can provide accurate on-policy answers that just go

[4:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=254s) round and around and around and never actually answer the customer's conversation. Incredibly frustrating. And four, it can be non-deterministic. Same inputs, completely different output. That means that your traditional software engineering assertions just don't really work very well here. And as a consequence of that, regression tests that have been built up over years by the existing Cora team, they really no longer apply. A scripted system's failure surface is completely bounded. We can write out every p- situation that might occur. We can test it, and we can be

[4:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=284s) really confident it's gonna do what we want it to. With an agent, that's really not possible, and we can't really traditionally test our way to safety. So how do we deal with this? Well, the first part for us was shrinking the attack surface before we actually then thought about hardening it. Introducing what we call the routing funnel. So every customer conversation comes in at the top here. First, we apply a contextual filter, and that allows us to remove kind of common events like button clicks. That's about twenty percent. Next, we have some

[5:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=314s) deterministic preclassification rules. So with these rules, we're kind of using the customer's utterance and some metadata to route them to a destination with no ML or AI involved at all at this point. It's only utterances that make it past these first two gates that we then actually start the models on. First, there's a relevancy and rephrasing step. That's kind of gated. It helps with those really detailed multi-context questions. And then our intent classifier. This is actually a neural network over embeddings. This isn't genitive AI, which is able to use a customer's utterance to predict the right

[5:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=344s) backend. Finally, there's some policy rules to ensure that this customer should actually be getting into that particular backend. Why'd we build it this way? Well, four reasons, a lot of them about being a bank. Firstly, access control, so we need to enforce that to ensure the customer should be getting to this ac-- this, this backend. Auditability, so every decision here needs to carry a code. It needs to be fully auditable and reproducible. The regulator ensures this. Resilience. In the case that a model or the gen AI isn't available, we still need to provide

[6:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=374s) the customer an answer. And I guess governance. So all of this is version config. We can make changes to it without a full redeployment. That's really important because it means that we can react fast to changing market conditions. We didn't just take a language model, slap some context into it, and point it at ninety million customers. We first thought really hard about what actually needed a model to solve. This brings us on to sort of the really important question of is this actually working? And the key metric that they looked at in kind of traditional Cora was accuracy when doing this. So if I told you this

[6:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=404s) intent writer is ninety-four percent accurate, that probably sounds really good, but does that actually answer the important question? So is it grounded in a real source? Did the customer's question actually get resolved? Did it know when it was out of its depth, when to stop, bring in a human? You can't embody that with one single score, and that's why we've moved away from quality of the score, started treating it more as a vector. Uh, every score-- every kind of answer we score across three different categories. So safety, could it harm the customer? Task, did it actually do its

[7:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=434s) job? Experience, how did it feel to deal with? All of these can be monitored and alarmed independently, and they allow us to differentiate between is it hallucinating versus is it right but just insufferable to deal with. There are different answers, and they require really different solutions. It's where-- The key point here is a router that's ninety-four percent accurate but a hundred percent confident the six percent of the time it's wrong is way more dangerous than one that's eighty-five percent confident and knows it. The first thing we

[7:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=464s) did here before we worried about automating this is what we always do, and we started off with some human validation. Uh, the numbers that you see here are from June validation that we did on our payment agent. Hundred and eleven conversations over thirty-seven different payment journeys, four brands, three independent human scorers per conversation. Headline here, eighty-three percent pass rate across the board. But as ever, the headline's pretty useless. Uh, let's dig into some of that data. So ninety-seven percent at-- on the happy path scenarios. That's great. Seventy-six percent edge cases.

[8:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=494s) Hmm, not quite so good. Topic changes, like that mortgage example that we discussed earlier, only forty-three percent. So that single eighty-three percent score was hiding that fifty-four percent spread. The second point on the right is probably more nuanced and less talked about, and that's do human validators actually agree with each other? On something objective like bias and fairness, you do actually see a ninety-five percent agreement rate, which is great. But on something more subjective, intent understanding, so, you know, are-- does the bot actually understand what

[8:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=524s) the, what the user wants? Only a sixty-four percent agreement rate. The common thing when we start automating this, bringing LLM as a judge, is that we want to make our judge match our human validators. But what are they meant to be matching here? There's only a sixty-four agreement re- agreement rate between humans, so that can't simply be the target here, essentially. One, one caveat, by the way, like, the two biggest causes of failure here actually turned out to be engineering issues, not a problem with the model at all. But nevertheless, that proves useful value of the human validation because it caught

[9:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=554s) them. So let's turn to actually automating this then, so LLM as a judge. So Cora Plus, that's not live yet, so I'm gonna turn to another internal project. So this is an internal complaints agent that we have. Essentially, the goal here is that customers will occasionally complain. This will take the complaint. It has access to a load of tools to then be able to write a summary for complaint handlers to essentially explain why has this particular user complained and how we can help them with it. It went live in February, and we've got around fourteen thousand of

[9:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=584s) these cases which have been scored by both a human validator and an LLM as a judge. If we dive into the data, so this is the faithfulness score here on the right. So you can see upper bar in the red there. That's the LLM as a judge. Mean point nine seven, right up at the top, very little spread. Human, uh, mean point eight three, much, much wider spread. Key point here is that if you look at that average error versus the human, it's point two. You might conclude that the judge is doing okay, but the correlation is almost zero.

[10:15](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=615s) Uh, in, in this case, like, a judge which doesn't discriminate between the actually good cases and the bad cases is basically worthless. You don't want your judge to be generous, and that is what LLMs want to do by default. Maybe one other point to make here is that, so that amber dotted line, that represents the threshold for this particular metric. For this judge model, you can see we're comfortably above that. However, we made a change to that model. Didn't change anything else. Same input, same output. Just changed the actual model behind it. And that dropped all the way from point eight

[10:45](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=645s) above to point two above. That means that the judge model itself actually becomes part of your configuration, and it can lead to breaking changes. A, a key point we took here, by the way, is that we moved our judge to a different model family from that of the, uh, you know, the actual agent. So judge and judged, different blind spots, and that's had to be really effective in pinning the two scores closer together. Coming back to the core example. So how does this scoring actually run? So live conversations between the agents and its guardrails, all of those are captured

[11:15](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=675s) via open source standards like O- OTEL and OpenInference. All of that detail is then sampled. So by default, we take five percent random sample, but we also want to look at everything interesting. By interesting, I mean conversations that hand off, fail, those are particularly long, one for the user of angry. We then score these with our LLM as a judge. So that's that raft and dimension of metrics that I talked about earlier. And then absolutely all of this ends up in Comet Obiq. So traces, scores, experiments. It's governed by a couple of bank constraints here. So we work,

[11:45](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=705s) work in a completely air-gapped environment, so no internet access to anything. So judge model is reached over a private path. We have an internal GenAI gateway. Comet, a SaaS product. We're consuming that as a signed partner app via AWS. Second around personally identifiable information. We have a zero-tolerance policy for any PII leaving our network. So that means a lot of this evaluation happens without that present. Different cadences here. So traces are being captured as they happen. We're doing offline scoring every four hours. And then regress-- This is also built

[12:15](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=735s) into our CI/CD. So if we make changes to prompts or to context, that can have, like, an outsized impact on what the agent is actually responding. So we need to C-c-constantly be comparing versus a known good baseline. We do that in CICD. They have a hard regression gate, so changes can't be merged in without us understanding exactly what's going to occur. I guess key point is that continuous evaluation is part of your production infrastructure. It's not a notebook that you run a couple of times when your model goes live.

[12:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=764s) So there's different models here, and they've got different failure modes. So on the right-hand side, we've got the agents, we've got the generative AI. So generative AI regresses. So as we've just talked about, somebody can make changes to a prompt, and that can have outsized impact on the behavior. That's why we have the CICD in place. You can see here a real OpenTrace, which is looked from our payment agent. Every node in the Lang graph, uh, becomes a different span. And when the agent is slow or is breaking in a particular place, this allows us or an agent to jump in and investigate exactly why.

[13:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=794s) On the classical s-- so the intent classifier is traditional ML, and thus we've got to monitor it in a more classical way. It drifts over time, right? So human conversation will naturally evolve from the point at which the model was trained, so we need to look at more traditional things like input feature drift, F1 score based off human scores... Sorry, human labels. And then, you know, we have RAG thresholds defined as to exactly when we need to trigger retraining for that model. On the right, you can see an NPM data drift chart. I, I really wouldn't underestimate, like, input feature drift here. It's a really good

[13:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=824s) canary as to when your model is starting to de-decline before your F1 score does, implying the need for retraining. So two different problems, two different instruments. But what we don't want is two different observability and evaluation frameworks. Comet's a really useful layer because it brings in both Open for the GenAI and NPM model performance monitor for the ML. So one platform. How do we actually deploy this then without kind of breaking the bank and creating a load of headlines? We start off with a shadow deployment. So what I mean by that is we put this

[14:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=854s) live, uh, we use it to start classifying customer utterances, but the existing scripted solution is what will actually respond to the customer. And that means that we can get a sense of the real date, you know, the, the input, uh, the input space for different customer queries with almost zero risk. I think this shadow mode is like one of the more transferable bits of this project for us because often we have this deadlock in a regulated industry where obviously we can't go live and get production data without the approval to do so. But equally, to

[14:44](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=884s) actually enable us to go live in many cases, our model risk colleagues increasingly want to see how our models are going to react to that actual production data. This allows us to kind of break that apart so we can break apart, observe safely from actually answer customer queries. Uh, and it takes one really big decision point, a big bang release, into a series of smaller ones, which we can make confidently. We then go brand by brand, so we have like percentage-based rollout within the brand, and we have kind of KPIs which allow us to determine go, no-go

[15:14](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=914s) at various points along that. Some of those decisions are automated, and some of those require manual approval. Um, I think the... And yeah, some of those require manual approval. What kind of makes that possible is that we u- make a lot of use of feature flagging here. So there's like a global flag that we can trigger to send everything back to the scripted solution. But also every single back end can be isolated either globally or on a particular dimension, such as brand. And all of that can be done in seconds. Before I wrap up, let me

[15:43](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=943s) maybe just talk very briefly and pull this back to that complaints project. So just to remind you, that was the colleague-facing agent which writes investigation summaries for, summaries for complaint handlers. In that project, human feedback is a gate. It's not a thumbs up. It's not a tick box activity. So if a complaint handler wants to rerun their investigation, they can't do so until they provided human feedback. We want to understand why they felt the need to do that rerun. But also it's allowed us to build up this quality data set of fourteen-- fourteen thousand customer of, uh,

[16:13](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=973s) of colleague labels, I guess, in this case. And that's really valuable. In a lot of projects, we struggle to get a tenth of that number. And I think the projects have also traded nicely here. So, you know, they found that judge problem in production. We have judge calibration and regression checks because of that. But equally, the architecture that we built, which relies heavily on Comet Open, they don't use that yet, and they're gonna be adopting that for their production monitoring. I guess it's a glimpse inside an enterprise, right? So the, the agent with the cheap part here, that's code that we can produce to solve the business problem. The valuation

[16:43](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=1003s) spine is the bit that we can then reuse again and again. So to wrap up, like, if you are going to be shipping customer-facing agents in a regulated environment, then these are probably the points I would give you. I'm not gonna go through everything or maybe just pick a couple. So three and four here, they're the, they're the points with the actual production judge data, right? So that judge pinned near the top is a broken instrument. Your judges are gonna want to be overly positive. You need to fight against that. Compare the spans, not just the averages.

[17:13](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=1033s) Ensure the judge is actually matching the human reviewer. Fourth, the judge model itself does become part of that threshold config, right? So if you are going to be making changes to the judge model, then, uh, you know, you can introduce a breaking change without knowing it. Uh, Comet is obviously absolutely key here. So we have that for both halves, both the generative AI and the traditional ML, which we do still use a little bit. So, and I guess the, the final point here is next month I'll find out how much of this is actually true when we go live. So thanks

[17:43](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=1063s) a lot for listening. It was really a pleasure to get to speak to all of you. So I'll be at the Comet booth after this. If you do have any questions, you know, disagreements, et cetera, really happy to hear them. But thanks a lot for your time.
