# Your Financial Agent Needs to Show Its Work - Anirudh Veeraragavan, Fieldguide

Anirudh Veeraragavan, Fieldguide

[Talk page](https://ai.engineer/talks/gdzsJZJ6DDs-your-financial-agent-needs-show-its-work) · [Video](https://www.youtube.com/watch?v=gdzsJZJ6DDs)

## Why Was This Evidence Sufficient? Building a Financial Agent That Can Show Its Work

Fieldguide’s Anirudh Veeraragavan follows a bank reconciliation from draft findings to inspection, showing what an agent must preserve so another auditor can examine a decision years later.

### Build the reconciliation around the opinion a reviewer will sign

[0:12](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=12s)

In audit, the opinion is the product, says Anirudh Veeraragavan, lead security engineer at Fieldguide. The value of that opinion depends on the basis for it. Consider a bank reconciliation. A client reports a cash balance on its books, and the bank reports a balance for the same client. Those numbers can legitimately differ. A deposit in transit can appear on the books before it reaches the bank. An outstanding check can reduce the balance on the books before it clears. The auditor tests whether those explanations account for the difference.

Veeraragavan sketches a reasonable first version of an agent for this high-volume work. Give it the client’s books, the list of reconciling items and the supporting evidence. Supply the audit methodology that tells it how to match the items. Prior-year audits can serve as worked examples, and previously reviewed audits can supply expected answers for an evaluation. The agent drafts its findings and presents them to a human reviewer for sign-off. In the scenario he asks the audience to imagine, the metrics look good, users are happy and the team ships the agent to general availability.

![Four boxes connected by arrows show the audit workflow. The agent matches the client’s records and bank evidence, previously reviewed audits supply the evaluation, and a human reviewer signs off.](https://ai.engineer/api/talk-essay-media/6749d460825c9450e8edf64dad870196b17b1504ae36b8f2ab39c9159ad6547b)

Four-Step Process

### An inspection narrows the question to one piece of evidence

[1:53](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=113s)

Then an audit is selected for inspection. Someone picks a particular test procedure, looks at one piece of evidence and asks, “Why was this sufficient?” The evaluation score describes performance across cases. The inspector wants the basis for the conclusion in this case. Even a satisfied reviewer who signed the finding leaves that question open for the next person who has to examine the work.

Veeraragavan cites Peregrine Financial Group, an Iowa brokerage, to put the stakes in perspective. For more than twenty years, the firm committed bank account fraud. On paper it claimed more than $200 million. At the bank there was only $5 million. His concern here is the distance between a reported financial position and the cash that someone could actually substantiate.

Audit practice gives him an acceptance test for the agent. An experienced auditor coming to the engagement fresh should be able to understand the tests performed, the evidence collected and the decision reached. To see what that requires, he introduces Calder Freight Lines, a fictional client closing its books for 2026. Its bank shows about $2.9 million, its books show about $2.7 million, and its reconciliation lists 41 items to explain the gap. He follows four of those items, two checks and two deposits.

### Authorization depends on whether this row has been reviewed

[4:27](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=267s)

A reviewer flags one of the checks as a possible exception and signs the row. A few days later, additional information explains the discrepancy and triggers another agent run. The agent has permission to edit the finding. It reads the new evidence correctly and changes the row to “no exception.” An inspector opening the row now sees a cleared check bearing the signature of the person who flagged it.

![An Authorization timeline shows a preparer flagging OC-14 and a reviewer signing it. After the agent runs again, OC-14 reads “No exception.” The signature remains.](https://ai.engineer/api/talk-essay-media/558563f87051250f3d25e30567afd2986a69ad6474e0c31a058d07ca859cadb3)

The Signature Outlived the Answer

The error occurred at the write. The new finding is sound, but the reviewer’s signature now appears to endorse a conclusion they never saw. Checking whether the agent may call the editing tool gives too coarse an answer. Authorization has to take account of who is making the call, which object it targets and the state of that object at the moment of the write.

For an unreviewed row, let the agent draft a finding under its own identity. For a signed row, give its new conclusion the status of a suggestion. That lets the reviewer consider the additional evidence while preserving what the earlier sign-off covered. The arrival of better evidence is a good reason to reopen a question. It gives the agent grounds to propose a change, while the decision about revising signed work stays with a person.

### Execution control makes a disagreement between runs inspectable

[5:37](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=337s)

Suppose the procedure runs in 2027 and returns no exception. A year or two later, during a review, someone reruns it and gets a different finding. Both runs have logs. Each says the agent fetched evidence, processed it and returned an answer. Read side by side, those logs describe the same sequence. They reveal little about where the second run departed from the first. Running the entire procedure again would give the auditor a third result to explain.

Veeraragavan would record the run as a chain of steps. Each step has an ID, inputs and outputs. The record distinguishes operations performed by deterministic code, such as fetching an item or applying a matching rule, from operations entrusted to the model, such as extracting a field from a PDF or arriving at the final conclusion. An auditor can then ask a narrower question at each boundary. Did the runs fetch the same item? What value did each extraction return? What reached the matching step?

Once the engineer locates a difference, they can replay that individual operation with its recorded inputs and compare its output with the earlier result. In Veeraragavan’s example, the original run was correct and the newer one was wrong. That direction matters. The inspection raised a legitimate question about signed work, and the record gives the reviewer a way to defend the original finding by examining how the agent reached it.

![Three lines of code rerun the extraction for OC-22 on the same pinned page and link the new record to the original. A callout shows the new model reading 69,400 while the page shows 96,400.](https://ai.engineer/api/talk-essay-media/d322a036e18fc4cc7a93c6571e29b81cd652e11a2fcae07f6192e68174111d94)

Rerun One Step, Not the Run

### Evidence provenance ties page two to a particular bank statement

[7:31](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=451s)

Another finding clears an item against the wrong account. Its citation points to page two of a January bank statement. The document can be real and the cited material can be on that page. A large engagement may contain several files called January bank statement. Even if the filename is unique, an auditor still needs to determine which version the model read and who supplied it. Checking that a file exists would miss the account error.

![Two panels show the account mismatch for DIT-02. The agent cited a $61,900 Norvale Foods entry in payroll account -4488. The operating account under test, -4471, had no such January credit.](https://ai.engineer/api/talk-essay-media/eedb0f43d78edfa6c41ee83963d61b34d82c138e519b15c11762d268a3086c31)

Right Bank, Right Month, Wrong Account

Give the citation a specific document ID, a version and a hash of the contents. Keep the page reference so the reviewer can inspect the passage in that exact document and check which account it describes. Record where the document came from as well. Identifying the contents and establishing their source answer two different audit questions.

Provenance also gives the agent a basis for enforcing evidence requirements. A procedure may require a statement obtained directly from the bank. A client-uploaded copy could have been altered. With a source record, the system can flag that copy for review even when the model finds the expected amount on the expected page.

### Evaluation integrity requires reopening some of the passes

[8:44](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=524s)

The first version of the agent treated last year’s reviewed audits as gold data. Veeraragavan gives three reasons to inspect those reference answers. A later decision may have superseded the finding selected for the dataset. A human may have relied on out-of-band evidence that the agent never received. Or the audit methodology may have changed, making a historically sound finding the wrong expected answer for the current procedure. Each calls for a different inquiry into the decision, the available evidence or the applicable rule.

![Two panels show how last year’s reviewed audits can penalize a correct agent when a decision was superseded or undocumented, and reward a wrong agent when the methodology has changed.](https://ai.engineer/api/talk-essay-media/cf2e595fc4109e53ac1f3c2abc922f6024eb9ef015325155b77435fcada3edad)

Audit the Gold, Not Just the Agent

His illustrative evaluation reports 97%. Start with the nine apparent misses. In four of them, the evaluation itself had the wrong answer and penalized the agent. A team that investigated only disagreements could reasonably come away thinking its agent performed better than the headline score suggested.

Now inspect the agreements. In twelve cases, the evaluation expected no exception and the agent received credit for clearing the item. Changes to business rules or audit methodology made those clearances wrong. These failures would never enter a queue populated only by failed tests. The 97% combines credit and penalties assigned for different reasons. To interpret it, the team has to examine what justified the labels on both sides of the pass line.

### Escalate the deposit that took eleven business days to clear

[10:09](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=609s)

Several events already encountered in the reconciliation give the agent a concrete reason to stop. It needs to revise signed work. A fresh run disagrees with the earlier finding. A document fails the procedure’s evidence requirements. In each case the handoff should identify the point that needs judgment. For the deposit Veeraragavan singles out, the relevant fact is that it took eleven business days to clear. Calling attention to that timing discrepancy tells the reviewer what to investigate before deciding whether to accept the explanation.

![A numbered list gives five reasons to ask for human review. Below it, an agent asks a preparer to investigate DIT-03, which cleared 11 business days out and came from an affiliate.](https://ai.engineer/api/talk-essay-media/1523f9bd0ef2c26f84fe7ae780da28fbc0c41f40d1da341f3f28695ce225fa59)

Stop and Ask a Human When…

Veeraragavan wants escalations to be easy to verify, even when approval takes time. Frequent prompts encourage automation bias as reviewers get used to clicking yes. He uses an illustrative 80/20 split: if the agent can handle 80% of the work, spend the reviewer’s attention on the 20% that warrants intervention. The time a person spends investigating an exceptional deposit is part of the design.

### The same four controls apply to a loan decision or an AML alert

[11:23](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=683s)

Each control covers a failure the others leave open. Authorization preserves the relationship between a signature and the finding reviewed. Execution control lets someone reconstruct a correct result and investigate a conflicting one. Evidence provenance lets that person trace a mistaken clearance back to the account and document the agent used. Evaluation integrity determines what the team can infer from its 97% score. A reconciliation can pass any three checks and still fail the fourth.

![A two-column table pairs each skipped control with an audit failure, from a reviewer signing a different answer to a late deposit receiving a passing evaluation.](https://ai.engineer/api/talk-essay-media/0f42bac72236bb9b80dfb846577dab53b470444893f09a4e23d4cc2bf51f2b44)

Defensibility Is an AND

In credit, the decision to examine might be an approval or a decline. In anti-money-laundering work, it might be the closure or escalation of an alert. The object under review changes, but an experienced person arriving later still needs to recover the evidence, follow the operations that produced the finding and see who had authority to act. That is what lets the institution stand behind a consequential decision after the people who made it have moved on.

## Key takeaways

- Check the row’s review state when the agent writes. Route new evidence about a signed finding into a proposed revision.
- Record each step’s ID, inputs and outputs, and bind cited passages to a document ID, version, content hash and source.
- Sample apparent passes when auditing an eval. Agreement with an obsolete reference answer can conceal a real failure.
- Give reviewers the discrepancy they need to investigate, such as an eleven-business-day clearing interval, when asking them to decide.

## Slides

### Testing a Bank Reconciliation

[0:39](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=39s) [A ≠ sign separates the cash balance in the books from the bank’s cash balance. A box below lists deposits in transit and outstanding checks as explanations.](https://assets.ai.engineer/sites/aiecode2025/sha256/d3ad4339c6e4da2d20788a6d0f45e61e07634e930db13cca290850fb14c2bd1e.webp)

LET’S BUILD A FINANCIAL AGENT · ONE AUDIT PROCEDURE
Testing a Bank Reconciliation

BOOKS
Cash balance
≠
BANK
Cash balance

DIFFERENCES EXPLAINED BY
Deposits in transit · Outstanding checks

### Four-Step Process

[1:16](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=76s) [Four boxes connected by arrows show the audit workflow. The agent matches the client’s records and bank evidence, previously reviewed audits supply the evaluation, and a human reviewer signs off.](https://assets.ai.engineer/sites/aiecode2025/sha256/6749d460825c9450e8edf64dad870196b17b1504ae36b8f2ab39c9159ad6547b.webp)

LET’S BUILD A FINANCIAL AGENT · A TYPICAL BUILD
Four-Step Process

INPUTS
Client’s books
Bank reconciliation
Bank evidence
→
AGENT
Does the matching
Methodology in
prompt
Prior-year audits
as examples
→
EVAL
Previously
reviewed audits
→
HUMAN
Reviewer signs off

### Everyone Wins

[1:55](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=115s) [Three green check marks accompany 97% for evals, green metrics, and happy users. Below them, an inspection question asks why that was sufficient.](https://assets.ai.engineer/sites/aiecode2025/sha256/e4518163609b488d9677c1db591a5be8ff63f441edae64238b41e477505b3096.webp)

LET’S BUILD A FINANCIAL AGENT · SHIP TO GA
Everyone Wins

Evals
97% ✓

Metrics
Green ✓

Users
Happy ✓

Selected for inspection: why was that sufficient?

### Peregrine Financial Group · July 2012

[2:21](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=141s) [The slide compares Peregrine Financial Group’s $223.8M on paper with about $5M at the bank. The bank figure appears in red above a line about nearly 20 years of forged bank statements.](https://assets.ai.engineer/sites/aiecode2025/sha256/d728f5bbbd3c0f9a0648327afe80a282d585a6605d239c8b2b40ce942ec13203.webp)

FRAUD IS REAL · PEREGRINE FINANCIAL GROUP · JULY 2012
On paper: $223.8M
At the bank: ~$5M

Forged bank statements, for nearly 20 years.

### An Experienced Auditor, New to the Work, Can Follow the Agent

[3:29](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=209s) [Four numbered boxes joined by AND list Authorization, Execution Control, Evidence Provenance, and Evaluation Integrity. A dashed box beneath all four describes where the agent stops and a human decides.](https://assets.ai.engineer/sites/aiecode2025/sha256/0436f967fcb7b3b5294e856352efea943d8fff155ace016d42c1adb5cb9039cc.webp)

PATTERN FOR ASSURANCE · INSPIRED BY AUDIT
An Experienced Auditor, New to the Work, Can Follow the Agent

1
Authorization
limit what it can do
AND
2
Execution
Control
preserve
what it did
AND
3
Evidence
Provenance
bind claims to
correct sources
AND
4
Evaluation
Integrity
trust the
benchmark

Where the agent stops and a human decides

### Bank Reconciliation · 12/31/2026

[3:57](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=237s) [For fictional Calder Freight Lines, the left table shows deposits and outstanding checks bringing the adjusted bank balance to 2,763,330, matching the client’s balance. The right table lists four of the 41 reconciling items.](https://assets.ai.engineer/sites/aiecode2025/sha256/cac8389fe7c75631322987b58f75e652018142f247cd3671210d1d81f07c72ea.webp)

PATTERN FOR ASSURANCE · CALDER FREIGHT LINES (FICTIONAL)
Bank Reconciliation · 12/31/2026

Balance per bank    2,914,600
+ Deposits in transit    580,150
− Outstanding checks    (731,420)
Adjusted bank balance    2,763,330
Balance per client    2,763,330 ✓

FOUR OF 41 RECONCILING ITEMS
OC-14 · check · Brightline Fuel    62,500
OC-22 · check · Rook Logistics    96,400
DIT-02 · 12/31 · Norvale Foods    61,900
DIT-03 · 12/31 · Lakeshore Retail    480,000

### The Signature Outlived the Answer

[4:26](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=266s) [An Authorization timeline shows a preparer flagging OC-14 and a reviewer signing it. After the agent runs again, OC-14 reads “No exception.” The signature remains.](https://assets.ai.engineer/sites/aiecode2025/sha256/558563f87051250f3d25e30567afd2986a69ad6474e0c31a058d07ca859cadb3.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
The Signature Outlived the Answer

01/21  Preparer flags OC-14:  exception
01/22  Reviewer signs
01/26  Statements land → trigger runs the agent on all 41
       OC-14: “No exception.”  Still signed.

### Same Call, Different State

[5:11](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=311s) [A green check permits the agent to draft unreviewed rows as itself. Red crosses mark writing directly to a reviewed row and replacing a human’s flag. For a reviewed row, the slide says to propose the change.](https://assets.ai.engineer/sites/aiecode2025/sha256/eb3919510cc5bb4c22b876df2cd04f4189126ca38a59dd98f42038ef27de8773.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
Same Call, Different State

✓ Draft unreviewed rows, as itself
× Write to a reviewed row: propose instead
× Replace a human’s flag

### “We Had Logs.”

[6:08](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=368s) [OC-22 changes from $96,400 with no exception in 2027 to $69,400 with an exception in 2028. Side-by-side panels compare the available log with the evidence and decisions needed to trace the result.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ac667e8c80be483e367e9d68d34d9cd56f46f5d71c517292805853c2d0d41ba.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
“We Had Logs.”

OC-22 · 2027 run: $96,400 no exception · 2028 rerun: $69,400 exception

WHAT WE HAD: A LOG
fetch_evidence(“January statement”)
Final text
A “reasoning” paragraph

WHAT WE NEEDED: A RECORD
Claim
Evidence @hash + page + values read
Rules applied
Run · model version · config
Human decision

### A Run Is a Chain of Recorded Steps

[6:18](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=378s) [Four boxes linked by arrows show code fetching a statement, a model extracting 96,400.00, code matching the amount, and a model concluding that the item cleared on 01/08. Each step has saved inputs and output.](https://assets.ai.engineer/sites/aiecode2025/sha256/213e1166616bd000c2d754304f2f112a85559e26de453dd229703d1cbd82fb15.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
A Run Is a Chain of Recorded Steps

FETCH
code
STMT-JAN-4471
p.3 · …9c1e
→
EXTRACT
model
#20977 · 01/08
96,400.00
→
MATCH
code
96,400 =
96,400 ✓
→
CONCLUDE
model
“Cleared 01/08. No
exception.”

step ID = run · item · step  →  inputs + output, saved

### Rerun One Step, Not the Run

[6:44](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=404s) [Three lines of code rerun the extraction for OC-22 on the same pinned page and link the new record to the original. A callout shows the new model reading 69,400 while the page shows 96,400.](https://assets.ai.engineer/sites/aiecode2025/sha256/d322a036e18fc4cc7a93c6571e29b81cd652e11a2fcae07f6192e68174111d94.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
Rerun One Step, Not the Run

orig = steps.get(run="R-0031", item="OC-22", step="extract")
new  = extract(orig.input.page_sha, model=CURRENT)  # same pinned input
steps.put(new, parent=orig.id)                      # new record, linked

Today’s model: 69,400 · Pinned page: 96,400 → the original stands

### Right Bank, Right Month, Wrong Account

[7:31](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=451s) [Two panels show the account mismatch for DIT-02. The agent cited a $61,900 Norvale Foods entry in payroll account -4488. The operating account under test, -4471, had no such January credit.](https://assets.ai.engineer/sites/aiecode2025/sha256/eedb0f43d78edfa6c41ee83963d61b34d82c138e519b15c11762d268a3086c31.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
Right Bank, Right Month, Wrong Account

DIT-02 · WHAT THE AGENT CITED
“January bank statement, p.2”
$61,900 · Norvale Foods · 01/05
Account -4488 (payroll)

THE ACCOUNT UNDER TEST
Account -4471 (operating)
January: no $61,900 credit

### A Citation Is an Identity, Not a String

[8:12](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=492s) [Evidence Provenance is highlighted above a code block that looks up a claim’s source in the run’s retrieval record, checks its account, and checks that the claim covers every required rule.](https://assets.ai.engineer/sites/aiecode2025/sha256/365a07c419452d6db67a58551f10f879f0454d9a276538c80f95c9ed69150e4d.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
A Citation Is an Identity, Not a String

src = run.retrieval_record[claim.source_ref]  # id + version + hash, not model text
assert src.account == item.account            # check what the claim depends on
assert required_rules(item) <= claim.rules    # and every rule it had to apply

### Audit the Gold, Not Just the Agent

[8:43](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=523s) [Two panels show how last year’s reviewed audits can penalize a correct agent when a decision was superseded or undocumented, and reward a wrong agent when the methodology has changed.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf2e595fc4109e53ac1f3c2abc922f6024eb9ef015325155b77435fcada3edad.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
Audit the Gold, Not Just the Agent
Gold = last year’s reviewed audits

AUDIT THE MISSES
Gold that punishes a right agent
Superseded: final call in a review note
Inquiry-only: no document shows it

AUDIT THE HITS
Gold that rewards a wrong agent
Outdated: changes in methodology

### The Score Hid It

[9:43](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=583s) [Three cards show a 97% overall score marked “SHIPS,” 4 of 9 misses attributed to bad gold, and 0 of 12 late-clearing cases scored “correct.”](https://assets.ai.engineer/sites/aiecode2025/sha256/1eed470889779fc8365f0768f8bbe82d67e2c5b386b53d7551f824556b6c3b2c.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
The Score Hid It

V1 · OVERALL
97%
✓ SHIPS

THE 9 MISSES
4
were bad gold, not the agent

LATE-CLEARING SLICE
0 / 12
scored “correct”

### Stop and Ask a Human When…

[10:08](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=608s) [A numbered list gives five reasons to ask for human review. Below it, an agent asks a preparer to investigate DIT-03, which cleared 11 business days out and came from an affiliate.](https://assets.ai.engineer/sites/aiecode2025/sha256/1523f9bd0ef2c26f84fe7ae780da28fbc0c41f40d1da341f3f28695ce225fa59.webp)

Authorization    Execution Control    Evidence Provenance    Evaluation Integrity    Human Decides
PATTERN FOR ASSURANCE
Stop and Ask a Human When…

① It would change work a human signed
② A rerun disagrees with the record
③ The evidence doesn’t match the item
④ A required rule fails
⑤ Sources disagree

AGENT → PREPARER
“DIT-03: cleared 01/19, 11 business days out, from an affiliate. Investigate before
concluding.”

### Defensibility Is an AND

[11:21](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=681s) [A two-column table pairs each skipped control with an audit failure, from a reviewer signing a different answer to a late deposit receiving a passing evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/0f42bac72236bb9b80dfb846577dab53b470444893f09a4e23d4cc2bf51f2b44.webp)

PATTERN FOR ASSURANCE · ALL OR NOTHING
Defensibility Is an AND

SKIP…    …THEN
Authorization    OC-14: signed by a reviewer who saw a different answer
Execution Control    OC-22: right answer, but you can’t show what ran
Evidence Provenance    DIT-02: cites the wrong account
Evaluation Integrity    DIT-03: a late deposit the 97% eval scored “correct”

### Same Four Questions, Different Nouns

[12:14](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=734s) [Three cards ask what supports a loan decline, who closed an AML alert and on what evidence, and which price source and time were used to mark a fund position.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c9eade29823645f1402e778644ffd33a44e52318ae7b7c22c48f981a75ac7d6.webp)

PATTERN FOR ASSURANCE · BEYOND AUDIT
Same Four Questions, Different Nouns

CREDIT
Loan declined: which
bureau file, which model
version, which credit
policy?

AML
Alert closed: on which
transactions, by whom,
with what evidence?

FUND VALUATION
Position marked: from
which price source, as of
what time?

## Full transcript

[0:12](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=12s) Uh, my name is Anirudh, and I'm lead security engineer at Fieldguide, where we're defining what agentic audit means. So, in audit, the opinion is the product. An opinion is only as good as the basis behind it. So it's not so much whether you're correct or wrong, it's really about the decision-making process behind what you're putting forward. And so this raises the question: What does an agentic decision-making process look like? So to kind of dive into this, let's build a

[0:42](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=42s) financial agent. So we'll take a financial audit use case, and we'll take this specific audit procedure, where we're testing a bank rec-re-reconciliation. Uh, to any auditors in the room, please bear with me on the simplification. But bank reconciliation is where you take the client's book and the cash balance that they have on the, those books, and you compare it to the cash balance that their bank has. And any differences should be explained by things like deposits in transit, outstanding checks. Bas-basically, you wanna make sure that whatever the client

[1:12](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=72s) is saying is their financial reality actually matches the truth. So how would you kind of build this out if you wanted to build an agent for this? Because this can be kinda tedious, high-volume work. So what you would do is you would take your client's book, you would take the list of reconciling items, you would take evidence, you would give it to your agent, which has some audit methodology for doing the, the matching. You can even have some prior year audits as examples. You can even build an eval of previously reviewed audits, and then you kind of surface everything to a human reviewer to finally sign off.

[1:42](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=102s) This is totally reasonable, great, great approach. So, you know, we build this out, we have great evals, great metrics, users are happy, everyone wins. We ship this to GA, and the story ends there. Until your audit is selected for inspection, and someone asks you, "Why was this sufficient?" That is, I don't care what your evals say, I don't care what your metrics say, I don't even care what your users say. For this specific audit, for this specific test procedure, for this specific

[2:12](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=132s) piece of evidence, how did you arrive at this decision? And you may be wondering, this seems like a little overkill. You know, this is maybe not gonna happen that much in practice. But the thing is, financial fraud is very much real. Peregrine Financial Group is, or I guess  I should say was, a brokerage firm that was, uh, based in Iowa, and for over twenty years they committed bank account fraud. On paper, they claimed they had over two hundred million dollars, and in reality, at the bank, there was only five million dollars. And when this

[2:42](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=162s) fraud came to light, two hundred million dollars basically just vanished. And I'm not here to litigate who or what or why this happened, but what this shows us is that in the financial industry, the consequences of being wrong are quite severe. So we need to have confidence behind the decisions that we make, and we need to be able to stand behind them in case things do go awry in the future. So in terms of how to do this, I think the audit industry has, like, a nice definition we can take some inspiration from. Uh, they say that an

[3:12](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=192s) experienced auditor who hasn't done the work before should be able to understand the tests you performed, the evidence you collected, the decision that was reached. In other words, an experienced auditor new to the work can follow it. And we can adopt this to the agent world, where we say an experienced auditor new to the work can follow the agent. And in order to follow the agent, you need these different controls in place around authorization, execution control, evidence provenance, and evaluation integrity. And

[3:41](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=221s) ultimately, all of this rests upon knowing when the agent should stop and the human should decide instead. So this is a pattern for decision assurance, and I'll kind of walk through this financial audit example again, but it really applies to many more domains than just this one. So let's kinda go back to our audit. You know, we're going back to bank reconciliation. We have this fictional client called Air Freight Lines, and we're closing the books for twenty twenty-six. So as per their bank, they have around two point nine million dollars. On their books,

[4:11](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=251s) they have two point seven million dollars, and the difference is explained by forty-one reconciling items. So we'll take these forward, two checks, two deposits, and let's kind of see what our agent does, and let's walk through these different controls to ensure we're making defensible decision-making. So the first one is around authorization. So you can imagine that a human reviewer is kind of going through these reconciling items. They flag one of these checks, says, "Hey, I think there may be something wrong here." And a few days later, we get some additional context, we get some more additional information

[4:41](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=281s) around why that is not an exception, which triggers the agent run, and then it rewrites that to no exception. Now, the thing h-here is that the human already-- a human already reviewed it, and they already signed it, but what they reviewed, the content behind it has now changed. The agent didn't do anything wrong. It had the permission, it had the evidence, and it arrived at the right conclusion. But in this specific instance, it shouldn't have overridden anything because now what the reviewer has claimed to review doesn't match what they actually looked

[5:12](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=312s) at. So what we're seeing here is that authorization isn't just about, is this tool call allowed? It's, is this tool call allowed for this object, given this state at this point in time? The agent should be allowed to draft unreviewed rows as itself, but it cannot overwrite what a human reviewed. It cannot replace what a human did. It has to kinda exist as its own suggestion as opposed to its own authoritative truth. The next control is around execution control. Uh, so you can imagine, you know, in twenty

[5:41](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=341s) twenty-seven, we kind of go through this exercise, we have no exception, and then maybe a year or two later, we're selected for review. We rerun it, and we get a completely different result. So now the question arises, like, what happened? How did this-- How could this have happened? So we have logs. We know that the agent fetched evidence, it did some reasoning, it outputted, uh, its final answer. But that's not enough to reconstruct what happened in twenty twenty-seven against what happened in twenty twenty-eight and identify the gap. So really what we need is a record. We need a

[6:11](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=371s) record of these were the steps that happened, this is the culmination of the run, and you can interrogate each point along the way. So this run is a chain of recorded steps where each step has a ID that kinda lets you understand what is being done here, what are the inputs, what are the outputs, and you can clearly dif- delineate between deterministic steps where you're using code to fetch an item or to do a match versus where you're using the agent to extract, you know, a field from a

[6:40](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=400s) PDF or to arrive at the final conclusion. And given this chain of recorded steps, you can then rerun individual steps as opposed to the whole thing and seeing what happens. And this gives you much stronger assurance around being able to contr- uh, being able to compare runs against different points in time, because now we can kinda compare a new run against our old run, identify the exact step where something went wrong. And in this example, we're seeing that our original run was actually correct, even though the

[7:10](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=430s) run of this newer run is wrong. But again, it's not so much about whether you're correct or not, it's about being able to demonstrate with a high level of conviction that you know you are right. You didn't just get lucky that the model picked the right answer. You can clearly show that you had these steps, and when you rerun this step, this is why there's now a difference compared to when we originally ran it, the model did arrive at the right answer. Then you-- next, we have evidence provenance. So in this example, the model arrived at the wrong conclusion. It said that this was

[7:40](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=460s) valid, but it actually pointed to the wrong account. Um, and if you look at the citation, the model claims that it found this piece of evidence from January bank statement page two. And the thing is, th-this citation isn't useful because you can imagine if you have a big engagement, you could have many such documents with the same name. Even if this is a unique document name, what version did the model look at? Who uploaded this? Like, where did this even come from? And this isn't even related to hallucination. It can be a real document, but we need to understand more about the

[8:10](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=490s) context behind that document. So what we arrive at is a citation is identity. It's not just a string. It's-- you have a very specific ID, a very specific version of the document, a hash of the contents behind it. And being able to concretely tie where an agent picks something from allows you to then assert rules around the evidence that's allowed or any kinda, um, quality constraints you want around it. For example, we may only want evidence that's sourced directly from the bank as opposed to something that the client

[8:40](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=520s) uploaded where they may be able to tamper w-with it in some sort of way. And finally, we arrive at evaluation integrity. So we claim we have this eval, this gold data that was based on last year's reviewed audits. But just because the audit was reviewed doesn't necessarily mean it's correct. So it's important to audit that data set as well because mistakes in it could punish a right agent or it could reward a wrong agent. So there's some examples here of how in the reviewed audit you could

[9:10](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=550s) have examples that were then superseded, you could have, uh, you could have decisions that were made based on out-of-band evidence, or you could even have changes in methodology and all of these are examples of how your underlying gold data may not be as gold as you thought it was. And what this can kinda look like in practice is that this kinda headline great overall score can hide a lot of underlying failures. So in this case we have ninety-seven percent, but out of nine of the misses, four was because the

[9:40](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=580s) eval itself was just wrong. So our agent is actually doing better than expected. But there are twelve examples where our eval said this should be no exception, but because of some business rule changes or because of some audit methodology changes, these are now wrong. So these are examples where the agent actually missed and was incorrect. So what we see is we don't know if our agent is doing better than expected, it's doing worse than expected. The score is hiding a lot of important context behind how effective our agent actually is.

[10:09](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=609s) And, you know, ultimately there is a point where the agent has to stop and ask a human, and all of this ties back to kinda examples we've already covered through these past few controls. If it's trying to change work a human has already reviewed, if a new run is disagreeing with a previous run, if evidence is not conforming with these different rules, all of these are examples where something has gone awry and a human needs to step in to determine what's happening. And the key thing here is that these escalations, they should be easy to verify but not necessarily easy to

[10:39](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=639s) approve. We know automation bias is real. If you can-- if you continually ping the human, they'll just start hitting yes to everything. So we only want to flag exceptional cases, and those exceptional cases should very clearly call out like what is wrong and where does someone need to step in. So the example here is we have this deposit. It was cleared in eleven business days, but it's calling out what is the discrepancy and what needs to be investigated. It may take some time for the human to do this review and to eventually arrive at an approval decision, but

[11:09](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=669s) that's okay because we know eighty percent of the time the agent will do what it needs to do and for those twenty percent of cases when we want the human to step in, we wanna make sure they're stepping in where it counts and they have the context they need to arrive at the right decision. So if we kinda take a step back, we've covered these four controls and what we arrive at is defensibility is an and in the sense that you need all of them, otherwise things can only s-- can always slip through the cracks. If you skip authorization, then you could

[11:39](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=699s) have, uh, a scenario where someone reviewed something but then the answer changes out underneath them. If you skip execution control, you can arrive at the right answer but you can't show how. You may as well attribute it to luck at that point. If you skip evidence provenance, you can have the wrong answer but you don't know where the wrong answer came from. It came from some citation that you can't tie back to a concrete source. And if you skip evaluation integrity, you can arrive at a great flagship number of ninety-seven percent, but you have no idea what the reality is. Are you

[12:09](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=729s) doing better than that? Are you doing worse than that? All you know is you have this number to go off of. And this pattern for assurance, you know, I've kind of been talking about audit, but it really scales beyond audit into many other domains. In credit, if you're approving or declining a loan, you can interrogate that. You know, why did you make that decision? In AML, if you're clo-closing or escalating alert, again, what was the evidence behind that? So these same four questions can apply to different domains where you may change the nouns behind them, but the principles, uh,

[12:39](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=759s) are still the same. So, you know, I think, uh, what this really comes down to is anytime you're making a financial decision where the impact could be quite, uh, tremendous, you need to have some defensibility behind how you're arriving at that decision. And if your a-- if your defense is the model said so, that's, uh, not really gonna cut it in today's day and world. So thank you all for your time. Uh, these are some of the things we're working

[13:09](https://www.youtube.com/watch?v=gdzsJZJ6DDs&t=789s) on at Fieldguide, so if you're interested, we are hiring. And if you want to learn-- talk more about this, feel free to find me at the conference. But yeah, thank you all for your time and hope you enjoy the rest of today.
