# Claude Code Looked Right but Missed $800K - Natalia Connolly & Kevin Madura, AlixPartners Kevin Madura, AlixPartners · Natalia Connolly, AlixPartners [Talk page](https://ai.engineer/talks/u7fsyrXsPBE-claude-code-looked-right-but-missed-800k) · [Video](https://www.youtube.com/watch?v=u7fsyrXsPBE) ## Claude Code Looked Right but Missed $800K Claude Code got annual spend right and missed roughly $800,000 in contract value. In a post-merger procurement analysis, those two numbers depended on different evidence. So did the deadline for doing anything about them. ### What can the buyer take into a negotiation? [0:01](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1s) A large healthcare company had acquired a smaller specialist. Its procurement team now had hundreds of vendors across the two companies, more than 40,000 distinct contract documents, and several databases with no links between them. Even for a vendor that both companies used, the buyer had to piece together what each company bought and on what terms. Natalia Connolly and Kevin Madura of AlixPartners used Claude Code on the real engagement and illustrated it with a synthetic replica. In that replica, Harbor Health had acquired Summit Health Plan. Both bought services from Acme Health Services. Could Harbor negotiate better terms on their combined business? How much would that save, and how soon did Harbor have to act? The procurement lead would be sitting across the table from Acme. If Acme challenged a rate or disputed how much Harbor had paid, the lead needed to be able to turn to a particular contract page or invoice line. ### The warehouse check caught two different errors [2:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=150s) Claude Code’s first answers looked complete enough to use. Which ones could the buyer rely on? Connolly and Madura went back to the warehouse and checked each answer. Total annual spend was correct. So were the split between the two companies and the contract counts. Total contract value, a separate figure, was off by approximately $800,000. The second error changed the buyer’s options. For the contract due to expire first, Claude gave December 1 as the deadline to prevent automatic renewal. The actual deadline was November 1. A buyer who waited until the reported date would already have let the contract renew. ### A missing document, a missing link, and a misleading sum [3:39](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=219s) Two failures can make a contract search look complete while leaving money out of the answer. - **A document never arrives.** A long contract can hang during preprocessing. If the job catches the timeout and moves on, later queries run against a database that never received the contract. - **A document arrives without its parent link.** An amendment can exist in the portfolio and still be unreachable by following the parent contract’s records. This was the cause of the roughly $800,000 miss. Claude could find the contract and still miss the amendment that changed its value. Finding every amendment creates a different question. Which of its dollars should enter the total? In another example, a master service agreement and two amendments each supplied a contract value. Extract the three dollar amounts and add them, and the result is about $17 million. Read the language around the amounts and the operation changes. Each amendment restated the *total* value of the agreement. The latest amendment put that value at $7 million. Summing the records counted earlier versions of the same commitment. Even a single document can separate a number from its meaning. A service rate table runs onto a second page. The rows continue, but the column headers stay on page one. An extractor can transcribe every digit on page two and still have to guess what each value represents. The extraction needs to carry the first page’s column meanings into the continuation. ### Four stages of preparation before Claude analyzes spend [5:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=346s) The two engineers spent about two weeks building a five-stage pipeline to handle problems like these. Once it was ready, it processed more than 40,000 contract documents overnight. 1. **Preprocess the documents.** Document processing and optical character recognition (OCR) made page contents available for extraction. 2. **Classify each document.** Identify a master service agreement, amendment, or nondisclosure agreement so the next stage knows what kind of document it is reading. 3. **Extract terms, rates, and spend.** Handle layout problems here, including rate tables that break across page boundaries. 4. **Map services to spend categories.** Give the services in the contracts a common set of categories for analysis. 5. **Analyze the extracted data.** Claude Code can now work across structured tables. Those same tables give the engineers a way to check its answers programmatically. ### What OCR leaves behind in a chart [7:08](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=428s) What would a reader know about a chart if all they had were its axis labels? Basic OCR can recover the text on the axes while losing the relationship drawn between them. The team supplied contract pages as images and extracted structured output from the visual contents. That gave the analysis a database of relevant elements even where the page conveyed information graphically. The choice to define fields up front came from the procurement questions. The team already knew it would need rate tiers and rebates. It could tell the extractor what to look for, then give Claude the resulting records to explore. Starting directly from unstructured pages would have left the agent to discover those categories as part of each analysis. ### Put the deal’s business rules in the tools [8:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=506s) With contract records, spend data, and invoices normalized together, Claude Code could list tables, execute SQL, and build more specialized tools for itself. Two operations useful on this deal were retrieving a rate-card page and looking up spend for a specific vendor. A fresh agent could ask for the business object it needed. The tool’s query could apply what the team had already learned about how to retrieve it. Exploration still mattered. Much as it could navigate an unfamiliar codebase, Claude Code could move through the tables and pull together the elements a question required. Once a retrieval method worked, the team could record it in a prompt. They generally preferred a custom tool or database view for a rule they needed to enforce. Each caller would then get the rule as part of the query, without having to interpret an instruction afresh. Custom MCP servers and configurations offered a way to carry those operations forward through the engagement. They also gave the team a place to govern access. It could define the set of operations an agent was allowed to perform against the underlying data and block calls outside that set. ### Find the governing clause and the agreement it changes [11:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=674s) December 1 had a source. Claude had pulled a date from the termination clause. Harbor’s question was about preventing auto-renewal, which required calculating the notice deadline from the renewal clause. That calculation gave November 1. A citation to the termination clause could have proved that Claude read a real date while doing nothing to help the buyer act on time. The reusable rule here was which clause to consult and what to calculate when a buyer asks about renewal. The missing amendment called for a different repair. A contract lifecycle management system (CLM) might already hold vendor names and termination dates. In Madura’s experience, its records could still be poorly maintained. The team used the available data and also had Claude read the documents from the bottom up to recover relationships the CLM had missed. The parent of an amendment might itself sit below the master agreement. A master service agreement can cover several statements of work, each with amendments of its own. Reconstructing that hierarchy gives a loose amendment a specific place to attach. An analyst can then follow one statement of work through its revisions, determine which terms still apply, and include its current value once in the vendor relationship. ### When a failed lookup looks like a zero-dollar invoice [13:25](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=805s) The two merging companies had different arrangements for accessing their data. Some tool calls encountered permission problems or returned a malformed value or a zero. Without feedback that explained the failure, Claude could carry on with the apparent value. A lookup problem had become an assertion about the business: the invoice was worth zero. An explicit access-denied result changes what the agent can reasonably do next. It can flag the missing information for a human, or consult another authoritative database if the same information is available there. Either route preserves the distinction a savings calculation depends on. A zero is an amount the calculation can use. A denied lookup leaves the amount unknown. ### Parallel searches can claim the same savings [14:37](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=877s) Experienced procurement colleagues had a library of ways to look for savings. One analysis asks where the two companies share a vendor, as Harbor and Summit did with Acme. Another asks where different vendors supply the same service. The team dispatched agents to pursue analyses like these in parallel. Each could turn up a plausible opportunity on its own. The dollar amounts could only be combined after someone checked whether multiple findings claimed the same underlying savings. Where should the analyses meet? The team added a reconciliation step after the agents returned their findings. Claude had latitude to explore within each analysis. At reconciliation the workflow prescribed which data points to compare, how to resolve overlap, and which next steps were acceptable. A common source or vendor gave the workflow a reason to examine two findings together. The question was whether realizing the first saving would already realize some or all of the second. Where do independent savings searches meet? Two examples from the procurement library show the split. The agents run independently. Their findings come together at reconciliation, where overlap matters to the combined estimate. - Procurement analyses — Assign agents to investigate different ways the merger could reduce spend. - Shared vendor — Look for a supplier used by both companies. - Shared service — Look for different suppliers providing the same service. - Reconcile findings — Compare the underlying opportunities under specified rules so the same savings enter the estimate once. - Procurement analyses → Shared vendor (Assign search) - Procurement analyses → Shared service (Assign search) - Shared vendor → Reconcile findings (Return candidate findings) - Shared service → Reconcile findings (Return candidate findings) ### Management had started three months ago [16:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=970s) Even a correctly calculated, unique opportunity could be useless as a recommendation. The merger was already under way. Claude could identify a promising action and management could dismiss it immediately because work had started three months earlier. The contracts could support the savings estimate. Knowing whether anyone still needed to act required information from the business. Timing that context was tricky. Constrain the search too early and an agent may skip productive analysis. Wait until the recommendations reach management and the team may spend the meeting rediscovering work in progress. Madura described tagging relevant project information as the analysis proceeded and making it available on later passes. A memory system or a tool could carry that knowledge forward so the next agent could spend its time on an open question. For this engagement, Connolly and Madura judged the up-front cost in engineering time, money, and tokens worthwhile because later agents could use the prepared data without repeatedly discovering how it fit together. Business context had a different cost. More room for contract text would still leave Claude unable to tell the procurement lead what management had started three months ago. Someone had to supply that fact while it could still change the recommendation. ## Key takeaways - A usable contract value depends on the parent link and the amendment language. A missing link hid roughly $800,000 in one example. Adding restated totals turned $7 million into about $17 million in another. - Give a fresh agent access to the rules that make a query meaningful. A renewal question calls for a deadline calculated from the renewal clause. A denied invoice lookup calls for explicit feedback so the unknown amount stays out of the arithmetic. - A savings proposal has two further tests. Does another finding already claim those dollars? Has management already started the work? Reconciliation answers the first. Current information from the business answers the second. ## Slides ### Claude Code, the financial analyst AlixPartners put Claude Code to work on a real sourcing analysis. Natalia Connolly and Kevin Madura explain what it took to trust the agent’s numbers, using fictional entities and figures in their examples. [0:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=12s) [The AlixPartners title slide asks what it took to trust an AI agent’s sourcing analysis and outlines the business problem, the solution and the lessons learned.](https://assets.ai.engineer/sites/aiecode2025/sha256/51a4ac664cf89a884913d251e6268e484784e669146622f7b5103196e706bf4a.webp) Claude Code, the financial analyst AI ENGINEER NYC · LEADERSHIP SESSION AlixPartners OCTOBER 2026 What happened when we put an AI agent on a real sourcing analysis, and what it took to trust its numbers. THE BUSINESS PROBLEM THE SOLUTION WHAT WE LEARNED Natalia Connolly | Kevin Madura | AI and Data · AlixPartners Client unnamed. Examples use a synthetic replica: entities and figures are fictional. 01 Interface text: 62:47 P presenter view N notes F full screen ? ### The healthcare acquisition and its procurement scope After the acquisition, procurement had to make sense of hundreds of vendors and 40,000+ contract documents across both companies. Three databases had never been joined, so the team lacked a single view of each vendor’s contracts. [1:11](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=71s) [Below the acquisition diagram, three boxes show hundreds of vendors across the two companies, 40,000-plus contract documents and three databases that had never been joined.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6bdb172ae4dcbb73ae7d2c03f2c4ffd6a83a211e8eb1f6e73f1ab887d4dbbe3.webp) THE SITUATION A large healthcare company bought a smaller business that runs one area very well AlixPartners THE ACQUIRER Large healthcare company Some presence in the target’s area, but not much ACQUIRED THE ACQUIRED BUSINESS Smaller specialist that runs one area extremely well Its own vendors, contracts and systems Hundreds of vendors across the two companies 40,000+ contract documents 3 databases that had never been joined AlixPartners · AI Engineer NYC · real engagement, client unnamed LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 02 Interface text: 63:25 P presenter view N notes F full screen ? ### Where can we save, and by when do we need to act? In the fictional deal, Harbor spends $4.64M a year with Acme and Summit spends $3.06M a year under separate contracts. To negotiate better terms, Harbor’s sourcing lead needs to trace each figure to a contract page or invoice line. [1:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=89s) [A diagram shows two fictional healthcare companies buying from the same vendor under separate contracts. Each connection lists annual spending, and the text asks for a source page supporting every figure.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ce73c89b4d1c4c7bab7202b62adedae580e88198c156d3d59dc93094bceee2a.webp) THE SITUATION · WHAT SOURCING ASKED Where can we save, and by when do we need to act? In our replica, Harbor bought Summit. Both buy from the same vendor, Acme, on separate contracts. The bar: a sourcing lead has to defend every number in front of the vendor, and show the page it came from. Harbor Health ACQUIRED Summit Health Plan NO COMBINED VIEW $4.64M per year · own contracts $3.06M per year · own contracts Acme Health Services AlixPartners AlixPartners · AI Engineer NYC · fictional entities LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 03 Interface text: [illegible] P presenter view N notes F full screen ? ### Two of the agent’s five numbers were wrong, and nothing flagged it The team recomputed all five figures from the warehouse. An unlinked amendment left Claude Code’s contract value $800,000 too low. Waiting until December 1 would have missed the November 1, 2026 deadline to stop renewal. [2:31](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=151s) [A five-row table lists total annual spend, spend by company, active contracts, current contract value and the next notice deadline. The heading reports two errors, but the rows have no verdicts yet.](https://assets.ai.engineer/sites/aiecode2025/sha256/375b93e6b8a13dba374c325ba015e49279b33a35d6a7de4b1904c4485483d43d.webp) PROBLEM · THE FIRST ANSWER Two of the agent’s five numbers were wrong, and nothing flagged it AlixPartners json_finding · vendor: Acme Health Services $ claude --strict-mcp-config # analyse our position with Acme Total annual spend $7,699,213 Harbor / Summit $4,635,096 / $3,064,117 Active contracts 4 Current contract value $16,700,000 Next notice deadline CW-10033 · 2026-12-01 AlixPartners · AI Engineer NYC · synthetic replica LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 04 Interface text: 04:33 P presenter view N notes F full screen ? [2:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=175s) [Three rows have green checks for annual spend, the spending split, and the active contract count. Two rows have red corrections: an unlinked amendment raised the contract value, and the notice deadline fell a month earlier than the agent reported.](https://assets.ai.engineer/sites/aiecode2025/sha256/f8674079824a4482c38af8b509e96a83f0d441576514ec8850e7e0d1081e0415.webp) PROBLEM · THE FIRST ANSWER Two of the agent’s five numbers were wrong, and nothing flagged it AlixPartners json_finding · vendor: Acme Health Services $ claude --strict-mcp-config # analyse our position with Acme Total annual spend $7,699,213 ✓ CORRECT Harbor / Summit $4,635,096 / $3,064,117 ✓ CORRECT Active contracts 4 ✓ CORRECT Current contract value $16,700,000 ✕ WRONG Actually $17,500,000: an amendment was never linked to its contract Next notice deadline CW-10033 · 2026-12-01 ✕ WRONG Actually 2026-11-01 (last day to stop the renewal). Waiting for the agent’s date would have missed it ✓ / ✕ Each figure checked against numbers recomputed directly from the warehouse We only caught them by recomputing every figure from the warehouse. AlixPartners · AI Engineer NYC · synthetic replica LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 04 Interface text: [illegible] P presenter view N notes F full screen ? ### Real contract data are messier than they appear A long contract stalls silently. A searchable amendment lacks a link to its contract, leaving $800K out of the total. Summing three versions turns a $7M contract into $17M. A page break separates an $88/hr rate from its header. [5:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=309s) [Four cards show a long contract stalled in preprocessing, an unlinked amendment missing from a total, a restated value double-counted by summing document history, and a rate table split across pages. The last card recommends grouping the table with its continuation to preserve the header.](https://assets.ai.engineer/sites/aiecode2025/sha256/3917e3cd6ae69ed605d593da13b057bd4e4faec558705d6dd91b80c57caf8ca8.webp) Real contract data are messier than they appear PROBLEM · MESSY CONTRACT DATA AlixPartners BREAK 01 · LENGTH A very long contract that never loads NEVER LOADED hangs in preprocessing Not in the warehouse, so no query can see it. Nothing errors. BREAK 02 · MISSING LINK An unlinked amendment $800K missing from the contract total Extracted and searchable. Not linked to its contract. BREAK 03 · HISTORY One contract, three documents MSA $4.0M Amendment 1 $6.0M Amendment 2 · RESTATES $7.0M SUMMED $17.0M $7.0M Amendments replace values. They don’t add to them. BREAK 04 · LAYOUT A rate table split by a page break [illegible] EXHIBIT B — RATE SCHEDULE The hourly rates applicable to the Services are set forth in the table below. Rates are stated in United States dollars and are firm and fixed for the initial Term pursuant to Section [illegible]. LABOR CATEGORY [illegible] RATE UNIT ONSHORE Correspondence Clerk [illegible] hour Yes Claims Intake Associate [illegible] hour Yes Claims Processor I [illegible] hour Yes Claims Processor II [illegible] hour Yes Claims Processor III [illegible] hour Yes [illegible] PAGE BREAK [illegible] LABOR CATEGORY [illegible] RATE UNIT ONSHORE Claims Examiner I [illegible] hour Yes Claims Examiner II [illegible] hour Yes Appeals Specialist [illegible] hour Yes Payment Integrity Analyst [illegible] hour Yes Senior Claims Analyst [illegible] hour Yes Quality Assurance Analyst [illegible] hour Yes “Senior Claims Analyst, $88/hr” lives on the continuation page, which has no header row. Fix: group continuation tables so the header carries over. AlixPartners · AI Engineer NYC LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 05 Interface text: 67:41 P presenter view N notes F full screen ? ### Two weeks of building, then 40,000 contracts processed overnight The pipeline turned 40,000+ contract documents into queryable tables overnight, work the team estimated would take 10 reviewers weeks. Claude Code used those tables for analysis, and the team could check each answer in code. [6:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=415s) [A five-stage pipeline processed overnight what the slide estimates would take ten reviewers weeks. The result cards show 40,000-plus documents converted to queryable tables, three databases joined into one contract and vendor view, and about two weeks to build the pipeline.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7b7d86136cf6205f0d88f7166e6e71917ed649c464f5a492f08d49fbe7e11ad.webp) Two weeks of building, then 40,000 contracts processed overnight SOLUTION · STRUCTURE FIRST AlixPartners READING 40,000+ CONTRACT DOCUMENTS Manual review Team of 10 · weeks The pipeline Overnight 1 · Preprocess Document Intelligence, DSPy extraction 2 · Classify MSA / SOW / amendment 3 · Terms, rates, spend 3 databases joined by a crosswalk 4 · SOW services Mapped to spend categories 5 · Analyst agent Claude Code over MCP 40,000+ documents become structured, queryable tables 3 → 1 databases stitched into one contract and vendor view ~2 wks to build the pipeline; after that it reruns on demand AlixPartners · AI Engineer NYC LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 06 Interface text: [illegible] P presenter view N notes F full screen ? ### Reading rebate tiers from a contract page image OCR captures the chart’s axis labels. The model reads the bar heights and pairs each annual spend threshold with its rebate. In this example, the $1 million threshold carries a 2.5% rebate, a value between two labeled ticks. [7:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=449s) [Beneath the illustrative chart and OCR text, a typed schema pairs minimum spend with rebate percentage. The four output records pair $0, $1 million, $5 million and $10 million with rebates of 1%, 2.5%, 4% and 5%, respectively.](https://assets.ai.engineer/sites/aiecode2025/sha256/9a0a46a844d54fa80241a2634f50c2f5baf65f8e565b1d9184aa500b74736a07.webp) Some data only lives in charts, so the model reads the page image — SOLUTION · TYPED AND MULTIMODAL AlixPartners A CONTRACT PAGE · ILLUSTRATIVE Figure 3. Volume rebate by annual spend tier Y-axis ticks: 6%, 4%, 2%, 0% X-axis tiers: $0-1M, $1-5M, $5-10M, $10M+ TRADITIONAL OCR RETURNS Figure 3. Volume rebate by annual spend tier 6% 4% 2% 0% $0–1M $1–5M $5–10M $10M+ Just loose text: no rebates, and nothing tying a tier to its percentage. class RebateTier(BaseModel): min_spend_usd: int rebate_pct: float class ReadRebateChart(dspy.Signature): """Read the volume-rebate chart on a contract page.""" page: dspy.Image = dspy.InputField() tiers: list[RebateTier] = dspy.OutputField() MULTIMODAL READ, TYPED OUTPUT RebateTier(min_spend_usd=0, rebate_pct=1.0) RebateTier(min_spend_usd=1000000, rebate_pct=2.5) RebateTier(min_spend_usd=5000000, rebate_pct=4.0) RebateTier(min_spend_usd=10000000, rebate_pct=5.0) Read straight off the bar heights. AlixPartners · AI Engineer NYC · illustrative chart LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 07 Interface text: 70:21 P presenter view N notes F full screen ? ### Verifying the amendment link restores $800K to the total A second Claude process found one matching Summit contract. Code verified that it appeared in Claude’s search results and that quotes matched both documents word for word. The $17.5M total then included the $800K increase. [13:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=787s) [A green line connects the amendment to its parent contract. Code checks that the contract appeared in the agent’s search results and that quotes match both documents word for word. The displayed total is $17.5M.](https://assets.ai.engineer/sites/aiecode2025/sha256/e03e8b91ae88a6d62602b8e7a6b89cf72544358d153ccefbb97686afd8e64131.webp) LESSON 2 · A DATA PROBLEM, REPAIRED BY A HARNESS STEP One unlinked amendment left $800K out of the total. AlixPartners UNLINKED AMENDMENT INTAKE-2026-0447 Raises a Summit contract from $5.4M to $6.2M. parent contract: NULL So no report counts it. ACME CONTRACTS ON FILE CW-10021 Harbor · MSA CW-10033 Harbor · SOW CW-10044 Harbor · SOW SUMMIT.4412 $6.2M SUMMIT.4418 Summit · SOW vendor Acme · customer Summit · type MSA → 1 match CODE CHECKS BEFORE LINKING ✓ The contract was in Claude’s own search results ✓ Claude’s quote from each document appears word for word CURRENT CONTRACT VALUE $17.5M A second Claude process finds the missing parent, and code checks the link before it counts. AlixPartners · AI Engineer NYC LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 10 Interface text: 75:17 P presenter view N notes F full screen ? ### Access controls led the agent to report zero spend The agent could see Summit’s Acme contracts. Row-level security silently hid the invoices, so the agent reported $0 in invoiced spend for 2025. The fix adds a tool warning and draws totals from an approved view. [14:25](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=865s) [A dashed bar marks hidden invoice rows. Beside it, the agent reports $0 in invoiced spend despite seeing contracts. The fix adds a tool warning and draws totals from an approved view.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1c019a2eed3bdeac470c8d59115560576c06aa7be47ced02620db5516530aab.webp) LESSON 3 · A PROCESS PROBLEM Access controls hid Summit’s spend, and the agent called it $0 AlixPartners ● ROW-LEVEL SECURITY ON · AGENT NOT TOLD Harbor · Acme spend $4,635,096 Summit · Acme spend $0 DASHED OUTLINE = THE ROWS THAT WERE REALLY THERE “Summit total $0 (contracts exist; no invoiced spend in 2025).” THE AGENT, WHO COULD SEE SUMMIT’S CONTRACTS BUT NONE OF THEIR INVOICES Hidden rows read as zero. The fix: the tool warns, and totals come from an approved view. AlixPartners · AI Engineer NYC LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 11 Interface text: 76:30 P presenter view N notes F full screen ? ### Counting each invoice line once reduces claimed savings to $1.4… In the example, five agents claimed $2.9M in potential savings. Matching invoice-line IDs exposed $1.5M in overlap. The harness counted each shared line once, giving termination priority over consolidation, then renegotiation. [15:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=917s) [The initial $2.9 million total is crossed out. Matching invoice lines reveals $1.5 million in overlap. Assigning each shared line to termination first, consolidation second and renegotiation third leaves $1.4 million.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7c45933f836dcc698107d8c1f9699b8e02c38d82d9d78c95cedb5e158549518.webp) LESSON 4 · A PROCESS PROBLEM Valid findings, but double counted. AlixPartners FIVE AGENTS, IN PARALLEL, EACH BLIND TO THE OTHERS Each claims an amount and the SQL for its invoices redundant_service $400K dead_contract $900K duplicate_vendor $800K term_arbitrage $700K rate_benchmark $100K NAIVE TOTAL $2.9M THE HARNESS MATCHES INVOICE-LINE IDS $1.5M of the claims sit on invoice lines another agent already claimed. A saving can only be made once. ONE RULE DECIDES WHO OWNS EACH SHARED LINE $1.4M 1 Terminate → 2 Consolidate → 3 Renegotiate Each invoice line is counted once, under the highest-priority saving. The two valid, independent claims stay in full. The rules are applied in code by the harness. The order is agreed with the commercial team. AlixPartners · AI Engineer NYC · illustrative example LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 12 Interface text: [illegible] P presenter view N notes F full screen ? ### The agent recommended savings the company was already acting on The agent correctly estimated the savings from consolidating call-centre providers. The company had already started the work. An analyst recorded that context so later runs would start with updated rules. [16:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=967s) [Green checks show that the agent analyzed 30 vendor relationships and estimated savings from rates, minimums and contract terms. Red crosses identify gaps in its view of ongoing projects, deal knowledge and actual commitments.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce6f0f85c5813c88772336c9aa7e1e2577be67ca74ebdea8b318c5b37d921375.webp) LESSON 5 · DEAL CONTEXT The agent recommended savings the company was already acting on AlixPartners WHAT THE AGENT GOT RIGHT ✓ Top 30 vendor relationships, analysed at the same time ✓ Savings sized from hourly bill rates and contracted minimums, combined with the contract terms ✓ By hand: about one analyst-day per relationship WHAT IT COULD NOT SEE × Projects already in flight or actioned × Deal knowledge that lives with the business, not in spend or contracts × Headline contract values that do not reflect what is really committed AlixPartners · AI Engineer NYC LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 13 Interface text: [illegible] P presenter view N notes F full screen ? [17:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1020s) [The agent correctly estimated savings from consolidating call-centre providers, but the work was already underway. Three arrows show an analyst adding deal context, knowledge accumulating, and later runs starting with updated rules.](https://assets.ai.engineer/sites/aiecode2025/sha256/df67788548d9c22b2a73150c6fbb2886e1a390cd47a3232032d61ea809e6ede8.webp) LESSON 5 · DEAL CONTEXT The agent recommended savings the company was already acting on AlixPartners WHAT THE AGENT GOT RIGHT ✓ Top 30 vendor relationships, analysed at the same time ✓ Savings sized from hourly bill rates and contracted minimums, combined with the contract terms ✓ By hand: about one analyst-day per relationship WHAT IT COULD NOT SEE × Projects already in flight or actioned × Deal knowledge that lives with the business, not in spend or contracts × Headline contract values that do not reflect what is really committed VALID ALREADY ACTIONED “Consolidate the call-centre providers.” Obvious, correctly sized, and already under way at the company. 1 · An analyst adds what the data can’t know Deal and client context 2 · Knowledge accrues Specific to the deal and the client 3 · Every later run starts with updated ruleset The agent learned as we did AlixPartners · AI Engineer NYC LESSONS 1 Rules in code 2 Repair the data 3 Silent failure 4 No double counting 5 Deal context 13 Interface text: [illegible] P presenter view N notes F full screen ? ### Two takeaways Structuring the data up front helps the agent avoid spending tokens rediscovering the environment. It cites a source for every claim. Checks recompute the numbers and send failures back to the agent. A person applies business context. [17:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1077s) [The slide recommends spending tokens up front to structure data and emphasizes business context. Arrows connect structured data, read-only agent tools, an answer with evidence, checks against the sources, and a person who makes the final calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/c55ae4026d631a7db2ab7d76b050b7874768f7a4c7a25e4d6427221e01c2653f.webp) TWO TAKEAWAYS 1 Invest tokens up front to structure your data 2 Context is still king ALIXPARTNERS · AI AND DATA AlixPartners THE DATA Structured data Built up front, so the agent can query it THE AGENT Read-only tools Reads the data, proposes an answer IT HANDS IN Answer + evidence Every claim shows its source BOUNDED REVIEW · BUILDS HUMAN-READY RESULTS • Recomputes the numbers from the source • Checks each claim against its evidence • Catches duplicates and gaps • Sends failures back to the agent A PERSON Judgment calls Makes the final calls and checks a sample AlixPartners · AI Engineer NYC 14 Interface text: 80:21 P presenter view N notes F full screen ? ## Full transcript [0:01](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1s) [music] [0:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=12s) >> Hey everybody. [0:13](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=13s) Nice to see you here. My name is Kevin [0:15](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=15s) Madeira. My colleague [0:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=17s) Natalia [0:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=19s) and I are from Alex Partners, [0:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=21s) which is a global management consulting [0:22](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=22s) firm. We focus primarily on bankruptcy [0:25](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=25s) and restructuring, business and [0:27](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=27s) technology advisory and risk and [0:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=28s) compliance. And I'm part of the AI [0:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=30s) engineering team there. [0:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=32s) >> Awesome. Thanks, Kevin. Natalia Conley, [0:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=34s) part of the AI and data team at Alex [0:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=36s) Partners. And today we'd like to share [0:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=38s) our experience with putting cloud code [0:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=41s) on a real procurement analysis with you. [0:43](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=43s) Our talk will have three parts. So we'll [0:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=45s) talk about the business context and then [0:47](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=47s) our approach and then some lessons [0:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=49s) learned. [0:50](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=50s) All right, let's get to it. [0:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=52s) Okay, so some business context first. [0:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=55s) Let's consider a very large healthcare [0:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=57s) company and it just acquired a smaller [1:01](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=61s) sort of niche player that's extremely [1:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=63s) good in one particular area. [1:05](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=65s) And the parent company has some [1:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=67s) experience in that area, but just not [1:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=69s) that much. [1:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=70s) So with that deal handed, the [1:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=72s) procurement team is hundreds of vendors [1:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=74s) across the two companies, [1:16](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=76s) over 40,000 distinct contract documents [1:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=79s) and multiple databases that really were [1:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=81s) not linked to one another. So there was [1:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=83s) no holistic view really of each vendor [1:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=86s) and each contract relationship. [1:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=89s) Okay. [1:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=90s) From here on now, let's deal with with a [1:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=92s) synthetic replica of the situation. So [1:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=94s) here it is, right? So Harbor Health is [1:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=96s) our big parent company and it acquired [1:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=98s) Summit Health Plan. That's the small [1:40](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=100s) regional player that's extremely good in [1:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=101s) this one area. [1:43](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=103s) And both of them have a relationship [1:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=105s) with a particular vendor. Let's call it [1:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=106s) Acme Health Services. [1:48](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=108s) Okay. So after the merger, what the [1:51](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=111s) procurement team at Harbor Health really [1:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=113s) wants is answers such as, well, can we [1:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=116s) maybe get better terms from Acme now [1:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=118s) that we've merged? And if so, uh can you [2:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=120s) maybe size up the savings? And how [2:02](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=122s) quickly do I have to act in order to [2:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=124s) realize those savings? Right? So that's [2:06](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=126s) sort of the bottom line. And our job is [2:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=129s) to make sure that that procurement lead [2:11](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=131s) from Harbor Health, who'll be sitting [2:13](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=133s) across from Acme during negotiations, is [2:15](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=135s) well equipped to answer any questions [2:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=137s) that they might raise, any objections, [2:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=139s) anything, right? With a specific [2:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=141s) citations, uh citation with a specific [2:24](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=144s) contract page or a specific invoice [2:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=146s) line. So everything needs to be traced [2:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=148s) back to the source. [2:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=150s) Okay. So originally when we put Cloud [2:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=152s) Code to it, this is what it gave us, [2:35](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=155s) right? It came back with these answers. [2:37](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=157s) They look complete. They're certainly [2:39](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=159s) well formatted. They look plausible [2:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=161s) enough. [2:42](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=162s) You know, and uh so then we went back to [2:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=164s) the warehouse and we checked every [2:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=166s) single answer. Here's what we found out. [2:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=169s) The total annual spend, the split [2:50](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=170s) between the two companies and the [2:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=172s) contract counts were all correct, so [2:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=173s) that was fine. However, [2:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=176s) the total contract value was wrong by [2:59](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=179s) approximately $800,000. [3:02](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=182s) Right? So that's one. The second one was [3:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=184s) the the date uh by which we had to act [3:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=187s) in order to prevent the soonest expiring [3:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=189s) contract from auto-renewing was also [3:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=192s) wrong. The agent said December 1st. The [3:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=194s) actual date was November 1st, right? And [3:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=197s) so if we just kind of believe the agent [3:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=199s) and waited until December 1st, the [3:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=201s) contract would have auto-renewed and uh [3:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=203s) you know, whether we wanted it or not, [3:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=206s) right? And so that's the problem. [3:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=208s) So what happened? How come Cloud Code [3:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=210s) did those things and got two answers [3:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=212s) wrong? Well, the first thing to check of [3:35](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=215s) course would be the data. So let's talk [3:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=216s) about the data just for a second. [3:39](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=219s) And when you think about it, it just [3:40](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=220s) seems so straightforward to extract data [3:42](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=222s) from contracts, right? In reality [3:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=224s) though, contract data are a lot messier [3:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=226s) than than appear than they appear. [3:48](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=228s) So just some things that can potentially [3:51](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=231s) go wrong with contract data. [3:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=233s) For instance, you can have very long [3:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=235s) contracts and they just get kind of [3:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=237s) stuck in preprocessing and they just [3:59](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=239s) hang there. And maybe you catch the [4:01](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=241s) exception um with the timeout and and [4:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=243s) you kind of go on and you marry away, [4:05](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=245s) you never even know that the contract [4:06](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=246s) never made it into the database. That's [4:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=249s) one. [4:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=250s) Another one is orphan amendments. So for [4:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=252s) instance, there's an amendment to a [4:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=254s) parent contract, but for whatever reason [4:16](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=256s) it's just not linked properly to its [4:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=257s) parent. So it's just kind of floating [4:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=259s) out there as an orphan. And this is [4:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=261s) actually exactly what happened in our [4:22](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=262s) example where Cloud Code missed the [4:25](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=265s) $800,000. That's because it couldn't [4:27](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=267s) find that orphan amendment. So that's [4:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=269s) that's another potential issue. Another [4:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=272s) one, and this is pretty common, is where [4:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=274s) you have multiple amendments restating [4:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=276s) the original contracts terms. So here [4:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=278s) you have one MSA, Master Service [4:40](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=280s) Agreement, and two amendments that [4:42](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=282s) restated the total contract value. [4:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=284s) And if you just extract the structured [4:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=286s) outputs, the dollar amounts, right? Then [4:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=289s) your Cloud Code will very happily go and [4:50](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=290s) just sum up those dollar amounts and go, [4:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=292s) oh, the total contract value is about [4:54](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=294s) $17 million. [4:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=296s) But if you read the actual text, not [4:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=298s) just look at the numbers, the dollar [4:59](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=299s) numbers, right? The text says that every [5:01](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=301s) amendment restates the total contract [5:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=303s) value. So the correct answer is seven, [5:05](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=305s) which is the most recent amendment's [5:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=307s) value. That's another sort of gotcha to [5:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=310s) watch out for. Finally, you can have [5:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=312s) situations like that where you have very [5:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=314s) long tables. Let's say these are the [5:16](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=316s) tables of rates for various services, [5:18](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=318s) right? [5:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=319s) And they go across multiple pages. [5:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=321s) And so if you look at page two, it just [5:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=323s) starts out with, you know, some some [5:25](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=325s) numbers and there's no header. And Cloud [5:27](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=327s) Code might not necessarily know the [5:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=329s) context. I mean, it will just do its [5:31](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=331s) best to kind of guess, but it might not [5:33](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=333s) know what each value really means. These [5:35](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=335s) are just some examples. And none of this [5:37](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=337s) is exotic, but it just shows that real [5:39](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=339s) contract portfolios are full of issues [5:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=341s) that may or may not be apparent when you [5:43](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=343s) first started out on this on this [5:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=345s) journey. [5:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=346s) And that's why Kevin and I spent quite [5:48](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=348s) some time really building a super robust [5:51](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=351s) data processing pipeline in the [5:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=353s) beginning. We spent about two weeks on [5:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=355s) it just making sure that every single [5:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=357s) little thing like that that you just saw [5:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=358s) and more, [6:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=360s) all of that stuff was accounted for. And [6:02](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=362s) so [laughter] it was really great. So we [6:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=364s) spent those two weeks just building that [6:06](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=366s) robust pipeline and then 40,000 plus [6:08](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=368s) contracts were processed overnight once [6:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=370s) it was done. Um I imagine if we did it [6:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=372s) by hand, it would have required probably [6:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=374s) a much larger team than just us two and [6:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=377s) probably a little bit more time than [6:18](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=378s) that. [6:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=379s) But just a summary of the pipeline, it [6:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=381s) was a five stage pipeline. The first one [6:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=383s) was mostly preprocessing, so document [6:25](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=385s) intelligence, OCR, that kind of stuff. [6:27](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=387s) The second step was classification. So [6:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=389s) what is each contract? Is it a master [6:31](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=391s) service agreement? Is it an amendment? [6:33](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=393s) Is it [6:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=394s) an NDA? Something like that. Then there [6:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=396s) was the terms rates and spend [6:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=398s) extraction. That's the next step. And [6:40](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=400s) this is where we would account for [6:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=401s) things like very long tables that could [6:42](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=402s) break across page boundaries. [6:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=405s) And then we would map all the services [6:47](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=407s) to spend categories. Only then would [6:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=409s) Claude code go over the extracted data [6:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=412s) and and sort of do its thing. It was [6:54](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=414s) very important to have those structured [6:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=416s) tables, not only because it helped out [6:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=418s) Claude code, but also because it made it [7:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=420s) that much easier for for us later on to [7:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=423s) go and check every answer [7:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=424s) programmatically in code. [7:06](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=426s) All right, so Kevin. [7:08](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=428s) >> Sure, thanks Natalia. And so obviously [7:11](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=431s) contract data is very text heavy, but [7:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=434s) not always. And so there's sometimes [7:15](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=435s) there's multimodal elements. There might [7:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=437s) be charts or graphs or other things that [7:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=439s) if you just run it through a naive OCR [7:22](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=442s) process, you might get what you see on [7:24](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=444s) the right in terms of the the axes, but [7:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=446s) you actually lose the information that's [7:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=448s) contained within the chart. [7:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=449s) And so a lot of what we're talking about [7:31](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=451s) now, if you're bitter lesson pill, this [7:33](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=453s) will likely not be relevant maybe like [7:35](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=455s) six to 12 months, we'll see. But what we [7:37](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=457s) did here to solve this specific problem [7:40](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=460s) was take the contract pages as images [7:43](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=463s) and feed them through using DS Pi to [7:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=466s) arrive at a highly structured output, [7:48](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=468s) which is what you see on the bottom [7:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=469s) right there. So you go from this large [7:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=473s) unstructured data set to basically a [7:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=475s) database of elements that you know are [7:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=477s) going to be relevant for the balance of [8:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=480s) your work. And so it's very tempting to [8:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=483s) just let Claude rip, you know, on on the [8:06](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=486s) unstructured data [8:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=487s) and kind of have it figure it out, which [8:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=489s) is [8:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=490s) which is a valid approach, but there's [8:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=492s) certain engagements like this one where [8:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=494s) we know upfront we want rate tiers, we [8:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=497s) know we want rebates, we we know we want [8:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=499s) certain data elements. So you structure [8:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=501s) some of that upfront and then let Claude [8:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=503s) fill in that framework behind it. [8:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=506s) And so once you have everything in this [8:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=508s) highly structured format, [8:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=510s) you can then use Claude code to [8:31](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=511s) dynamically build tools for itself. And [8:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=514s) so you have common ones, of course, like [8:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=516s) execute SQL or list tables or things [8:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=518s) like that. But we found that it was it [8:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=521s) was really helpful for later [8:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=524s) implementations or later agents when [8:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=526s) they're coming in fresh to to the data [8:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=529s) to have some of these tools available to [8:50](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=530s) them. So invest a lot of time upfront to [8:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=533s) structure the data, as I mentioned [8:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=535s) previously, where you you essentially [8:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=538s) create these derivative databases off of [9:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=540s) contract data or other spend data or [9:02](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=542s) invoices. You normalize all of that [9:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=544s) together, and then you have Claude build [9:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=547s) these custom tools so that it has you [9:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=549s) know, kind of a shortcut, if you will, [9:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=550s) to get certain information that are [9:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=552s) specific, in this case, to this deal. Um [9:15](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=555s) and so you have things like get rate [9:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=557s) card page or get specific vendor spend. [9:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=559s) And that's where you can embed some of [9:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=561s) the business logic where you might know [9:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=563s) of some gotchas or some context that [9:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=566s) isn't immediately apparent or obvious in [9:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=568s) the contract data. You can start to [9:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=570s) embed some of that in the tool calls [9:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=572s) themselves, the way that it queries data [9:33](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=573s) or pulls things back um is very helpful [9:37](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=577s) to enforce some of those, you know, more [9:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=578s) custom pieces of logic. [9:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=581s) Um and Claude code was perfect for [9:43](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=583s) something like this because it's used to [9:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=585s) navigating autonomously around kind of [9:47](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=587s) messy code bases. This is a a similar [9:50](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=590s) concept. It's just contract data and [9:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=592s) spend data. So it's all largely in [9:54](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=594s) tabular format. Um but it can [9:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=596s) dynamically explore and pull in the the [9:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=598s) different elements that I mentioned [10:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=600s) previously. [10:01](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=601s) Um and then once we found something that [10:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=603s) was useful, that would work, we would [10:05](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=605s) package that up either into something [10:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=607s) specific in the prompt, but I think we [10:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=609s) we lean more on these custom tools and [10:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=612s) custom views within the database to [10:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=614s) enforce some of that business logic in [10:16](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=616s) code versus just relying on on a prompt. [10:20](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=620s) Um and once you have all that package [10:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=621s) together, you can start to develop [10:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=623s) custom MCP servers and and [10:25](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=625s) configurations that live on. And it's [10:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=628s) it's a it's a bunch of different [10:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=629s) elements uh that really get at how do [10:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=632s) you perpetuate and and accrue knowledge [10:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=634s) across the the life cycle, in this case [10:37](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=637s) for particular engagement. But if you're [10:39](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=639s) working on something internally, you [10:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=641s) right, you want to accrue the the [10:43](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=643s) knowledge and accrue certain business [10:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=645s) rules um that you want to enforce later [10:47](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=647s) down the line. So that this is a nice [10:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=649s) way to do it. [10:51](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=651s) And then of course, you have the ability [10:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=652s) to better govern access to the [10:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=655s) underlying data so you can block certain [10:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=657s) calls. There was a great talk right [10:59](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=659s) before this um regarding the security of [11:02](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=662s) how agents interact with the rest of the [11:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=663s) environment. It's actually pretty [11:05](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=665s) simple, I think, at the end of the day. [11:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=667s) You just have a defined set of [11:08](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=668s) operations that you're comfortable with [11:11](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=671s) and you block everything else and you [11:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=672s) can enforce that up front. [11:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=674s) So we'll walk through just a a couple of [11:16](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=676s) lessons here uh very quickly. Natalia [11:18](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=678s) mentioned previously about the agent [11:20](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=680s) using the wrong date. And so it's a good [11:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=683s) example of again, some external business [11:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=686s) context that is helpful and useful to [11:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=688s) embed into the agent up front because in [11:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=692s) this case, yes, it had pulled out the [11:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=694s) date from the termination clause, but [11:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=696s) actually the question we were asking was [11:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=698s) supposed to use the date from calculated [11:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=701s) from the renewal clause. And so it it [11:43](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=703s) seems like a minor thing, but it's it's [11:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=706s) uh it's a lot of these nuanced issues [11:48](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=708s) which compound your error rates. And so [11:51](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=711s) the more you can build these sorts of [11:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=713s) rules in up front, the better [11:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=715s) uh the result ultimately of course will [11:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=717s) be. [11:59](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=719s) Uh Natalia mentioned also the kind of [12:01](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=721s) the dangling references for different [12:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=723s) amendments. [12:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=724s) Um and so this is something that we run [12:06](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=726s) across a lot. So many organizations have [12:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=730s) a contract lifecycle management system, [12:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=732s) a CLM. [12:13](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=733s) It's intended to be uh a highly [12:15](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=735s) structured database of your contract [12:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=737s) data, right, the vendor name and ex uh [12:20](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=740s) extraction of termination dates and [12:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=741s) whatnot. They're almost never maintained [12:24](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=744s) properly or well. [12:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=746s) And so we often find we have to of [12:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=748s) course use the data that's available to [12:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=749s) us, but take somewhat of a bottoms-up [12:31](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=751s) approach to actually have Claude read [12:33](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=753s) through the contents of the contracts to [12:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=756s) find [12:37](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=757s) um instances of where it might not be [12:40](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=760s) properly captured in the CLM. And so you [12:42](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=762s) you have to go through and have Claude [12:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=764s) kind of build out this hierarchy of [12:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=765s) relationships. So you have [12:47](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=767s) an MSA that is tied to SOWs and then [12:51](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=771s) there are multiple amendments to those [12:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=773s) uh to those SOWs, and sometimes they're [12:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=776s) just not organized properly. And so [12:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=778s) again, structuring everything up front [13:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=780s) makes it really easy to tack on these [13:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=783s) kind of loose documents later on, which [13:05](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=785s) has a meaningful impact if you're [13:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=787s) looking for savings opportunities. In [13:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=789s) this case it was a an M&A, you know, [13:11](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=791s) consolidation. So you don't want to [13:13](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=793s) double count something, but you also [13:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=794s) want to capture the latest kind of state [13:16](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=796s) of the relationship with the vendor, [13:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=799s) which this kind of process allows you to [13:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=801s) do. So Claude was very good at at doing [13:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=803s) that. [13:25](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=805s) Uh I made a point before about governed [13:27](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=807s) access to the data. This can come back [13:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=809s) to bite you as well because there were [13:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=812s) two separate entities that were being uh [13:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=814s) put together here. The way that we [13:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=816s) accessed data from entity A versus [13:39](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=819s) entity B was different. [13:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=821s) And so we ran into a few instances where [13:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=824s) the returned value would have been [13:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=826s) something that was malformed or zero or [13:48](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=828s) there was some type of access [13:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=829s) permission. And Claude when making the [13:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=832s) tool call didn't really get the the [13:54](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=834s) right type of feedback, and so it kind [13:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=836s) of just went forward thinking that maybe [13:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=838s) an invoice was zero or or, you know, [14:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=840s) basically an incorrect value. [14:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=843s) And so a lesson learned there for us was [14:05](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=845s) anytime there's some sort of [14:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=847s) um you know, governance failure or [14:09](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=849s) access denied error or something like [14:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=850s) that, giving Claude more feedback about [14:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=854s) what went wrong allows it fail [14:15](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=855s) gracefully or otherwise recover, maybe [14:18](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=858s) flag something to a human or say, okay, [14:20](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=860s) I know this information is actually in [14:22](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=862s) this other authoritative database I can [14:24](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=864s) go to instead. [14:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=866s) Um [14:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=866s) because when you're relying on it for [14:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=868s) some sort of savings analysis or [14:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=870s) something similar, [14:31](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=871s) uh you want Claude to have as much of [14:33](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=873s) that context of course as possible. [14:37](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=877s) Another thing we did um was [14:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=881s) uh dispatch agents for different savings [14:43](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=883s) levers. So our colleagues who have been [14:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=885s) doing this for 20 years or whatever it [14:47](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=887s) is, they have a whole library of ways to [14:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=889s) identify savings. Some are more obvious [14:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=892s) than others, but [14:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=893s) okay, you're putting two companies [14:54](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=894s) together. Maybe they share vendors, [14:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=896s) maybe different vendors but same [14:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=898s) services. So you have a whole library of [15:01](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=901s) different activities and analyses that [15:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=903s) you could do. [15:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=904s) So we dispatched agents to do each of [15:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=907s) those in parallel um and they came back [15:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=910s) uh with actually pretty good results, [15:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=912s) but there's of course this step [15:13](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=913s) afterwards where you need to deduplicate [15:16](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=916s) the findings that come back. So [15:18](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=918s) um you can't ultimately double count in [15:20](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=920s) this case a potential savings if [15:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=923s) um you know if it's coming from the same [15:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=926s) source or the same vendor as an example. [15:29](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=929s) Um and so the way that we did this was [15:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=932s) um essentially build out an additional [15:34](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=934s) process and a and a [15:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=936s) very specific and precise [15:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=938s) uh set of workflows where you would kind [15:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=941s) of let Claude do its thing, do its [15:42](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=942s) analysis, but then the reconciliation [15:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=944s) step is very [15:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=946s) uh [15:47](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=947s) it's built around kind of DSPI [15:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=949s) ultimately where you're very [15:50](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=950s) prescriptive about which data points [15:52](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=952s) you're evaluating, how you do the [15:54](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=954s) reconciliation, and what's an acceptable [15:56](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=956s) next step in the process. There was [15:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=958s) another pretty good talk this morning [16:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=960s) about uh using DAGs for for policy [16:02](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=962s) execution. It's the same idea. You have [16:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=964s) a highly structured workflow for how you [16:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=967s) do this this sort of work. [16:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=970s) Uh and then the most common one in [16:11](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=971s) retrospect and the most obvious I [16:13](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=973s) suppose is just making sure that the [16:15](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=975s) agent has general context about what [16:18](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=978s) you're trying to achieve, but more [16:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=979s) importantly, like what's actually going [16:22](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=982s) on in the business at the time. So, in [16:24](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=984s) this case, it's an active M&A deal. You [16:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=986s) know, there were things that management [16:28](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=988s) had already started actioning. And so, [16:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=990s) in our pursuit of potential savings [16:33](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=993s) opportunities, it will come up with [16:35](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=995s) these great recommendations, but it [16:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=996s) would be shot down immediately because, [16:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=998s) hey, we already started that 3 months [16:40](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1000s) ago. So, seems obvious on its face, but [16:42](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1002s) it's actually a pretty tricky problem to [16:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1004s) try and get [16:45](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1005s) the right context and the right elements [16:48](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1008s) into Claude at the right point in the [16:50](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1010s) process such that you don't hamstring [16:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1013s) the analysis up front, but you don't [16:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1015s) come out with recommendations that are [16:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1017s) irrelevant or invalid at the end of the [16:59](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1019s) day. [17:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1020s) Um [17:02](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1022s) and so, it goes back to the point about [17:04](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1024s) knowledge accrual of just starting with [17:07](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1027s) an analysis, accruing things that you've [17:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1030s) kind of tagged along the way as, hey, [17:12](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1032s) this isn't, you know, a project that's [17:14](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1034s) in um in motion or something something [17:17](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1037s) else that's relevant to the ultimate [17:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1039s) objective of what you're trying to do. [17:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1041s) And having Claude have some sort of, [17:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1043s) whether it's a memory system or you [17:26](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1046s) embed it in your tools or something [17:27](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1047s) similar where you don't have Claude kind [17:30](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1050s) of spending tokens to rediscover the [17:32](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1052s) same problems you've already [17:33](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1053s) you've already flagged. It It's already [17:35](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1055s) just embedded in the way that it does [17:36](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1056s) its work. [17:38](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1058s) And so, I'll leave you or we will leave [17:39](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1059s) you with two main takeaways. The first [17:41](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1061s) is it's worthwhile to invest some tokens [17:44](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1064s) and effort and money upfront to [17:46](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1066s) structure your data first precisely so [17:49](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1069s) that Claude doesn't have to [17:51](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1071s) spin its wheels and spend additional [17:53](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1073s) tokens rediscovering the environment. [17:55](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1075s) So, highly structured data is very [17:57](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1077s) helpful. [17:58](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1078s) And then ultimately, of course, context [18:00](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1080s) is still king. We don't have 100 million [18:03](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1083s) token context windows. Yes, we have [18:05](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1085s) RLMs. I know Raymond in the back is a [18:08](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1088s) big fan of those. [18:10](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1090s) Um [18:11](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1091s) but that doesn't help you with the [18:13](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1093s) business context. So, putting in the [18:15](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1095s) right business context is incredibly [18:16](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1096s) important. [18:18](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1098s) Um [18:19](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1099s) and that's uh that's what we found [18:21](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1101s) during this engagement. So, [18:23](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1103s) happy to take any questions afterwards, [18:24](https://www.youtube.com/watch?v=u7fsyrXsPBE&t=1104s) but thanks so much for your time.