[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12500,
    "text": ">> Hey everybody."
  },
  {
    "startMs": 13800,
    "text": "Nice to see you here. My name is Kevin"
  },
  {
    "startMs": 15600,
    "text": "Madeira. My colleague"
  },
  {
    "startMs": 17800,
    "text": "Natalia"
  },
  {
    "startMs": 19000,
    "text": "and I are from Alex Partners,"
  },
  {
    "startMs": 21300,
    "text": "which is a global management consulting"
  },
  {
    "startMs": 22800,
    "text": "firm. We focus primarily on bankruptcy"
  },
  {
    "startMs": 25200,
    "text": "and restructuring, business and"
  },
  {
    "startMs": 27000,
    "text": "technology advisory and risk and"
  },
  {
    "startMs": 28900,
    "text": "compliance. And I'm part of the AI"
  },
  {
    "startMs": 30500,
    "text": "engineering team there."
  },
  {
    "startMs": 32500,
    "text": ">> Awesome. Thanks, Kevin. Natalia Conley,"
  },
  {
    "startMs": 34300,
    "text": "part of the AI and data team at Alex"
  },
  {
    "startMs": 36300,
    "text": "Partners. And today we'd like to share"
  },
  {
    "startMs": 38800,
    "text": "our experience with putting cloud code"
  },
  {
    "startMs": 41000,
    "text": "on a real procurement analysis with you."
  },
  {
    "startMs": 43800,
    "text": "Our talk will have three parts. So we'll"
  },
  {
    "startMs": 45100,
    "text": "talk about the business context and then"
  },
  {
    "startMs": 47200,
    "text": "our approach and then some lessons"
  },
  {
    "startMs": 49400,
    "text": "learned."
  },
  {
    "startMs": 50600,
    "text": "All right, let's get to it."
  },
  {
    "startMs": 52800,
    "text": "Okay, so some business context first."
  },
  {
    "startMs": 55500,
    "text": "Let's consider a very large healthcare"
  },
  {
    "startMs": 57600,
    "text": "company and it just acquired a smaller"
  },
  {
    "startMs": 61200,
    "text": "sort of niche player that's extremely"
  },
  {
    "startMs": 63700,
    "text": "good in one particular area."
  },
  {
    "startMs": 65900,
    "text": "And the parent company has some"
  },
  {
    "startMs": 67500,
    "text": "experience in that area, but just not"
  },
  {
    "startMs": 69200,
    "text": "that much."
  },
  {
    "startMs": 70400,
    "text": "So with that deal handed, the"
  },
  {
    "startMs": 72300,
    "text": "procurement team is hundreds of vendors"
  },
  {
    "startMs": 74700,
    "text": "across the two companies,"
  },
  {
    "startMs": 76600,
    "text": "over 40,000 distinct contract documents"
  },
  {
    "startMs": 79600,
    "text": "and multiple databases that really were"
  },
  {
    "startMs": 81700,
    "text": "not linked to one another. So there was"
  },
  {
    "startMs": 83900,
    "text": "no holistic view really of each vendor"
  },
  {
    "startMs": 86600,
    "text": "and each contract relationship."
  },
  {
    "startMs": 89500,
    "text": "Okay."
  },
  {
    "startMs": 90800,
    "text": "From here on now, let's deal with with a"
  },
  {
    "startMs": 92500,
    "text": "synthetic replica of the situation. So"
  },
  {
    "startMs": 94500,
    "text": "here it is, right? So Harbor Health is"
  },
  {
    "startMs": 96100,
    "text": "our big parent company and it acquired"
  },
  {
    "startMs": 98500,
    "text": "Summit Health Plan. That's the small"
  },
  {
    "startMs": 100100,
    "text": "regional player that's extremely good in"
  },
  {
    "startMs": 101700,
    "text": "this one area."
  },
  {
    "startMs": 103100,
    "text": "And both of them have a relationship"
  },
  {
    "startMs": 105100,
    "text": "with a particular vendor. Let's call it"
  },
  {
    "startMs": 106900,
    "text": "Acme Health Services."
  },
  {
    "startMs": 108900,
    "text": "Okay. So after the merger, what the"
  },
  {
    "startMs": 111000,
    "text": "procurement team at Harbor Health really"
  },
  {
    "startMs": 113200,
    "text": "wants is answers such as, well, can we"
  },
  {
    "startMs": 116100,
    "text": "maybe get better terms from Acme now"
  },
  {
    "startMs": 118000,
    "text": "that we've merged? And if so, uh can you"
  },
  {
    "startMs": 120792,
    "text": "maybe size up the savings? And how"
  },
  {
    "startMs": 122892,
    "text": "quickly do I have to act in order to"
  },
  {
    "startMs": 124692,
    "text": "realize those savings? Right? So that's"
  },
  {
    "startMs": 126592,
    "text": "sort of the bottom line. And our job is"
  },
  {
    "startMs": 129692,
    "text": "to make sure that that procurement lead"
  },
  {
    "startMs": 131392,
    "text": "from Harbor Health, who'll be sitting"
  },
  {
    "startMs": 133092,
    "text": "across from Acme during negotiations, is"
  },
  {
    "startMs": 135292,
    "text": "well equipped to answer any questions"
  },
  {
    "startMs": 137492,
    "text": "that they might raise, any objections,"
  },
  {
    "startMs": 139592,
    "text": "anything, right? With a specific"
  },
  {
    "startMs": 141492,
    "text": "citations, uh citation with a specific"
  },
  {
    "startMs": 144192,
    "text": "contract page or a specific invoice"
  },
  {
    "startMs": 146292,
    "text": "line. So everything needs to be traced"
  },
  {
    "startMs": 148592,
    "text": "back to the source."
  },
  {
    "startMs": 150292,
    "text": "Okay. So originally when we put Cloud"
  },
  {
    "startMs": 152092,
    "text": "Code to it, this is what it gave us,"
  },
  {
    "startMs": 155192,
    "text": "right? It came back with these answers."
  },
  {
    "startMs": 157892,
    "text": "They look complete. They're certainly"
  },
  {
    "startMs": 159692,
    "text": "well formatted. They look plausible"
  },
  {
    "startMs": 161392,
    "text": "enough."
  },
  {
    "startMs": 162492,
    "text": "You know, and uh so then we went back to"
  },
  {
    "startMs": 164992,
    "text": "the warehouse and we checked every"
  },
  {
    "startMs": 166492,
    "text": "single answer. Here's what we found out."
  },
  {
    "startMs": 169092,
    "text": "The total annual spend, the split"
  },
  {
    "startMs": 170792,
    "text": "between the two companies and the"
  },
  {
    "startMs": 172092,
    "text": "contract counts were all correct, so"
  },
  {
    "startMs": 173992,
    "text": "that was fine. However,"
  },
  {
    "startMs": 176492,
    "text": "the total contract value was wrong by"
  },
  {
    "startMs": 179392,
    "text": "approximately $800,000."
  },
  {
    "startMs": 182092,
    "text": "Right? So that's one. The second one was"
  },
  {
    "startMs": 184592,
    "text": "the the date uh by which we had to act"
  },
  {
    "startMs": 187192,
    "text": "in order to prevent the soonest expiring"
  },
  {
    "startMs": 189492,
    "text": "contract from auto-renewing was also"
  },
  {
    "startMs": 192092,
    "text": "wrong. The agent said December 1st. The"
  },
  {
    "startMs": 194792,
    "text": "actual date was November 1st, right? And"
  },
  {
    "startMs": 197492,
    "text": "so if we just kind of believe the agent"
  },
  {
    "startMs": 199392,
    "text": "and waited until December 1st, the"
  },
  {
    "startMs": 201492,
    "text": "contract would have auto-renewed and uh"
  },
  {
    "startMs": 203992,
    "text": "you know, whether we wanted it or not,"
  },
  {
    "startMs": 206092,
    "text": "right? And so that's the problem."
  },
  {
    "startMs": 208492,
    "text": "So what happened? How come Cloud Code"
  },
  {
    "startMs": 210792,
    "text": "did those things and got two answers"
  },
  {
    "startMs": 212892,
    "text": "wrong? Well, the first thing to check of"
  },
  {
    "startMs": 215092,
    "text": "course would be the data. So let's talk"
  },
  {
    "startMs": 216992,
    "text": "about the data just for a second."
  },
  {
    "startMs": 219592,
    "text": "And when you think about it, it just"
  },
  {
    "startMs": 220792,
    "text": "seems so straightforward to extract data"
  },
  {
    "startMs": 222892,
    "text": "from contracts, right? In reality"
  },
  {
    "startMs": 224592,
    "text": "though, contract data are a lot messier"
  },
  {
    "startMs": 226692,
    "text": "than than appear than they appear."
  },
  {
    "startMs": 228992,
    "text": "So just some things that can potentially"
  },
  {
    "startMs": 231392,
    "text": "go wrong with contract data."
  },
  {
    "startMs": 233792,
    "text": "For instance, you can have very long"
  },
  {
    "startMs": 235392,
    "text": "contracts and they just get kind of"
  },
  {
    "startMs": 237692,
    "text": "stuck in preprocessing and they just"
  },
  {
    "startMs": 239492,
    "text": "hang there. And maybe you catch the"
  },
  {
    "startMs": 241192,
    "text": "exception um with the timeout and and"
  },
  {
    "startMs": 243594,
    "text": "you kind of go on and you marry away,"
  },
  {
    "startMs": 245094,
    "text": "you never even know that the contract"
  },
  {
    "startMs": 246694,
    "text": "never made it into the database. That's"
  },
  {
    "startMs": 249094,
    "text": "one."
  },
  {
    "startMs": 250094,
    "text": "Another one is orphan amendments. So for"
  },
  {
    "startMs": 252694,
    "text": "instance, there's an amendment to a"
  },
  {
    "startMs": 254094,
    "text": "parent contract, but for whatever reason"
  },
  {
    "startMs": 256094,
    "text": "it's just not linked properly to its"
  },
  {
    "startMs": 257694,
    "text": "parent. So it's just kind of floating"
  },
  {
    "startMs": 259394,
    "text": "out there as an orphan. And this is"
  },
  {
    "startMs": 261294,
    "text": "actually exactly what happened in our"
  },
  {
    "startMs": 262894,
    "text": "example where Cloud Code missed the"
  },
  {
    "startMs": 265094,
    "text": "$800,000. That's because it couldn't"
  },
  {
    "startMs": 267594,
    "text": "find that orphan amendment. So that's"
  },
  {
    "startMs": 269894,
    "text": "that's another potential issue. Another"
  },
  {
    "startMs": 272394,
    "text": "one, and this is pretty common, is where"
  },
  {
    "startMs": 274394,
    "text": "you have multiple amendments restating"
  },
  {
    "startMs": 276694,
    "text": "the original contracts terms. So here"
  },
  {
    "startMs": 278994,
    "text": "you have one MSA, Master Service"
  },
  {
    "startMs": 280694,
    "text": "Agreement, and two amendments that"
  },
  {
    "startMs": 282394,
    "text": "restated the total contract value."
  },
  {
    "startMs": 284994,
    "text": "And if you just extract the structured"
  },
  {
    "startMs": 286494,
    "text": "outputs, the dollar amounts, right? Then"
  },
  {
    "startMs": 289094,
    "text": "your Cloud Code will very happily go and"
  },
  {
    "startMs": 290794,
    "text": "just sum up those dollar amounts and go,"
  },
  {
    "startMs": 292694,
    "text": "oh, the total contract value is about"
  },
  {
    "startMs": 294394,
    "text": "$17 million."
  },
  {
    "startMs": 296094,
    "text": "But if you read the actual text, not"
  },
  {
    "startMs": 298194,
    "text": "just look at the numbers, the dollar"
  },
  {
    "startMs": 299494,
    "text": "numbers, right? The text says that every"
  },
  {
    "startMs": 301694,
    "text": "amendment restates the total contract"
  },
  {
    "startMs": 303894,
    "text": "value. So the correct answer is seven,"
  },
  {
    "startMs": 305794,
    "text": "which is the most recent amendment's"
  },
  {
    "startMs": 307294,
    "text": "value. That's another sort of gotcha to"
  },
  {
    "startMs": 310494,
    "text": "watch out for. Finally, you can have"
  },
  {
    "startMs": 312294,
    "text": "situations like that where you have very"
  },
  {
    "startMs": 314194,
    "text": "long tables. Let's say these are the"
  },
  {
    "startMs": 316294,
    "text": "tables of rates for various services,"
  },
  {
    "startMs": 318294,
    "text": "right?"
  },
  {
    "startMs": 319294,
    "text": "And they go across multiple pages."
  },
  {
    "startMs": 321594,
    "text": "And so if you look at page two, it just"
  },
  {
    "startMs": 323294,
    "text": "starts out with, you know, some some"
  },
  {
    "startMs": 325094,
    "text": "numbers and there's no header. And Cloud"
  },
  {
    "startMs": 327794,
    "text": "Code might not necessarily know the"
  },
  {
    "startMs": 329194,
    "text": "context. I mean, it will just do its"
  },
  {
    "startMs": 331494,
    "text": "best to kind of guess, but it might not"
  },
  {
    "startMs": 333594,
    "text": "know what each value really means. These"
  },
  {
    "startMs": 335794,
    "text": "are just some examples. And none of this"
  },
  {
    "startMs": 337194,
    "text": "is exotic, but it just shows that real"
  },
  {
    "startMs": 339194,
    "text": "contract portfolios are full of issues"
  },
  {
    "startMs": 341694,
    "text": "that may or may not be apparent when you"
  },
  {
    "startMs": 343394,
    "text": "first started out on this on this"
  },
  {
    "startMs": 345394,
    "text": "journey."
  },
  {
    "startMs": 346694,
    "text": "And that's why Kevin and I spent quite"
  },
  {
    "startMs": 348894,
    "text": "some time really building a super robust"
  },
  {
    "startMs": 351794,
    "text": "data processing pipeline in the"
  },
  {
    "startMs": 353194,
    "text": "beginning. We spent about two weeks on"
  },
  {
    "startMs": 355494,
    "text": "it just making sure that every single"
  },
  {
    "startMs": 357494,
    "text": "little thing like that that you just saw"
  },
  {
    "startMs": 358894,
    "text": "and more,"
  },
  {
    "startMs": 360694,
    "text": "all of that stuff was accounted for. And"
  },
  {
    "startMs": 362894,
    "text": "so [laughter] it was really great. So we"
  },
  {
    "startMs": 364694,
    "text": "spent those two weeks just building that"
  },
  {
    "startMs": 366295,
    "text": "robust pipeline and then 40,000 plus"
  },
  {
    "startMs": 368395,
    "text": "contracts were processed overnight once"
  },
  {
    "startMs": 370295,
    "text": "it was done. Um I imagine if we did it"
  },
  {
    "startMs": 372895,
    "text": "by hand, it would have required probably"
  },
  {
    "startMs": 374495,
    "text": "a much larger team than just us two and"
  },
  {
    "startMs": 377295,
    "text": "probably a little bit more time than"
  },
  {
    "startMs": 378495,
    "text": "that."
  },
  {
    "startMs": 379595,
    "text": "But just a summary of the pipeline, it"
  },
  {
    "startMs": 381495,
    "text": "was a five stage pipeline. The first one"
  },
  {
    "startMs": 383395,
    "text": "was mostly preprocessing, so document"
  },
  {
    "startMs": 385195,
    "text": "intelligence, OCR, that kind of stuff."
  },
  {
    "startMs": 387595,
    "text": "The second step was classification. So"
  },
  {
    "startMs": 389795,
    "text": "what is each contract? Is it a master"
  },
  {
    "startMs": 391795,
    "text": "service agreement? Is it an amendment?"
  },
  {
    "startMs": 393595,
    "text": "Is it"
  },
  {
    "startMs": 394695,
    "text": "an NDA? Something like that. Then there"
  },
  {
    "startMs": 396995,
    "text": "was the terms rates and spend"
  },
  {
    "startMs": 398495,
    "text": "extraction. That's the next step. And"
  },
  {
    "startMs": 400095,
    "text": "this is where we would account for"
  },
  {
    "startMs": 401195,
    "text": "things like very long tables that could"
  },
  {
    "startMs": 402895,
    "text": "break across page boundaries."
  },
  {
    "startMs": 405495,
    "text": "And then we would map all the services"
  },
  {
    "startMs": 407295,
    "text": "to spend categories. Only then would"
  },
  {
    "startMs": 409495,
    "text": "Claude code go over the extracted data"
  },
  {
    "startMs": 412595,
    "text": "and and sort of do its thing. It was"
  },
  {
    "startMs": 414395,
    "text": "very important to have those structured"
  },
  {
    "startMs": 416295,
    "text": "tables, not only because it helped out"
  },
  {
    "startMs": 418095,
    "text": "Claude code, but also because it made it"
  },
  {
    "startMs": 420695,
    "text": "that much easier for for us later on to"
  },
  {
    "startMs": 423495,
    "text": "go and check every answer"
  },
  {
    "startMs": 424695,
    "text": "programmatically in code."
  },
  {
    "startMs": 426795,
    "text": "All right, so Kevin."
  },
  {
    "startMs": 428395,
    "text": ">> Sure, thanks Natalia. And so obviously"
  },
  {
    "startMs": 431695,
    "text": "contract data is very text heavy, but"
  },
  {
    "startMs": 434095,
    "text": "not always. And so there's sometimes"
  },
  {
    "startMs": 435995,
    "text": "there's multimodal elements. There might"
  },
  {
    "startMs": 437595,
    "text": "be charts or graphs or other things that"
  },
  {
    "startMs": 439895,
    "text": "if you just run it through a naive OCR"
  },
  {
    "startMs": 442095,
    "text": "process, you might get what you see on"
  },
  {
    "startMs": 444495,
    "text": "the right in terms of the the axes, but"
  },
  {
    "startMs": 446595,
    "text": "you actually lose the information that's"
  },
  {
    "startMs": 448195,
    "text": "contained within the chart."
  },
  {
    "startMs": 449895,
    "text": "And so a lot of what we're talking about"
  },
  {
    "startMs": 451995,
    "text": "now, if you're bitter lesson pill, this"
  },
  {
    "startMs": 453895,
    "text": "will likely not be relevant maybe like"
  },
  {
    "startMs": 455895,
    "text": "six to 12 months, we'll see. But what we"
  },
  {
    "startMs": 457995,
    "text": "did here to solve this specific problem"
  },
  {
    "startMs": 460295,
    "text": "was take the contract pages as images"
  },
  {
    "startMs": 463495,
    "text": "and feed them through using DS Pi to"
  },
  {
    "startMs": 466495,
    "text": "arrive at a highly structured output,"
  },
  {
    "startMs": 468595,
    "text": "which is what you see on the bottom"
  },
  {
    "startMs": 469795,
    "text": "right there. So you go from this large"
  },
  {
    "startMs": 473095,
    "text": "unstructured data set to basically a"
  },
  {
    "startMs": 475595,
    "text": "database of elements that you know are"
  },
  {
    "startMs": 477795,
    "text": "going to be relevant for the balance of"
  },
  {
    "startMs": 480495,
    "text": "your work. And so it's very tempting to"
  },
  {
    "startMs": 483095,
    "text": "just let Claude rip, you know, on on the"
  },
  {
    "startMs": 486295,
    "text": "unstructured data"
  },
  {
    "startMs": 487795,
    "text": "and kind of have it figure it out, which"
  },
  {
    "startMs": 489197,
    "text": "is"
  },
  {
    "startMs": 490197,
    "text": "which is a valid approach, but there's"
  },
  {
    "startMs": 492797,
    "text": "certain engagements like this one where"
  },
  {
    "startMs": 494697,
    "text": "we know upfront we want rate tiers, we"
  },
  {
    "startMs": 497297,
    "text": "know we want rebates, we we know we want"
  },
  {
    "startMs": 499097,
    "text": "certain data elements. So you structure"
  },
  {
    "startMs": 501197,
    "text": "some of that upfront and then let Claude"
  },
  {
    "startMs": 503197,
    "text": "fill in that framework behind it."
  },
  {
    "startMs": 506797,
    "text": "And so once you have everything in this"
  },
  {
    "startMs": 508297,
    "text": "highly structured format,"
  },
  {
    "startMs": 510097,
    "text": "you can then use Claude code to"
  },
  {
    "startMs": 511997,
    "text": "dynamically build tools for itself. And"
  },
  {
    "startMs": 514297,
    "text": "so you have common ones, of course, like"
  },
  {
    "startMs": 516697,
    "text": "execute SQL or list tables or things"
  },
  {
    "startMs": 518997,
    "text": "like that. But we found that it was it"
  },
  {
    "startMs": 521297,
    "text": "was really helpful for later"
  },
  {
    "startMs": 524497,
    "text": "implementations or later agents when"
  },
  {
    "startMs": 526297,
    "text": "they're coming in fresh to to the data"
  },
  {
    "startMs": 529097,
    "text": "to have some of these tools available to"
  },
  {
    "startMs": 530697,
    "text": "them. So invest a lot of time upfront to"
  },
  {
    "startMs": 533897,
    "text": "structure the data, as I mentioned"
  },
  {
    "startMs": 535897,
    "text": "previously, where you you essentially"
  },
  {
    "startMs": 538197,
    "text": "create these derivative databases off of"
  },
  {
    "startMs": 540797,
    "text": "contract data or other spend data or"
  },
  {
    "startMs": 542897,
    "text": "invoices. You normalize all of that"
  },
  {
    "startMs": 544797,
    "text": "together, and then you have Claude build"
  },
  {
    "startMs": 547097,
    "text": "these custom tools so that it has you"
  },
  {
    "startMs": 549097,
    "text": "know, kind of a shortcut, if you will,"
  },
  {
    "startMs": 550997,
    "text": "to get certain information that are"
  },
  {
    "startMs": 552397,
    "text": "specific, in this case, to this deal. Um"
  },
  {
    "startMs": 555997,
    "text": "and so you have things like get rate"
  },
  {
    "startMs": 557097,
    "text": "card page or get specific vendor spend."
  },
  {
    "startMs": 559697,
    "text": "And that's where you can embed some of"
  },
  {
    "startMs": 561197,
    "text": "the business logic where you might know"
  },
  {
    "startMs": 563497,
    "text": "of some gotchas or some context that"
  },
  {
    "startMs": 566197,
    "text": "isn't immediately apparent or obvious in"
  },
  {
    "startMs": 568497,
    "text": "the contract data. You can start to"
  },
  {
    "startMs": 570197,
    "text": "embed some of that in the tool calls"
  },
  {
    "startMs": 572097,
    "text": "themselves, the way that it queries data"
  },
  {
    "startMs": 573997,
    "text": "or pulls things back um is very helpful"
  },
  {
    "startMs": 577197,
    "text": "to enforce some of those, you know, more"
  },
  {
    "startMs": 578897,
    "text": "custom pieces of logic."
  },
  {
    "startMs": 581697,
    "text": "Um and Claude code was perfect for"
  },
  {
    "startMs": 583097,
    "text": "something like this because it's used to"
  },
  {
    "startMs": 585797,
    "text": "navigating autonomously around kind of"
  },
  {
    "startMs": 587797,
    "text": "messy code bases. This is a a similar"
  },
  {
    "startMs": 590597,
    "text": "concept. It's just contract data and"
  },
  {
    "startMs": 592497,
    "text": "spend data. So it's all largely in"
  },
  {
    "startMs": 594597,
    "text": "tabular format. Um but it can"
  },
  {
    "startMs": 596897,
    "text": "dynamically explore and pull in the the"
  },
  {
    "startMs": 598897,
    "text": "different elements that I mentioned"
  },
  {
    "startMs": 600097,
    "text": "previously."
  },
  {
    "startMs": 601297,
    "text": "Um and then once we found something that"
  },
  {
    "startMs": 603597,
    "text": "was useful, that would work, we would"
  },
  {
    "startMs": 605097,
    "text": "package that up either into something"
  },
  {
    "startMs": 607597,
    "text": "specific in the prompt, but I think we"
  },
  {
    "startMs": 609497,
    "text": "we lean more on these custom tools and"
  },
  {
    "startMs": 612398,
    "text": "custom views within the database to"
  },
  {
    "startMs": 614598,
    "text": "enforce some of that business logic in"
  },
  {
    "startMs": 616898,
    "text": "code versus just relying on on a prompt."
  },
  {
    "startMs": 620198,
    "text": "Um and once you have all that package"
  },
  {
    "startMs": 621598,
    "text": "together, you can start to develop"
  },
  {
    "startMs": 623298,
    "text": "custom MCP servers and and"
  },
  {
    "startMs": 625298,
    "text": "configurations that live on. And it's"
  },
  {
    "startMs": 628298,
    "text": "it's a it's a bunch of different"
  },
  {
    "startMs": 629698,
    "text": "elements uh that really get at how do"
  },
  {
    "startMs": 632598,
    "text": "you perpetuate and and accrue knowledge"
  },
  {
    "startMs": 634698,
    "text": "across the the life cycle, in this case"
  },
  {
    "startMs": 637198,
    "text": "for particular engagement. But if you're"
  },
  {
    "startMs": 639098,
    "text": "working on something internally, you"
  },
  {
    "startMs": 641098,
    "text": "right, you want to accrue the the"
  },
  {
    "startMs": 643498,
    "text": "knowledge and accrue certain business"
  },
  {
    "startMs": 645098,
    "text": "rules um that you want to enforce later"
  },
  {
    "startMs": 647798,
    "text": "down the line. So that this is a nice"
  },
  {
    "startMs": 649798,
    "text": "way to do it."
  },
  {
    "startMs": 651698,
    "text": "And then of course, you have the ability"
  },
  {
    "startMs": 652898,
    "text": "to better govern access to the"
  },
  {
    "startMs": 655398,
    "text": "underlying data so you can block certain"
  },
  {
    "startMs": 657498,
    "text": "calls. There was a great talk right"
  },
  {
    "startMs": 659198,
    "text": "before this um regarding the security of"
  },
  {
    "startMs": 662298,
    "text": "how agents interact with the rest of the"
  },
  {
    "startMs": 663798,
    "text": "environment. It's actually pretty"
  },
  {
    "startMs": 665498,
    "text": "simple, I think, at the end of the day."
  },
  {
    "startMs": 667198,
    "text": "You just have a defined set of"
  },
  {
    "startMs": 668998,
    "text": "operations that you're comfortable with"
  },
  {
    "startMs": 671098,
    "text": "and you block everything else and you"
  },
  {
    "startMs": 672598,
    "text": "can enforce that up front."
  },
  {
    "startMs": 674898,
    "text": "So we'll walk through just a a couple of"
  },
  {
    "startMs": 676498,
    "text": "lessons here uh very quickly. Natalia"
  },
  {
    "startMs": 678998,
    "text": "mentioned previously about the agent"
  },
  {
    "startMs": 680898,
    "text": "using the wrong date. And so it's a good"
  },
  {
    "startMs": 683598,
    "text": "example of again, some external business"
  },
  {
    "startMs": 686598,
    "text": "context that is helpful and useful to"
  },
  {
    "startMs": 688898,
    "text": "embed into the agent up front because in"
  },
  {
    "startMs": 692398,
    "text": "this case, yes, it had pulled out the"
  },
  {
    "startMs": 694598,
    "text": "date from the termination clause, but"
  },
  {
    "startMs": 696698,
    "text": "actually the question we were asking was"
  },
  {
    "startMs": 698598,
    "text": "supposed to use the date from calculated"
  },
  {
    "startMs": 701098,
    "text": "from the renewal clause. And so it it"
  },
  {
    "startMs": 703898,
    "text": "seems like a minor thing, but it's it's"
  },
  {
    "startMs": 706498,
    "text": "uh it's a lot of these nuanced issues"
  },
  {
    "startMs": 708598,
    "text": "which compound your error rates. And so"
  },
  {
    "startMs": 711198,
    "text": "the more you can build these sorts of"
  },
  {
    "startMs": 713098,
    "text": "rules in up front, the better"
  },
  {
    "startMs": 715298,
    "text": "uh the result ultimately of course will"
  },
  {
    "startMs": 717098,
    "text": "be."
  },
  {
    "startMs": 719098,
    "text": "Uh Natalia mentioned also the kind of"
  },
  {
    "startMs": 721198,
    "text": "the dangling references for different"
  },
  {
    "startMs": 723098,
    "text": "amendments."
  },
  {
    "startMs": 724398,
    "text": "Um and so this is something that we run"
  },
  {
    "startMs": 726898,
    "text": "across a lot. So many organizations have"
  },
  {
    "startMs": 730398,
    "text": "a contract lifecycle management system,"
  },
  {
    "startMs": 732298,
    "text": "a CLM."
  },
  {
    "startMs": 733399,
    "text": "It's intended to be uh a highly"
  },
  {
    "startMs": 735499,
    "text": "structured database of your contract"
  },
  {
    "startMs": 737299,
    "text": "data, right, the vendor name and ex uh"
  },
  {
    "startMs": 740099,
    "text": "extraction of termination dates and"
  },
  {
    "startMs": 741899,
    "text": "whatnot. They're almost never maintained"
  },
  {
    "startMs": 744799,
    "text": "properly or well."
  },
  {
    "startMs": 746499,
    "text": "And so we often find we have to of"
  },
  {
    "startMs": 748099,
    "text": "course use the data that's available to"
  },
  {
    "startMs": 749799,
    "text": "us, but take somewhat of a bottoms-up"
  },
  {
    "startMs": 751799,
    "text": "approach to actually have Claude read"
  },
  {
    "startMs": 753899,
    "text": "through the contents of the contracts to"
  },
  {
    "startMs": 756499,
    "text": "find"
  },
  {
    "startMs": 757799,
    "text": "um instances of where it might not be"
  },
  {
    "startMs": 760399,
    "text": "properly captured in the CLM. And so you"
  },
  {
    "startMs": 762999,
    "text": "you have to go through and have Claude"
  },
  {
    "startMs": 764399,
    "text": "kind of build out this hierarchy of"
  },
  {
    "startMs": 765999,
    "text": "relationships. So you have"
  },
  {
    "startMs": 767999,
    "text": "an MSA that is tied to SOWs and then"
  },
  {
    "startMs": 771199,
    "text": "there are multiple amendments to those"
  },
  {
    "startMs": 773699,
    "text": "uh to those SOWs, and sometimes they're"
  },
  {
    "startMs": 776399,
    "text": "just not organized properly. And so"
  },
  {
    "startMs": 778999,
    "text": "again, structuring everything up front"
  },
  {
    "startMs": 780599,
    "text": "makes it really easy to tack on these"
  },
  {
    "startMs": 783499,
    "text": "kind of loose documents later on, which"
  },
  {
    "startMs": 785999,
    "text": "has a meaningful impact if you're"
  },
  {
    "startMs": 787199,
    "text": "looking for savings opportunities. In"
  },
  {
    "startMs": 789699,
    "text": "this case it was a an M&A, you know,"
  },
  {
    "startMs": 791499,
    "text": "consolidation. So you don't want to"
  },
  {
    "startMs": 793399,
    "text": "double count something, but you also"
  },
  {
    "startMs": 794799,
    "text": "want to capture the latest kind of state"
  },
  {
    "startMs": 796999,
    "text": "of the relationship with the vendor,"
  },
  {
    "startMs": 799199,
    "text": "which this kind of process allows you to"
  },
  {
    "startMs": 801199,
    "text": "do. So Claude was very good at at doing"
  },
  {
    "startMs": 803699,
    "text": "that."
  },
  {
    "startMs": 805399,
    "text": "Uh I made a point before about governed"
  },
  {
    "startMs": 807399,
    "text": "access to the data. This can come back"
  },
  {
    "startMs": 809899,
    "text": "to bite you as well because there were"
  },
  {
    "startMs": 812099,
    "text": "two separate entities that were being uh"
  },
  {
    "startMs": 814899,
    "text": "put together here. The way that we"
  },
  {
    "startMs": 816499,
    "text": "accessed data from entity A versus"
  },
  {
    "startMs": 819699,
    "text": "entity B was different."
  },
  {
    "startMs": 821699,
    "text": "And so we ran into a few instances where"
  },
  {
    "startMs": 824599,
    "text": "the returned value would have been"
  },
  {
    "startMs": 826099,
    "text": "something that was malformed or zero or"
  },
  {
    "startMs": 828399,
    "text": "there was some type of access"
  },
  {
    "startMs": 829499,
    "text": "permission. And Claude when making the"
  },
  {
    "startMs": 832399,
    "text": "tool call didn't really get the the"
  },
  {
    "startMs": 834599,
    "text": "right type of feedback, and so it kind"
  },
  {
    "startMs": 836299,
    "text": "of just went forward thinking that maybe"
  },
  {
    "startMs": 838499,
    "text": "an invoice was zero or or, you know,"
  },
  {
    "startMs": 840899,
    "text": "basically an incorrect value."
  },
  {
    "startMs": 843099,
    "text": "And so a lesson learned there for us was"
  },
  {
    "startMs": 845299,
    "text": "anytime there's some sort of"
  },
  {
    "startMs": 847499,
    "text": "um you know, governance failure or"
  },
  {
    "startMs": 849299,
    "text": "access denied error or something like"
  },
  {
    "startMs": 850899,
    "text": "that, giving Claude more feedback about"
  },
  {
    "startMs": 854099,
    "text": "what went wrong allows it fail"
  },
  {
    "startMs": 855701,
    "text": "gracefully or otherwise recover, maybe"
  },
  {
    "startMs": 858101,
    "text": "flag something to a human or say, okay,"
  },
  {
    "startMs": 860301,
    "text": "I know this information is actually in"
  },
  {
    "startMs": 862401,
    "text": "this other authoritative database I can"
  },
  {
    "startMs": 864301,
    "text": "go to instead."
  },
  {
    "startMs": 866001,
    "text": "Um"
  },
  {
    "startMs": 866901,
    "text": "because when you're relying on it for"
  },
  {
    "startMs": 868501,
    "text": "some sort of savings analysis or"
  },
  {
    "startMs": 870301,
    "text": "something similar,"
  },
  {
    "startMs": 871801,
    "text": "uh you want Claude to have as much of"
  },
  {
    "startMs": 873601,
    "text": "that context of course as possible."
  },
  {
    "startMs": 877201,
    "text": "Another thing we did um was"
  },
  {
    "startMs": 881101,
    "text": "uh dispatch agents for different savings"
  },
  {
    "startMs": 883301,
    "text": "levers. So our colleagues who have been"
  },
  {
    "startMs": 885001,
    "text": "doing this for 20 years or whatever it"
  },
  {
    "startMs": 887101,
    "text": "is, they have a whole library of ways to"
  },
  {
    "startMs": 889901,
    "text": "identify savings. Some are more obvious"
  },
  {
    "startMs": 892101,
    "text": "than others, but"
  },
  {
    "startMs": 893501,
    "text": "okay, you're putting two companies"
  },
  {
    "startMs": 894501,
    "text": "together. Maybe they share vendors,"
  },
  {
    "startMs": 896901,
    "text": "maybe different vendors but same"
  },
  {
    "startMs": 898701,
    "text": "services. So you have a whole library of"
  },
  {
    "startMs": 901101,
    "text": "different activities and analyses that"
  },
  {
    "startMs": 903401,
    "text": "you could do."
  },
  {
    "startMs": 904801,
    "text": "So we dispatched agents to do each of"
  },
  {
    "startMs": 907101,
    "text": "those in parallel um and they came back"
  },
  {
    "startMs": 910301,
    "text": "uh with actually pretty good results,"
  },
  {
    "startMs": 912401,
    "text": "but there's of course this step"
  },
  {
    "startMs": 913901,
    "text": "afterwards where you need to deduplicate"
  },
  {
    "startMs": 916201,
    "text": "the findings that come back. So"
  },
  {
    "startMs": 918701,
    "text": "um you can't ultimately double count in"
  },
  {
    "startMs": 920801,
    "text": "this case a potential savings if"
  },
  {
    "startMs": 923701,
    "text": "um you know if it's coming from the same"
  },
  {
    "startMs": 926301,
    "text": "source or the same vendor as an example."
  },
  {
    "startMs": 929601,
    "text": "Um and so the way that we did this was"
  },
  {
    "startMs": 932501,
    "text": "um essentially build out an additional"
  },
  {
    "startMs": 934301,
    "text": "process and a and a"
  },
  {
    "startMs": 936001,
    "text": "very specific and precise"
  },
  {
    "startMs": 938801,
    "text": "uh set of workflows where you would kind"
  },
  {
    "startMs": 941101,
    "text": "of let Claude do its thing, do its"
  },
  {
    "startMs": 942601,
    "text": "analysis, but then the reconciliation"
  },
  {
    "startMs": 944501,
    "text": "step is very"
  },
  {
    "startMs": 946301,
    "text": "uh"
  },
  {
    "startMs": 947601,
    "text": "it's built around kind of DSPI"
  },
  {
    "startMs": 949101,
    "text": "ultimately where you're very"
  },
  {
    "startMs": 950201,
    "text": "prescriptive about which data points"
  },
  {
    "startMs": 952501,
    "text": "you're evaluating, how you do the"
  },
  {
    "startMs": 954001,
    "text": "reconciliation, and what's an acceptable"
  },
  {
    "startMs": 956401,
    "text": "next step in the process. There was"
  },
  {
    "startMs": 958701,
    "text": "another pretty good talk this morning"
  },
  {
    "startMs": 960001,
    "text": "about uh using DAGs for for policy"
  },
  {
    "startMs": 962601,
    "text": "execution. It's the same idea. You have"
  },
  {
    "startMs": 964801,
    "text": "a highly structured workflow for how you"
  },
  {
    "startMs": 967101,
    "text": "do this this sort of work."
  },
  {
    "startMs": 970201,
    "text": "Uh and then the most common one in"
  },
  {
    "startMs": 971501,
    "text": "retrospect and the most obvious I"
  },
  {
    "startMs": 973101,
    "text": "suppose is just making sure that the"
  },
  {
    "startMs": 975401,
    "text": "agent has general context about what"
  },
  {
    "startMs": 978001,
    "text": "you're trying to achieve, but more"
  },
  {
    "startMs": 979902,
    "text": "importantly, like what's actually going"
  },
  {
    "startMs": 982302,
    "text": "on in the business at the time. So, in"
  },
  {
    "startMs": 984102,
    "text": "this case, it's an active M&A deal. You"
  },
  {
    "startMs": 986502,
    "text": "know, there were things that management"
  },
  {
    "startMs": 988102,
    "text": "had already started actioning. And so,"
  },
  {
    "startMs": 990702,
    "text": "in our pursuit of potential savings"
  },
  {
    "startMs": 993102,
    "text": "opportunities, it will come up with"
  },
  {
    "startMs": 995002,
    "text": "these great recommendations, but it"
  },
  {
    "startMs": 996902,
    "text": "would be shot down immediately because,"
  },
  {
    "startMs": 998802,
    "text": "hey, we already started that 3 months"
  },
  {
    "startMs": 1000302,
    "text": "ago. So, seems obvious on its face, but"
  },
  {
    "startMs": 1002602,
    "text": "it's actually a pretty tricky problem to"
  },
  {
    "startMs": 1004302,
    "text": "try and get"
  },
  {
    "startMs": 1005502,
    "text": "the right context and the right elements"
  },
  {
    "startMs": 1008802,
    "text": "into Claude at the right point in the"
  },
  {
    "startMs": 1010702,
    "text": "process such that you don't hamstring"
  },
  {
    "startMs": 1013102,
    "text": "the analysis up front, but you don't"
  },
  {
    "startMs": 1015202,
    "text": "come out with recommendations that are"
  },
  {
    "startMs": 1017502,
    "text": "irrelevant or invalid at the end of the"
  },
  {
    "startMs": 1019402,
    "text": "day."
  },
  {
    "startMs": 1020802,
    "text": "Um"
  },
  {
    "startMs": 1022502,
    "text": "and so, it goes back to the point about"
  },
  {
    "startMs": 1024802,
    "text": "knowledge accrual of just starting with"
  },
  {
    "startMs": 1027602,
    "text": "an analysis, accruing things that you've"
  },
  {
    "startMs": 1030302,
    "text": "kind of tagged along the way as, hey,"
  },
  {
    "startMs": 1032402,
    "text": "this isn't, you know, a project that's"
  },
  {
    "startMs": 1034402,
    "text": "in um in motion or something something"
  },
  {
    "startMs": 1037502,
    "text": "else that's relevant to the ultimate"
  },
  {
    "startMs": 1039802,
    "text": "objective of what you're trying to do."
  },
  {
    "startMs": 1041602,
    "text": "And having Claude have some sort of,"
  },
  {
    "startMs": 1043702,
    "text": "whether it's a memory system or you"
  },
  {
    "startMs": 1046302,
    "text": "embed it in your tools or something"
  },
  {
    "startMs": 1047802,
    "text": "similar where you don't have Claude kind"
  },
  {
    "startMs": 1050002,
    "text": "of spending tokens to rediscover the"
  },
  {
    "startMs": 1052102,
    "text": "same problems you've already"
  },
  {
    "startMs": 1053902,
    "text": "you've already flagged. It It's already"
  },
  {
    "startMs": 1055302,
    "text": "just embedded in the way that it does"
  },
  {
    "startMs": 1056702,
    "text": "its work."
  },
  {
    "startMs": 1058102,
    "text": "And so, I'll leave you or we will leave"
  },
  {
    "startMs": 1059402,
    "text": "you with two main takeaways. The first"
  },
  {
    "startMs": 1061802,
    "text": "is it's worthwhile to invest some tokens"
  },
  {
    "startMs": 1064902,
    "text": "and effort and money upfront to"
  },
  {
    "startMs": 1066502,
    "text": "structure your data first precisely so"
  },
  {
    "startMs": 1069302,
    "text": "that Claude doesn't have to"
  },
  {
    "startMs": 1071302,
    "text": "spin its wheels and spend additional"
  },
  {
    "startMs": 1073002,
    "text": "tokens rediscovering the environment."
  },
  {
    "startMs": 1075402,
    "text": "So, highly structured data is very"
  },
  {
    "startMs": 1077502,
    "text": "helpful."
  },
  {
    "startMs": 1078602,
    "text": "And then ultimately, of course, context"
  },
  {
    "startMs": 1080602,
    "text": "is still king. We don't have 100 million"
  },
  {
    "startMs": 1083502,
    "text": "token context windows. Yes, we have"
  },
  {
    "startMs": 1085902,
    "text": "RLMs. I know Raymond in the back is a"
  },
  {
    "startMs": 1088702,
    "text": "big fan of those."
  },
  {
    "startMs": 1090302,
    "text": "Um"
  },
  {
    "startMs": 1091202,
    "text": "but that doesn't help you with the"
  },
  {
    "startMs": 1093302,
    "text": "business context. So, putting in the"
  },
  {
    "startMs": 1095102,
    "text": "right business context is incredibly"
  },
  {
    "startMs": 1096902,
    "text": "important."
  },
  {
    "startMs": 1098102,
    "text": "Um"
  },
  {
    "startMs": 1099102,
    "text": "and that's uh that's what we found"
  },
  {
    "startMs": 1101303,
    "text": "during this engagement. So,"
  },
  {
    "startMs": 1103103,
    "text": "happy to take any questions afterwards,"
  },
  {
    "startMs": 1104603,
    "text": "but thanks so much for your time."
  }
]