[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12520,
    "text": ">> Today I'm going to talk about building"
  },
  {
    "startMs": 14280,
    "text": "better agents through end-to-end evals."
  },
  {
    "startMs": 16360,
    "text": "I'm going to talk about one of Ramp's"
  },
  {
    "startMs": 17360,
    "text": "products, Stacks, a a stack R A I agent."
  },
  {
    "startMs": 22280,
    "text": "Uh so"
  },
  {
    "startMs": 23360,
    "text": "this is a short talk briefly about me,"
  },
  {
    "startMs": 25400,
    "text": "about Ramp. Ramp, 70,000 plus"
  },
  {
    "startMs": 27640,
    "text": "businesses."
  },
  {
    "startMs": 29040,
    "text": "Um we have card, travel, AP, AR. We're"
  },
  {
    "startMs": 32520,
    "text": "an expense management platform. I lead"
  },
  {
    "startMs": 34320,
    "text": "our applied sciences team. These are AI"
  },
  {
    "startMs": 36400,
    "text": "engineers, MLEs, product data"
  },
  {
    "startMs": 38240,
    "text": "scientists, formerly of Meta, uh and I"
  },
  {
    "startMs": 41840,
    "text": "have a PhD in economics. I care a lot"
  },
  {
    "startMs": 43480,
    "text": "about measurement and evaluation, which"
  },
  {
    "startMs": 45120,
    "text": "is what we're going to talk about today."
  },
  {
    "startMs": 47440,
    "text": "So,"
  },
  {
    "startMs": 48280,
    "text": "three things."
  },
  {
    "startMs": 50000,
    "text": "What's your agent's goal?"
  },
  {
    "startMs": 51960,
    "text": "Design those evals, and then how did we"
  },
  {
    "startMs": 54280,
    "text": "get uh value from the evals that we"
  },
  {
    "startMs": 56200,
    "text": "designed?"
  },
  {
    "startMs": 58640,
    "text": "Uh"
  },
  {
    "startMs": 60000,
    "text": "so I'm particularly opinionated about"
  },
  {
    "startMs": 62080,
    "text": "this, that people try to identify a"
  },
  {
    "startMs": 64480,
    "text": "process that they don't really"
  },
  {
    "startMs": 65720,
    "text": "understand."
  },
  {
    "startMs": 67040,
    "text": "And the first task in uh building an"
  },
  {
    "startMs": 70680,
    "text": "agent for a process is breaking down the"
  },
  {
    "startMs": 72800,
    "text": "job, the tasks, and what success looks"
  },
  {
    "startMs": 74920,
    "text": "like."
  },
  {
    "startMs": 76280,
    "text": "So, this is what a bookkeeper's job"
  },
  {
    "startMs": 78640,
    "text": "tends to look like."
  },
  {
    "startMs": 80040,
    "text": "So, they want to close the books. We're"
  },
  {
    "startMs": 81480,
    "text": "not going to do a ton of accounting"
  },
  {
    "startMs": 82880,
    "text": "today, but generally they have a set of"
  },
  {
    "startMs": 85040,
    "text": "tasks that they're trying to accomplish."
  },
  {
    "startMs": 87800,
    "text": "When we're building our benchmark, uh we"
  },
  {
    "startMs": 90680,
    "text": "created a set of categories. These are"
  },
  {
    "startMs": 92760,
    "text": "three of them. There's going to be a"
  },
  {
    "startMs": 93720,
    "text": "fourth that you're going to see a little"
  },
  {
    "startMs": 94800,
    "text": "later. Um and then we thought about what"
  },
  {
    "startMs": 97280,
    "text": "success looks like. Um once we knew the"
  },
  {
    "startMs": 99680,
    "text": "tasks, once we knew success,"
  },
  {
    "startMs": 101880,
    "text": "we sat down with bookkeepers, we sat"
  },
  {
    "startMs": 104720,
    "text": "down with accounting firms, and then we"
  },
  {
    "startMs": 106840,
    "text": "wanted to understand exactly the full"
  },
  {
    "startMs": 109040,
    "text": "end-to-end process to solve a specific"
  },
  {
    "startMs": 111200,
    "text": "task. This is an example of a"
  },
  {
    "startMs": 113000,
    "text": "reconciliation."
  },
  {
    "startMs": 114640,
    "text": "A reconciliation is actually a pretty"
  },
  {
    "startMs": 116040,
    "text": "simple exercise. You have on the"
  },
  {
    "startMs": 118680,
    "text": "left-hand side, you have an ERP uh that"
  },
  {
    "startMs": 121612,
    "text": "represents the accounts payable ledger."
  },
  {
    "startMs": 125092,
    "text": "Those are bills that you have to pay. If"
  },
  {
    "startMs": 127132,
    "text": "you don't know the term accounts"
  },
  {
    "startMs": 128092,
    "text": "payable, I didn't until a few years ago"
  },
  {
    "startMs": 129812,
    "text": "either."
  },
  {
    "startMs": 131052,
    "text": "And then on the right, you have the"
  },
  {
    "startMs": 132292,
    "text": "system that has the bills. In this case,"
  },
  {
    "startMs": 134172,
    "text": "you can pay your bills with Ramp. You"
  },
  {
    "startMs": 135972,
    "text": "need these two systems to have the same"
  },
  {
    "startMs": 137532,
    "text": "numbers. It's a pretty simple exercise."
  },
  {
    "startMs": 140212,
    "text": "What does an accountant do? Well, they"
  },
  {
    "startMs": 142612,
    "text": "go into Excel."
  },
  {
    "startMs": 143972,
    "text": "They bring in the data for the ERP row"
  },
  {
    "startMs": 146812,
    "text": "by row. They bring in the data from the"
  },
  {
    "startMs": 149652,
    "text": "system that is accounts payable."
  },
  {
    "startMs": 151812,
    "text": "Do they reconcile? Do they tie? Yes, no."
  },
  {
    "startMs": 154652,
    "text": "And then they figure out why they don't,"
  },
  {
    "startMs": 156292,
    "text": "and then they make them tie."
  },
  {
    "startMs": 158212,
    "text": "Pretty easy to identify this process if"
  },
  {
    "startMs": 161092,
    "text": "you understand exactly what it is."
  },
  {
    "startMs": 163732,
    "text": "So, we want to build a benchmark."
  },
  {
    "startMs": 166772,
    "text": "Uh I'm going to interchangeably use"
  },
  {
    "startMs": 169212,
    "text": "terms like benchmarks, evals,"
  },
  {
    "startMs": 171612,
    "text": "verifiability,"
  },
  {
    "startMs": 173572,
    "text": "because frankly, I think it's a soup of"
  },
  {
    "startMs": 176252,
    "text": "language"
  },
  {
    "startMs": 177532,
    "text": "that are all trying to do the same"
  },
  {
    "startMs": 178972,
    "text": "thing. These are the four sets of"
  },
  {
    "startMs": 181132,
    "text": "categories of things"
  },
  {
    "startMs": 183452,
    "text": "that we wanted to eval. There's"
  },
  {
    "startMs": 186412,
    "text": "reconciliation, what I just described,"
  },
  {
    "startMs": 188452,
    "text": "data entry, variance analysis, it's why"
  },
  {
    "startMs": 191052,
    "text": "do numbers move,"
  },
  {
    "startMs": 192732,
    "text": "schedules and accruals, not going to go"
  },
  {
    "startMs": 194092,
    "text": "deep into there,"
  },
  {
    "startMs": 195812,
    "text": "but it is a complex thing that an"
  },
  {
    "startMs": 197932,
    "text": "accountant is doing."
  },
  {
    "startMs": 200212,
    "text": "So, this is just like a very high-level"
  },
  {
    "startMs": 204212,
    "text": "schematized version"
  },
  {
    "startMs": 206772,
    "text": "of how to think about the process of"
  },
  {
    "startMs": 208932,
    "text": "building evaluations."
  },
  {
    "startMs": 210932,
    "text": "You can think about the state before you"
  },
  {
    "startMs": 212252,
    "text": "have users or at least enough users that"
  },
  {
    "startMs": 214052,
    "text": "you think you have a good distribution"
  },
  {
    "startMs": 215332,
    "text": "of data. And then with users. So, before"
  },
  {
    "startMs": 219172,
    "text": "you have users, remember all agents are"
  },
  {
    "startMs": 221372,
    "text": "doing is they're calling tools that hit"
  },
  {
    "startMs": 223212,
    "text": "a database. That's it. You don't have"
  },
  {
    "startMs": 225892,
    "text": "users. You need simulation. You need"
  },
  {
    "startMs": 229332,
    "text": "synthetic data."
  },
  {
    "startMs": 231012,
    "text": "I would highly recommend anytime you're"
  },
  {
    "startMs": 233492,
    "text": "building an agent for an area where you"
  },
  {
    "startMs": 235292,
    "text": "don't have enough users, you should"
  },
  {
    "startMs": 238532,
    "text": "build synthetic data. Um"
  },
  {
    "startMs": 241292,
    "text": "we built this benchmark in partnership"
  },
  {
    "startMs": 244134,
    "text": "with a company called Mercore."
  },
  {
    "startMs": 246534,
    "text": "Uh, they were a great partner for this."
  },
  {
    "startMs": 248574,
    "text": "Um, you don't have to build it with a"
  },
  {
    "startMs": 250774,
    "text": "company, um, but simulation is pretty"
  },
  {
    "startMs": 253694,
    "text": "important. Once you have users, things"
  },
  {
    "startMs": 255454,
    "text": "get a little easier, if not harder."
  },
  {
    "startMs": 258014,
    "text": "We're not going to talk about that phase"
  },
  {
    "startMs": 259854,
    "text": "in this talk. I only have 18 minutes."
  },
  {
    "startMs": 261694,
    "text": "That would be longer than an 18-minute"
  },
  {
    "startMs": 264534,
    "text": "talk. Um, so data is the bottleneck to"
  },
  {
    "startMs": 267494,
    "text": "evals."
  },
  {
    "startMs": 268534,
    "text": "I think that's an obvious statement, but"
  },
  {
    "startMs": 270014,
    "text": "I want that to be like very clear that"
  },
  {
    "startMs": 272174,
    "text": "to evaluate, you have to have clean data"
  },
  {
    "startMs": 273934,
    "text": "to do things on."
  },
  {
    "startMs": 276014,
    "text": "So,"
  },
  {
    "startMs": 277094,
    "text": "how do we design the evals? Like,"
  },
  {
    "startMs": 279494,
    "text": "how did we think about making a data"
  },
  {
    "startMs": 282814,
    "text": "set, a set of task criteria that we"
  },
  {
    "startMs": 286334,
    "text": "would get value out of?"
  },
  {
    "startMs": 288334,
    "text": "Uh, so we built synthetic businesses."
  },
  {
    "startMs": 291654,
    "text": "Uh, this is an example of one of our"
  },
  {
    "startMs": 293574,
    "text": "synthetic businesses."
  },
  {
    "startMs": 295294,
    "text": "Uh, they're a D2C lifestyle company. Uh,"
  },
  {
    "startMs": 298614,
    "text": "they use Amazon. Uh, there's some"
  },
  {
    "startMs": 301614,
    "text": "complexity to how they do their"
  },
  {
    "startMs": 303414,
    "text": "accounting. They have a calendar year."
  },
  {
    "startMs": 306734,
    "text": "They want to do just close at the end of"
  },
  {
    "startMs": 308494,
    "text": "the month. They're multi-channel. They"
  },
  {
    "startMs": 310654,
    "text": "get revenue from Shopify, Amazon, a"
  },
  {
    "startMs": 313454,
    "text": "bunch of different stuff. Uh, the"
  },
  {
    "startMs": 315654,
    "text": "details of this are not that important"
  },
  {
    "startMs": 318254,
    "text": "of who this company is, but what we also"
  },
  {
    "startMs": 320814,
    "text": "did is we built a set of documents or"
  },
  {
    "startMs": 323574,
    "text": "data for this company. We built their"
  },
  {
    "startMs": 325894,
    "text": "ERP. We built their customer list, their"
  },
  {
    "startMs": 328014,
    "text": "vendor list, the accounts payable,"
  },
  {
    "startMs": 329974,
    "text": "everything"
  },
  {
    "startMs": 331574,
    "text": "that would be needed for a synthetic"
  },
  {
    "startMs": 333294,
    "text": "business,"
  },
  {
    "startMs": 334494,
    "text": "uh, in order to do good offline"
  },
  {
    "startMs": 336254,
    "text": "evaluation."
  },
  {
    "startMs": 338454,
    "text": "Uh,"
  },
  {
    "startMs": 339534,
    "text": "all we're doing here, by the way, is"
  },
  {
    "startMs": 341254,
    "text": "just backtesting agents and measuring"
  },
  {
    "startMs": 343934,
    "text": "success. Like I said, um,"
  },
  {
    "startMs": 347454,
    "text": "people use these terms interchangeably."
  },
  {
    "startMs": 350774,
    "text": "My training is as an MLE. Uh, before I"
  },
  {
    "startMs": 354774,
    "text": "came at Ramp, I was the first MLE at"
  },
  {
    "startMs": 356894,
    "text": "Ramp."
  },
  {
    "startMs": 357934,
    "text": "Um, and I think one of the problems on"
  },
  {
    "startMs": 360094,
    "text": "the discourse on evaluations,"
  },
  {
    "startMs": 362454,
    "text": "verifiability, and benchmarks,"
  },
  {
    "startMs": 365294,
    "text": "uh, is that they're just trying to find"
  },
  {
    "startMs": 367415,
    "text": "labels. They're just trying to find ways"
  },
  {
    "startMs": 370775,
    "text": "to measure success for things."
  },
  {
    "startMs": 373295,
    "text": "Um and they're all sort of saying the"
  },
  {
    "startMs": 375375,
    "text": "same thing."
  },
  {
    "startMs": 376935,
    "text": "Uh but really you're just looking to run"
  },
  {
    "startMs": 379535,
    "text": "back tests. You're looking to run your"
  },
  {
    "startMs": 381695,
    "text": "agents against the same set of data"
  },
  {
    "startMs": 385255,
    "text": "day over day over day to find ways to"
  },
  {
    "startMs": 388135,
    "text": "optimize it. I sort of have a theory"
  },
  {
    "startMs": 390455,
    "text": "that the industry's been a little um"
  },
  {
    "startMs": 394535,
    "text": "uh I the term is not hurt, but I think"
  },
  {
    "startMs": 397775,
    "text": "we had it good when it's just coding"
  },
  {
    "startMs": 399735,
    "text": "agents cuz Git by nature allows you to"
  },
  {
    "startMs": 402615,
    "text": "replay things in the past. State is"
  },
  {
    "startMs": 405415,
    "text": "always captured. When you start to think"
  },
  {
    "startMs": 407775,
    "text": "about production agents,"
  },
  {
    "startMs": 410215,
    "text": "how do you capture state over time so"
  },
  {
    "startMs": 412855,
    "text": "you can replay?"
  },
  {
    "startMs": 414735,
    "text": "Uh we're in a nice place when we have no"
  },
  {
    "startMs": 417655,
    "text": "users. We build that state and we keep"
  },
  {
    "startMs": 419975,
    "text": "that state the same over time."
  },
  {
    "startMs": 422975,
    "text": "But that's really just our goal. We're"
  },
  {
    "startMs": 424495,
    "text": "running a back test. We're measuring"
  },
  {
    "startMs": 426175,
    "text": "success."
  },
  {
    "startMs": 427735,
    "text": "And so this is is sort of what the"
  },
  {
    "startMs": 430255,
    "text": "trajectory of the process looks like. We"
  },
  {
    "startMs": 433935,
    "text": "build a task, in this case a"
  },
  {
    "startMs": 435935,
    "text": "reconciliation."
  },
  {
    "startMs": 437695,
    "text": "We call these things worlds. It's a term"
  },
  {
    "startMs": 439655,
    "text": "that Mercor"
  },
  {
    "startMs": 441175,
    "text": "um came up with and I quite like it. Uh"
  },
  {
    "startMs": 444855,
    "text": "a world could also be your production"
  },
  {
    "startMs": 446775,
    "text": "data, a snapshot of it. There's"
  },
  {
    "startMs": 450295,
    "text": "no reason to swap that in. It's the same"
  },
  {
    "startMs": 452855,
    "text": "thing."
  },
  {
    "startMs": 453895,
    "text": "We have the agent run and then we just"
  },
  {
    "startMs": 456335,
    "text": "evaluate on the final output. You can"
  },
  {
    "startMs": 458655,
    "text": "see the criteria that we have. For each"
  },
  {
    "startMs": 461255,
    "text": "task we tended to have anywhere from 10"
  },
  {
    "startMs": 464615,
    "text": "to 20 criteria. We didn't judge the"
  },
  {
    "startMs": 467935,
    "text": "trajectories um themselves like were the"
  },
  {
    "startMs": 470895,
    "text": "right tools called, were the right"
  },
  {
    "startMs": 472455,
    "text": "skills invoked? Did it take an efficient"
  },
  {
    "startMs": 476055,
    "text": "path? We do care about latency, cost,"
  },
  {
    "startMs": 479575,
    "text": "and like success. Um"
  },
  {
    "startMs": 483295,
    "text": "there are different reasons you might"
  },
  {
    "startMs": 485295,
    "text": "care about trajectories, but when you're"
  },
  {
    "startMs": 487135,
    "text": "initially building your agent, you just"
  },
  {
    "startMs": 488937,
    "text": "want it to be right, to be cheap, and to"
  },
  {
    "startMs": 491177,
    "text": "be fast. How it gets there is a little"
  },
  {
    "startMs": 494177,
    "text": "bit less important."
  },
  {
    "startMs": 496097,
    "text": "Uh so, that's not what we focused on."
  },
  {
    "startMs": 499217,
    "text": "And like I had said my mini rant before,"
  },
  {
    "startMs": 502817,
    "text": "in this as example, we're looking for a"
  },
  {
    "startMs": 505257,
    "text": "few things that define success. If I"
  },
  {
    "startMs": 508497,
    "text": "were just in MLE, I would say these are"
  },
  {
    "startMs": 510297,
    "text": "the labels. Um the thing I want to say"
  },
  {
    "startMs": 513337,
    "text": "is the process to build this whole thing"
  },
  {
    "startMs": 515897,
    "text": "was"
  },
  {
    "startMs": 517097,
    "text": "4 to 6 months before launching this"
  },
  {
    "startMs": 519857,
    "text": "agent. And I think that level of time"
  },
  {
    "startMs": 522617,
    "text": "investment in evals uh is typical in"
  },
  {
    "startMs": 526697,
    "text": "companies that are trying to build"
  },
  {
    "startMs": 529137,
    "text": "agents for"
  },
  {
    "startMs": 531137,
    "text": "areas that are not fully optimized uh by"
  },
  {
    "startMs": 535337,
    "text": "the frontier models. You're throwing"
  },
  {
    "startMs": 536897,
    "text": "something as a coding agent, sure, it'll"
  },
  {
    "startMs": 539217,
    "text": "probably work great out of the box. If"
  },
  {
    "startMs": 542217,
    "text": "you're in a front an a frontier area, an"
  },
  {
    "startMs": 544697,
    "text": "area that's"
  },
  {
    "startMs": 546017,
    "text": "slightly orthogonal to what they're"
  },
  {
    "startMs": 548017,
    "text": "optimized against, um you need to spend"
  },
  {
    "startMs": 551737,
    "text": "some time building these things."
  },
  {
    "startMs": 554137,
    "text": "So,"
  },
  {
    "startMs": 555097,
    "text": "how do we measure performance? I'm sure"
  },
  {
    "startMs": 556577,
    "text": "there's a lot of people in this room"
  },
  {
    "startMs": 558017,
    "text": "that have seen these metrics. Um one is"
  },
  {
    "startMs": 560577,
    "text": "accuracy, that's pretty simple. So, like"
  },
  {
    "startMs": 562937,
    "text": "I said, this given task had six"
  },
  {
    "startMs": 565897,
    "text": "criteria, four of those six are right,"
  },
  {
    "startMs": 568777,
    "text": "67%, so partial credit counts. Uh the"
  },
  {
    "startMs": 572417,
    "text": "other way um that you'll see reported a"
  },
  {
    "startMs": 574537,
    "text": "lot is pass at K. Um this is an example"
  },
  {
    "startMs": 577657,
    "text": "of three runs on this given agent. Uh in"
  },
  {
    "startMs": 581537,
    "text": "the first run, it got four right. Second"
  },
  {
    "startMs": 584657,
    "text": "run, it got five right. Final attempt,"
  },
  {
    "startMs": 587377,
    "text": "it got six right, so it got all of them"
  },
  {
    "startMs": 589457,
    "text": "right. Pass at K is an all or nothing"
  },
  {
    "startMs": 591977,
    "text": "metric. Um you could think for"
  },
  {
    "startMs": 595617,
    "text": "areas where the cost of error is very"
  },
  {
    "startMs": 598257,
    "text": "high, maybe like accounting, uh pass at"
  },
  {
    "startMs": 601817,
    "text": "K might be the thing you're optimizing"
  },
  {
    "startMs": 603577,
    "text": "for. You need everything to be right. If"
  },
  {
    "startMs": 605737,
    "text": "an accountant can't trust that one of"
  },
  {
    "startMs": 607937,
    "text": "the steps is wrong,"
  },
  {
    "startMs": 609897,
    "text": "then they may not use your agent. Um so,"
  },
  {
    "startMs": 613738,
    "text": "this is pass at K. You can see pass at"
  },
  {
    "startMs": 615978,
    "text": "one is zero, pass at two is zero, and"
  },
  {
    "startMs": 618338,
    "text": "then pass at three is one because it"
  },
  {
    "startMs": 619818,
    "text": "finally solved it."
  },
  {
    "startMs": 622098,
    "text": "Okay. So, now I'm going to talk about"
  },
  {
    "startMs": 624538,
    "text": "that's how we built the evals. That's"
  },
  {
    "startMs": 625938,
    "text": "what our evals were."
  },
  {
    "startMs": 627498,
    "text": "I'm going to talk about all the things"
  },
  {
    "startMs": 628778,
    "text": "we now do with these evals now that we"
  },
  {
    "startMs": 630498,
    "text": "spent that four to six months, now that"
  },
  {
    "startMs": 632058,
    "text": "we spent all the money on it and time."
  },
  {
    "startMs": 634578,
    "text": "I mean, the first obvious case, right,"
  },
  {
    "startMs": 635938,
    "text": "is you're going to like pick a frontier"
  },
  {
    "startMs": 637738,
    "text": "model. You're going to be like, which"
  },
  {
    "startMs": 639058,
    "text": "model is best at this? Um this is a"
  },
  {
    "startMs": 641178,
    "text": "little old. This is from a blog post we"
  },
  {
    "startMs": 643258,
    "text": "wrote when Stack launched. Um as you"
  },
  {
    "startMs": 646058,
    "text": "could sort of see at the time,"
  },
  {
    "startMs": 648698,
    "text": "uh the Opus models uh were the highest"
  },
  {
    "startMs": 651658,
    "text": "performing. That's still what we see"
  },
  {
    "startMs": 653858,
    "text": "when we rerun these benchmarks. 55 is"
  },
  {
    "startMs": 656138,
    "text": "the best model. What you'll also see is"
  },
  {
    "startMs": 658578,
    "text": "accuracy is very high. Pass at one is"
  },
  {
    "startMs": 661338,
    "text": "very low."
  },
  {
    "startMs": 662738,
    "text": "These aren't saturated yet, which is"
  },
  {
    "startMs": 664538,
    "text": "great. Um it's great for us, too,"
  },
  {
    "startMs": 667338,
    "text": "because we find harness optimizations"
  },
  {
    "startMs": 669858,
    "text": "allow us to get extra pass at. Um so, it"
  },
  {
    "startMs": 673698,
    "text": "means you can't just throw a frontier"
  },
  {
    "startMs": 675778,
    "text": "model at this problem and have it"
  },
  {
    "startMs": 677418,
    "text": "solved."
  },
  {
    "startMs": 678818,
    "text": "We dug deeper into why the models are"
  },
  {
    "startMs": 682578,
    "text": "failing to spot areas for optimization."
  },
  {
    "startMs": 685778,
    "text": "Uh so, we looked at things like"
  },
  {
    "startMs": 686898,
    "text": "reasoning failure. Does it have all the"
  },
  {
    "startMs": 688738,
    "text": "data? It's just reasoning through it"
  },
  {
    "startMs": 690138,
    "text": "incorrectly?"
  },
  {
    "startMs": 691418,
    "text": "Uh does it not gather the right"
  },
  {
    "startMs": 693098,
    "text": "information? Does it not follow"
  },
  {
    "startMs": 694938,
    "text": "instructions?"
  },
  {
    "startMs": 696458,
    "text": "Um"
  },
  {
    "startMs": 697058,
    "text": "and then like reflection failure, which"
  },
  {
    "startMs": 698618,
    "text": "is I would actually call it other. Um"
  },
  {
    "startMs": 701258,
    "text": "interesting result that we see is that"
  },
  {
    "startMs": 703058,
    "text": "for accounting, it's reasoning failure."
  },
  {
    "startMs": 705018,
    "text": "What that means is that these harnesses"
  },
  {
    "startMs": 707098,
    "text": "are pretty good at actually gathering"
  },
  {
    "startMs": 708578,
    "text": "the right amount of data. Accounting is"
  },
  {
    "startMs": 710738,
    "text": "not that difficult of an"
  },
  {
    "startMs": 713378,
    "text": "gathering exercise. Accounts payable is"
  },
  {
    "startMs": 715898,
    "text": "like something I could explain to like a"
  },
  {
    "startMs": 718098,
    "text": "five-year-old, so you could sort of get"
  },
  {
    "startMs": 720378,
    "text": "a well, sophisticated five-year-old. Um"
  },
  {
    "startMs": 722818,
    "text": "so, you could sort of get uh what files"
  },
  {
    "startMs": 725378,
    "text": "you would need. Uh and so, what this"
  },
  {
    "startMs": 727258,
    "text": "sort of says is like, you know, the"
  },
  {
    "startMs": 728658,
    "text": "models, they just ain't that smart."
  },
  {
    "startMs": 731418,
    "text": "Uh and we've got some room to run here."
  },
  {
    "startMs": 734019,
    "text": "Uh, what was interesting was Music Spark"
  },
  {
    "startMs": 735379,
    "text": "actually did really well in this."
  },
  {
    "startMs": 737659,
    "text": "Um, so we ran a ton of experiments."
  },
  {
    "startMs": 740219,
    "text": "These are all the things that failed."
  },
  {
    "startMs": 742299,
    "text": "Um, I sort of like this because like"
  },
  {
    "startMs": 744939,
    "text": "you'll see papers of like these awesome"
  },
  {
    "startMs": 748059,
    "text": "things that they did to optimize and"
  },
  {
    "startMs": 751019,
    "text": "tune their agents and they don't show"
  },
  {
    "startMs": 753299,
    "text": "you the 30 things that failed. Actually"
  },
  {
    "startMs": 755739,
    "text": "want to highlight one. I'm going to go"
  },
  {
    "startMs": 757019,
    "text": "on like a mini side quest for a moment"
  },
  {
    "startMs": 759579,
    "text": "here."
  },
  {
    "startMs": 760659,
    "text": "Uh, Jeepa, auto tuning, auto researching"
  },
  {
    "startMs": 764139,
    "text": "skills."
  },
  {
    "startMs": 765779,
    "text": "No good. Uh,"
  },
  {
    "startMs": 768179,
    "text": "it might be our agent,"
  },
  {
    "startMs": 770819,
    "text": "um, but I think the idea of hill"
  },
  {
    "startMs": 773579,
    "text": "climbing with evals on text space"
  },
  {
    "startMs": 777379,
    "text": "is not the first thing I would do here."
  },
  {
    "startMs": 780619,
    "text": "Uh, intuitively it's difficult to"
  },
  {
    "startMs": 782419,
    "text": "understand why it would work, i.e. how"
  },
  {
    "startMs": 785099,
    "text": "you can get direction on a gradient. Um,"
  },
  {
    "startMs": 788939,
    "text": "we did not see success here. We were"
  },
  {
    "startMs": 790499,
    "text": "still not seeing success. I just want to"
  },
  {
    "startMs": 792419,
    "text": "call this out because I think there's a"
  },
  {
    "startMs": 793739,
    "text": "lot of noise on auto research. Um, the"
  },
  {
    "startMs": 797219,
    "text": "Karpathy famous tweet is about tuning ML"
  },
  {
    "startMs": 801059,
    "text": "models, which obviously it has a"
  },
  {
    "startMs": 803019,
    "text": "gradient, it has a direction. It's also"
  },
  {
    "startMs": 805499,
    "text": "a finite space it's searching over."
  },
  {
    "startMs": 807419,
    "text": "Great. Um, so one lesson to take away"
  },
  {
    "startMs": 810139,
    "text": "from here is maybe don't think that auto"
  },
  {
    "startMs": 811939,
    "text": "research is going to solve your"
  },
  {
    "startMs": 814379,
    "text": "bad agent problem."
  },
  {
    "startMs": 816259,
    "text": "So here's some experiments that did"
  },
  {
    "startMs": 817699,
    "text": "work."
  },
  {
    "startMs": 818699,
    "text": "Um, I'm going to talk about two."
  },
  {
    "startMs": 820619,
    "text": "I'm going to talk about skill ablation"
  },
  {
    "startMs": 822899,
    "text": "and trajectory distilled memories. So"
  },
  {
    "startMs": 826339,
    "text": "I'm an MLE by training."
  },
  {
    "startMs": 828459,
    "text": "Uh, usually what you do is you build a"
  },
  {
    "startMs": 830259,
    "text": "bunch of features,"
  },
  {
    "startMs": 832099,
    "text": "uh, throw them in a model,"
  },
  {
    "startMs": 834659,
    "text": "and then you might do like a leave one"
  },
  {
    "startMs": 836419,
    "text": "out feature, or you might add features"
  },
  {
    "startMs": 839499,
    "text": "back in. It's all just trying to assess"
  },
  {
    "startMs": 842139,
    "text": "the incrementality"
  },
  {
    "startMs": 843859,
    "text": "of a given feature on model performance."
  },
  {
    "startMs": 846739,
    "text": "Uh, agents with many skills, many tools."
  },
  {
    "startMs": 849259,
    "text": "We didn't do with tools, we did with"
  },
  {
    "startMs": 850459,
    "text": "skills. Tools are a little bit more"
  },
  {
    "startMs": 852339,
    "text": "difficult to deal with, but you could do"
  },
  {
    "startMs": 853979,
    "text": "a similar thing."
  },
  {
    "startMs": 855341,
    "text": "Uh, we just tried to trim the number of"
  },
  {
    "startMs": 858141,
    "text": "skills saving context. That's your goal."
  },
  {
    "startMs": 861021,
    "text": "Um, and what this basically shows is if"
  },
  {
    "startMs": 864181,
    "text": "you look at no skill baseline, basically"
  },
  {
    "startMs": 866341,
    "text": "this is you could just think it's like"
  },
  {
    "startMs": 867661,
    "text": "the raw model, whatever's in the"
  },
  {
    "startMs": 869181,
    "text": "weights,"
  },
  {
    "startMs": 870301,
    "text": "um, versus"
  },
  {
    "startMs": 872021,
    "text": "uh,"
  },
  {
    "startMs": 873501,
    "text": "removing uh, uh,"
  },
  {
    "startMs": 875381,
    "text": "versus like adding the skill. I I think"
  },
  {
    "startMs": 877461,
    "text": "I've reversed these actually. I think"
  },
  {
    "startMs": 878981,
    "text": "the negatives are like it's good with"
  },
  {
    "startMs": 881581,
    "text": "the skill and the positives is bad with"
  },
  {
    "startMs": 883501,
    "text": "the skill. Regardless, um, what you can"
  },
  {
    "startMs": 885821,
    "text": "see is there's definitely differences in"
  },
  {
    "startMs": 887381,
    "text": "skills being there versus not being"
  },
  {
    "startMs": 889061,
    "text": "there. And this created a candidate set"
  },
  {
    "startMs": 891341,
    "text": "for us to then be like maybe we should"
  },
  {
    "startMs": 893341,
    "text": "delete these skills, compress these"
  },
  {
    "startMs": 895421,
    "text": "skills,"
  },
  {
    "startMs": 897141,
    "text": "basically make things more efficient."
  },
  {
    "startMs": 899101,
    "text": "So, skill ablation, you should do it."
  },
  {
    "startMs": 901741,
    "text": "Tool ablation, you can also do."
  },
  {
    "startMs": 904381,
    "text": "Trajectory distilled memory. So,"
  },
  {
    "startMs": 906701,
    "text": "a lot of people talk about memories. Uh,"
  },
  {
    "startMs": 908941,
    "text": "I think the rise of personal agents,"
  },
  {
    "startMs": 910541,
    "text": "people talk about the memories and how"
  },
  {
    "startMs": 912181,
    "text": "good they are,"
  },
  {
    "startMs": 913781,
    "text": "blah, blah, blah."
  },
  {
    "startMs": 915461,
    "text": "Um,"
  },
  {
    "startMs": 916101,
    "text": "but people don't really test their"
  },
  {
    "startMs": 917501,
    "text": "memory systems. Um, or at least I don't"
  },
  {
    "startMs": 920381,
    "text": "tend to see people testing their memory"
  },
  {
    "startMs": 922141,
    "text": "systems. They'll have a prompt, they'll"
  },
  {
    "startMs": 924061,
    "text": "compress some information, they'll put"
  },
  {
    "startMs": 925381,
    "text": "it into some store, uh, and then they'll"
  },
  {
    "startMs": 927581,
    "text": "read it back. Uh, so, one little"
  },
  {
    "startMs": 930301,
    "text": "innovation we did here is what we call"
  },
  {
    "startMs": 932061,
    "text": "roll forward worlds. So, it's a little"
  },
  {
    "startMs": 935021,
    "text": "complex, so I'm going to walk you"
  },
  {
    "startMs": 936621,
    "text": "through that chart on the left. Uh, but"
  },
  {
    "startMs": 939461,
    "text": "basically, uh, you do the same task over"
  },
  {
    "startMs": 943901,
    "text": "two time periods."
  },
  {
    "startMs": 945781,
    "text": "You write a memory after the first task,"
  },
  {
    "startMs": 949701,
    "text": "uh, and then you either inject the"
  },
  {
    "startMs": 951861,
    "text": "memory in if you want to cheat a little"
  },
  {
    "startMs": 953981,
    "text": "bit, or you let the agent search over a"
  },
  {
    "startMs": 957101,
    "text": "set of memories,"
  },
  {
    "startMs": 958981,
    "text": "um, for the same task in the following"
  },
  {
    "startMs": 961861,
    "text": "period. So,"
  },
  {
    "startMs": 964181,
    "text": "in the example I have here is like uh,"
  },
  {
    "startMs": 967501,
    "text": "we want to match some records, do a"
  },
  {
    "startMs": 968981,
    "text": "reconciliation. There's some implicit"
  },
  {
    "startMs": 971941,
    "text": "customer rules and how they like to do"
  },
  {
    "startMs": 974821,
    "text": "that. Uh,"
  },
  {
    "startMs": 976541,
    "text": "those will be not fully encoded, but you"
  },
  {
    "startMs": 979662,
    "text": "can infer them from the success criteria"
  },
  {
    "startMs": 982702,
    "text": "like you have to learn these rules in"
  },
  {
    "startMs": 985422,
    "text": "order to be successful at this task. We"
  },
  {
    "startMs": 988582,
    "text": "have a memory prompt that then"
  },
  {
    "startMs": 990622,
    "text": "compresses, writes, and then we rerun"
  },
  {
    "startMs": 992942,
    "text": "it, and it reads again. And what we"
  },
  {
    "startMs": 994822,
    "text": "actually saw that there is a ton of"
  },
  {
    "startMs": 996822,
    "text": "alpha here. Like the types of memories"
  },
  {
    "startMs": 1000782,
    "text": "that were written for tasks really"
  },
  {
    "startMs": 1003902,
    "text": "depended on the prompts, and then"
  },
  {
    "startMs": 1007222,
    "text": "success was pretty in like also"
  },
  {
    "startMs": 1010222,
    "text": "dependent on those prompts. We saw a"
  },
  {
    "startMs": 1012062,
    "text": "very large gain. This was actually one"
  },
  {
    "startMs": 1013502,
    "text": "of the biggest gains we saw from having"
  },
  {
    "startMs": 1016142,
    "text": "uh, these types of evals."
  },
  {
    "startMs": 1018662,
    "text": "So,"
  },
  {
    "startMs": 1019982,
    "text": "uh, you have a great data set uh, that"
  },
  {
    "startMs": 1024142,
    "text": "you can run evals on. The thing that I"
  },
  {
    "startMs": 1027182,
    "text": "think people don't do enough of is what"
  },
  {
    "startMs": 1029222,
    "text": "I call regression testing. I'm not a big"
  },
  {
    "startMs": 1032542,
    "text": "believer in CI gating based off of"
  },
  {
    "startMs": 1036022,
    "text": "agentic evals as someone that's seen a"
  },
  {
    "startMs": 1038222,
    "text": "lot of non-determinism, a lot of 80%,"
  },
  {
    "startMs": 1040982,
    "text": "70% passage. Uh, CI gating is"
  },
  {
    "startMs": 1044942,
    "text": "not usually the best idea, but instead"
  },
  {
    "startMs": 1047022,
    "text": "what we do now is we have an internal"
  },
  {
    "startMs": 1049382,
    "text": "platform that hosts our evals that"
  },
  {
    "startMs": 1051822,
    "text": "people can add the evals to. This top"
  },
  {
    "startMs": 1055102,
    "text": "uh, picture you see is an example of"
  },
  {
    "startMs": 1057502,
    "text": "stack. We have a bunch of different"
  },
  {
    "startMs": 1059382,
    "text": "cases. We actually tag these cases so"
  },
  {
    "startMs": 1061822,
    "text": "that we can cluster and organize them a"
  },
  {
    "startMs": 1063902,
    "text": "little bit better. We set a recurring"
  },
  {
    "startMs": 1066182,
    "text": "schedule, so these should run in the"
  },
  {
    "startMs": 1068062,
    "text": "background every week, every day,"
  },
  {
    "startMs": 1070462,
    "text": "whatever. We run it, we see how things"
  },
  {
    "startMs": 1073222,
    "text": "perform. We have a background agent that"
  },
  {
    "startMs": 1075302,
    "text": "then looks at like failures,"
  },
  {
    "startMs": 1077102,
    "text": "regressions, all of those things. Uh,"
  },
  {
    "startMs": 1079782,
    "text": "and then we get a Slack message that"
  },
  {
    "startMs": 1081542,
    "text": "tells us, \"Ooh, it looks like"
  },
  {
    "startMs": 1082502,
    "text": "something's regressed. Go investigate"
  },
  {
    "startMs": 1085102,
    "text": "it. Here's what I think broke.\""
  },
  {
    "startMs": 1087222,
    "text": "So, getting the value out of your evals"
  },
  {
    "startMs": 1089822,
    "text": "is important. So, takeaways. I'm at"
  },
  {
    "startMs": 1092622,
    "text": "time. Break down a job into discrete,"
  },
  {
    "startMs": 1095142,
    "text": "you know, tasks. Build realistic"
  },
  {
    "startMs": 1097222,
    "text": "environments."
  },
  {
    "startMs": 1098502,
    "text": "Focus on extracting like value as much"
  },
  {
    "startMs": 1101623,
    "text": "as possible Um, and then figure out how"
  },
  {
    "startMs": 1104743,
    "text": "to consistently put those into workflows"
  },
  {
    "startMs": 1106783,
    "text": "to measure performance."
  },
  {
    "startMs": 1108543,
    "text": "Cool. Thank you for listening and yeah,"
  },
  {
    "startMs": 1110863,
    "text": "appreciate the time."
  },
  {
    "startMs": 1112857,
    "text": ">> [applause]"
  }
]