[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12800,
    "text": "All right. Hello everybody."
  },
  {
    "startMs": 15920,
    "text": "Thank you for coming to this"
  },
  {
    "startMs": 17359,
    "text": "delightfully nerdy talk. Uh this talk"
  },
  {
    "startMs": 20560,
    "text": "has a really long title. Uh so let me"
  },
  {
    "startMs": 22560,
    "text": "give you the short version up front. You"
  },
  {
    "startMs": 24160,
    "text": "write skills files uh and stuff them"
  },
  {
    "startMs": 26640,
    "text": "full of instructions. At some point the"
  },
  {
    "startMs": 28560,
    "text": "model stops keeping track of all of"
  },
  {
    "startMs": 30560,
    "text": "them. The question is where is that"
  },
  {
    "startMs": 32719,
    "text": "point? At what point have you put too"
  },
  {
    "startMs": 34399,
    "text": "many instructions in your skills files?"
  },
  {
    "startMs": 36480,
    "text": "Uh and the answer has changed a lot in"
  },
  {
    "startMs": 39120,
    "text": "the last year. I'm Lori. I'm head of"
  },
  {
    "startMs": 42079,
    "text": "developer relations at Arise AI. Uh in a"
  },
  {
    "startMs": 44399,
    "text": "former life, I co-founded npm Inc. So"
  },
  {
    "startMs": 46320,
    "text": "some of you may know me from the days of"
  },
  {
    "startMs": 47680,
    "text": "JavaScript. These days I spend a lot of"
  },
  {
    "startMs": 49680,
    "text": "time thinking about AI and how to test"
  },
  {
    "startMs": 51360,
    "text": "it."
  },
  {
    "startMs": 53360,
    "text": "Uh a few months ago I was at AI engineer"
  },
  {
    "startMs": 55360,
    "text": "in Miami which was a good conference. Uh"
  },
  {
    "startMs": 57440,
    "text": "and I was watching a talk by Dexter"
  },
  {
    "startMs": 58960,
    "text": "Horthy. Uh it was a good talk. It was"
  },
  {
    "startMs": 61039,
    "text": "not about this topic at all. Uh but"
  },
  {
    "startMs": 63199,
    "text": "while he was giving that talk he"
  },
  {
    "startMs": 64720,
    "text": "mentioned as an aside uh that an agent"
  },
  {
    "startMs": 67680,
    "text": "can follow up to about 200 instructions"
  },
  {
    "startMs": 70560,
    "text": "uh before it starts forgetting those"
  },
  {
    "startMs": 72560,
    "text": "instructions. Uh and he then he moved on"
  },
  {
    "startMs": 75360,
    "text": "in his talk and it was entirely an"
  },
  {
    "startMs": 76880,
    "text": "aside. Uh and he he mentioned that that"
  },
  {
    "startMs": 79040,
    "text": "figure is from 2025 so things might be"
  },
  {
    "startMs": 81360,
    "text": "better now. Um, and I stopped listening"
  },
  {
    "startMs": 84560,
    "text": "for a second because I was like, 200"
  },
  {
    "startMs": 86479,
    "text": "instructions. Uh, is not very many"
  },
  {
    "startMs": 90000,
    "text": "instructions at all. Right? A decent"
  },
  {
    "startMs": 91920,
    "text": "skills file blows past 200 instructions"
  },
  {
    "startMs": 93920,
    "text": "almost immediately. Um, if the user says"
  },
  {
    "startMs": 96720,
    "text": "X, do Y, always include a section on Z,"
  },
  {
    "startMs": 99280,
    "text": "never use the phrase W, every one of"
  },
  {
    "startMs": 101040,
    "text": "those is a separate instruction. Uh, and"
  },
  {
    "startMs": 103200,
    "text": "if the model quietly stops tracking them"
  },
  {
    "startMs": 104960,
    "text": "after 200, that's a really hard ceiling"
  },
  {
    "startMs": 107119,
    "text": "on the complexity of what you can build."
  },
  {
    "startMs": 109680,
    "text": "Uh, so I wanted to know where he got"
  },
  {
    "startMs": 111360,
    "text": "that number first. Uh, and I wanted to"
  },
  {
    "startMs": 113439,
    "text": "know if it was true. Uh, so you know the"
  },
  {
    "startMs": 116399,
    "text": "feeling that I'm talking about. You"
  },
  {
    "startMs": 117439,
    "text": "write this big beautiful skills file,"
  },
  {
    "startMs": 118880,
    "text": "pages of rules, edge cases, tone,"
  },
  {
    "startMs": 120560,
    "text": "formatting, you hand it to the agent, it"
  },
  {
    "startMs": 122159,
    "text": "does the thing. Uh, and you look at the"
  },
  {
    "startMs": 124479,
    "text": "output and go, did it actually pay"
  },
  {
    "startMs": 126880,
    "text": "attention? Did it actually follow all of"
  },
  {
    "startMs": 128720,
    "text": "these rules or did it just sort of, you"
  },
  {
    "startMs": 130560,
    "text": "know, do what it felt like and sort of"
  },
  {
    "startMs": 132239,
    "text": "give me a close simulacum of what I was"
  },
  {
    "startMs": 134959,
    "text": "expecting? Um, you can't really tell or"
  },
  {
    "startMs": 138080,
    "text": "can you? More on that later. Um, and so"
  },
  {
    "startMs": 141200,
    "text": "you live with this lowgrade anxiety"
  },
  {
    "startMs": 143440,
    "text": "every time you hit run. Um, and that"
  },
  {
    "startMs": 145520,
    "text": "feeling is what this research is about"
  },
  {
    "startMs": 147280,
    "text": "and what we're trying to find out if we"
  },
  {
    "startMs": 149200,
    "text": "can avoid."
  },
  {
    "startMs": 150879,
    "text": "So here's my promise for your next 18"
  },
  {
    "startMs": 152959,
    "text": "minutes. Uh, I'm going to show you where"
  },
  {
    "startMs": 154720,
    "text": "that 200 number came from, whether it's"
  },
  {
    "startMs": 156560,
    "text": "still true, and what the real number is"
  },
  {
    "startMs": 158480,
    "text": "today. Uh, because it moved by an order"
  },
  {
    "startMs": 160560,
    "text": "of magnitude. Uh, and then we're going"
  },
  {
    "startMs": 162720,
    "text": "to talk about what that means for you to"
  },
  {
    "startMs": 165120,
    "text": "take away. uh how long your skills and"
  },
  {
    "startMs": 167680,
    "text": "prompts can actually be and what that"
  },
  {
    "startMs": 170400,
    "text": "should what changes you should make to"
  },
  {
    "startMs": 172080,
    "text": "your workflow as a result."
  },
  {
    "startMs": 174720,
    "text": "So the 200 number isn't folklore. Uh it"
  },
  {
    "startMs": 177280,
    "text": "comes from a real benchmark called"
  },
  {
    "startMs": 178480,
    "text": "IFScale uh from a paper uh by this guy"
  },
  {
    "startMs": 182319,
    "text": "whose name I'm going to mess up"
  },
  {
    "startMs": 183840,
    "text": "Jeroslowitch uh and co-authors last"
  },
  {
    "startMs": 186480,
    "text": "year. And the test is beautifully"
  },
  {
    "startMs": 188319,
    "text": "simple. Uh here's how if scale works. If"
  },
  {
    "startMs": 192159,
    "text": "you ask the model to write a business"
  },
  {
    "startMs": 193840,
    "text": "report uh and you give it a list of"
  },
  {
    "startMs": 195920,
    "text": "specific words that it has to include"
  },
  {
    "startMs": 197680,
    "text": "exactly in the report, include the exact"
  },
  {
    "startMs": 200159,
    "text": "word customer, include the exact word"
  },
  {
    "startMs": 201920,
    "text": "revenue, and so on for as many words as"
  },
  {
    "startMs": 203840,
    "text": "you want. Each of those is an"
  },
  {
    "startMs": 205120,
    "text": "instruction that it has to follow. Uh"
  },
  {
    "startMs": 207519,
    "text": "and then you count how many of those"
  },
  {
    "startMs": 209120,
    "text": "exact words showed up. Um"
  },
  {
    "startMs": 212799,
    "text": "so because the test is so simple, you"
  },
  {
    "startMs": 214959,
    "text": "only have to keep two numbers in your"
  },
  {
    "startMs": 216159,
    "text": "head. One is density, which we call n."
  },
  {
    "startMs": 218319,
    "text": "That is how many rules we're talking"
  },
  {
    "startMs": 220080,
    "text": "about at once. And the second is"
  },
  {
    "startMs": 221920,
    "text": "accuracy, which is the percentage of"
  },
  {
    "startMs": 223440,
    "text": "those rules uh that it was able to"
  },
  {
    "startMs": 225440,
    "text": "actually follow. Uh now you might say uh"
  },
  {
    "startMs": 229040,
    "text": "that including random words in a report"
  },
  {
    "startMs": 230879,
    "text": "is not the same as following real"
  },
  {
    "startMs": 232400,
    "text": "instructions and fair enough and we're"
  },
  {
    "startMs": 234000,
    "text": "going to talk about that. Um but the"
  },
  {
    "startMs": 236239,
    "text": "keywords are a proxy. Uh include the"
  },
  {
    "startMs": 239200,
    "text": "word revenue is the same shape of task"
  },
  {
    "startMs": 241200,
    "text": "as include a section on pricing, right?"
  },
  {
    "startMs": 243040,
    "text": "Or never use this phrase. It is a"
  },
  {
    "startMs": 244720,
    "text": "discrete named constraint that you've"
  },
  {
    "startMs": 246720,
    "text": "told the agent that it has to follow."
  },
  {
    "startMs": 248959,
    "text": "Um, if a model can't track 200 words in"
  },
  {
    "startMs": 251519,
    "text": "one prompt, it's definitely going to"
  },
  {
    "startMs": 252879,
    "text": "struggle with 200 more complicated"
  },
  {
    "startMs": 254959,
    "text": "instructions. Uh, so uh, if anything,"
  },
  {
    "startMs": 259440,
    "text": "it's going to do worse. So, this number"
  },
  {
    "startMs": 260880,
    "text": "is a ceiling. Uh, this number is as high"
  },
  {
    "startMs": 263280,
    "text": "as you can go. If you give it more"
  },
  {
    "startMs": 264720,
    "text": "complicated instructions, the number is"
  },
  {
    "startMs": 266160,
    "text": "probably going to get lower. And 200 is"
  },
  {
    "startMs": 268240,
    "text": "a really low ceiling. Um so before"
  },
  {
    "startMs": 271680,
    "text": "chasing new models you have to do good"
  },
  {
    "startMs": 273199,
    "text": "science which means that you have to"
  },
  {
    "startMs": 274560,
    "text": "replicate the uh old result and make"
  },
  {
    "startMs": 276880,
    "text": "sure uh that the 200 ceiling is real. So"
  },
  {
    "startMs": 280080,
    "text": "I reran the original benchmark. Um the"
  },
  {
    "startMs": 283199,
    "text": "original paper tested a whole batch of"
  },
  {
    "startMs": 284800,
    "text": "models uh and models uh live and die"
  },
  {
    "startMs": 287280,
    "text": "really fast. So uh by the time I got"
  },
  {
    "startMs": 289680,
    "text": "around to doing this testing only three"
  },
  {
    "startMs": 291280,
    "text": "of the models in the original set of 10"
  },
  {
    "startMs": 293120,
    "text": "models that they used were still"
  },
  {
    "startMs": 294479,
    "text": "available via any kind of API. Uh so"
  },
  {
    "startMs": 297120,
    "text": "they were GPT 4.1, Claude Sonnet 4, and"
  },
  {
    "startMs": 299759,
    "text": "Gemini 2.5 Pro. Those were models that"
  },
  {
    "startMs": 302320,
    "text": "were available 12 months ago that are"
  },
  {
    "startMs": 303840,
    "text": "still available now. Um and that is why"
  },
  {
    "startMs": 306080,
    "text": "we tested those three because they were"
  },
  {
    "startMs": 307680,
    "text": "what was left. Um and since I first"
  },
  {
    "startMs": 310800,
    "text": "published this research a couple of"
  },
  {
    "startMs": 312160,
    "text": "weeks ago, uh one of those three models"
  },
  {
    "startMs": 314080,
    "text": "has been retired. So this was the last"
  },
  {
    "startMs": 315600,
    "text": "possible time that I could have run this"
  },
  {
    "startMs": 317120,
    "text": "test. Um so of that lineup, we're"
  },
  {
    "startMs": 320080,
    "text": "already down to two. So don't get"
  },
  {
    "startMs": 321360,
    "text": "attached to your models. Um here is the"
  },
  {
    "startMs": 323919,
    "text": "results that we got replicating the"
  },
  {
    "startMs": 325360,
    "text": "original if scale finding. Uh that is"
  },
  {
    "startMs": 327919,
    "text": "accuracy on the vertical axis. So it"
  },
  {
    "startMs": 330160,
    "text": "starts at 100% and begins to fall off."
  },
  {
    "startMs": 332240,
    "text": "Uh and then the number of rules uh going"
  },
  {
    "startMs": 334560,
    "text": "up along the bottom on log scale. So"
  },
  {
    "startMs": 336400,
    "text": "every time it gets halfway across it has"
  },
  {
    "startMs": 338160,
    "text": "doubled uh the number of rules that it's"
  },
  {
    "startMs": 340639,
    "text": "dealing with. Um so by 500 rules you're"
  },
  {
    "startMs": 344320,
    "text": "losing 30 40 50% of them. Uh our curves"
  },
  {
    "startMs": 347520,
    "text": "matched the results in the original"
  },
  {
    "startMs": 349120,
    "text": "paper within the noise boundary. So the"
  },
  {
    "startMs": 350800,
    "text": "finding was real. uh a year ago"
  },
  {
    "startMs": 353039,
    "text": "somewhere around 200 to 300 rules"
  },
  {
    "startMs": 355280,
    "text": "frontier models started falling apart."
  },
  {
    "startMs": 357840,
    "text": "That is a really low ceiling. Uh so that"
  },
  {
    "startMs": 361280,
    "text": "is our baseline and now comes the fun"
  },
  {
    "startMs": 363360,
    "text": "part where we took the exact same test"
  },
  {
    "startMs": 364960,
    "text": "and pointed it at the current frontier"
  },
  {
    "startMs": 367039,
    "text": "or rather what the current frontier was"
  },
  {
    "startMs": 369120,
    "text": "when I ran this test. So I ran GPT 5.5,"
  },
  {
    "startMs": 372400,
    "text": "Claude Opus 4.7 because 4.8 came out a"
  },
  {
    "startMs": 375280,
    "text": "week after I ran this test. Uh Gemini"
  },
  {
    "startMs": 377840,
    "text": "3.1 Pro and Deepseek V4 Pro. So, I gave"
  },
  {
    "startMs": 380880,
    "text": "them the same prompt, the same words,"
  },
  {
    "startMs": 382400,
    "text": "the same everything. And I immediately"
  },
  {
    "startMs": 384240,
    "text": "ran into a problem, which is that they"
  },
  {
    "startMs": 386160,
    "text": "aced it. They all scored 100%"
  },
  {
    "startMs": 389440,
    "text": "immediately on this test. Absolutely no"
  },
  {
    "startMs": 391680,
    "text": "bugs. Uh,"
  },
  {
    "startMs": 394400,
    "text": "so we'd built a test to find the ceiling"
  },
  {
    "startMs": 396080,
    "text": "and the models had walked straight"
  },
  {
    "startMs": 397199,
    "text": "through the ceiling without noticing"
  },
  {
    "startMs": 398400,
    "text": "that the ceiling was there. Um, and that"
  },
  {
    "startMs": 400960,
    "text": "was a problem because the benchmark was"
  },
  {
    "startMs": 402319,
    "text": "written to top out at 500 words. So, I"
  },
  {
    "startMs": 404080,
    "text": "had to change the benchmark in order to"
  },
  {
    "startMs": 405600,
    "text": "be able to find the new ceiling. So, I"
  },
  {
    "startMs": 407680,
    "text": "moved the goalposts. I gave it more"
  },
  {
    "startMs": 409520,
    "text": "words to include. I doubled uh it from"
  },
  {
    "startMs": 411840,
    "text": "500 to a,000. I doubled it again from"
  },
  {
    "startMs": 413759,
    "text": "a,000 to 2,000. And I kept doing that"
  },
  {
    "startMs": 415759,
    "text": "until I hit a 10,000word vocabulary. And"
  },
  {
    "startMs": 418160,
    "text": "that is where I began to find the"
  },
  {
    "startMs": 419919,
    "text": "ceiling of what models can do these"
  },
  {
    "startMs": 422400,
    "text": "days. Um, so let me put up the this is"
  },
  {
    "startMs": 425599,
    "text": "the money slide. This is the results."
  },
  {
    "startMs": 427919,
    "text": "Remember log scale on the on the uh"
  },
  {
    "startMs": 431039,
    "text": "x-axis there. So it's going from 500 to"
  },
  {
    "startMs": 433599,
    "text": "1,000 to 5,000 to 10,000. Uh so it looks"
  },
  {
    "startMs": 436720,
    "text": "like that scale is falling off of a"
  },
  {
    "startMs": 438080,
    "text": "cliff and it's actually happening over"
  },
  {
    "startMs": 439360,
    "text": "like a thousand numbers. Um"
  },
  {
    "startMs": 442960,
    "text": "but uh look how far to the right these"
  },
  {
    "startMs": 445360,
    "text": "new curves get before they bend. A year"
  },
  {
    "startMs": 446800,
    "text": "ago they were falling over at 200 to 300"
  },
  {
    "startMs": 449280,
    "text": "instructions and now depending on the"
  },
  {
    "startMs": 450639,
    "text": "model the boundary is closer to 2,000."
  },
  {
    "startMs": 452560,
    "text": "And for the best of them it is up to"
  },
  {
    "startMs": 454560,
    "text": "5,000 instructions before they begin to"
  },
  {
    "startMs": 457039,
    "text": "fall off a cliff. So in about 12 months"
  },
  {
    "startMs": 459759,
    "text": "frontier models got close to 10 times"
  },
  {
    "startMs": 461840,
    "text": "better at following instructions"
  },
  {
    "startMs": 463280,
    "text": "simultaneously. That is the headline"
  },
  {
    "startMs": 465520,
    "text": "fighting and there is a lot of nuance"
  },
  {
    "startMs": 467120,
    "text": "that we need to get into. Um the"
  },
  {
    "startMs": 470080,
    "text": "capacity to track 2,000 named"
  },
  {
    "startMs": 471840,
    "text": "constraints in a single prompt is there."
  },
  {
    "startMs": 474000,
    "text": "Um and that's really interesting because"
  },
  {
    "startMs": 476560,
    "text": "I think uh I don't know if everybody"
  },
  {
    "startMs": 478879,
    "text": "else feels this way but like it feel it"
  },
  {
    "startMs": 481440,
    "text": "felt to me like the the jump from you"
  },
  {
    "startMs": 484400,
    "text": "know GPT 5.1 to GPT 5.5 was kind of"
  },
  {
    "startMs": 486960,
    "text": "incremental, right? It didn't feel like"
  },
  {
    "startMs": 488560,
    "text": "we'd got 10 times better. But this is a"
  },
  {
    "startMs": 491520,
    "text": "test that really matters to uh a very"
  },
  {
    "startMs": 494240,
    "text": "practical thing like how long can my"
  },
  {
    "startMs": 496000,
    "text": "skills file be? Uh and in the course of"
  },
  {
    "startMs": 498080,
    "text": "a year we got 10 times better. Uh and"
  },
  {
    "startMs": 502080,
    "text": "that thing that gets me is that this"
  },
  {
    "startMs": 503360,
    "text": "benchmark is barely a a year old. A year"
  },
  {
    "startMs": 505599,
    "text": "later 500 is a rounding error. Uh and"
  },
  {
    "startMs": 508000,
    "text": "this keeps moving under my feet. I"
  },
  {
    "startMs": 509440,
    "text": "tested 4.7 uh opus 4.7. Opus 4.8 is even"
  },
  {
    "startMs": 513120,
    "text": "better. Um so this chart is a little out"
  },
  {
    "startMs": 516240,
    "text": "of date already which is kind of the"
  },
  {
    "startMs": 517519,
    "text": "whole point. If you set your engineering"
  },
  {
    "startMs": 519200,
    "text": "assumptions about how skills files"
  },
  {
    "startMs": 521200,
    "text": "should work, about how prompt how long"
  },
  {
    "startMs": 523519,
    "text": "your prompt can be, and you did that"
  },
  {
    "startMs": 525200,
    "text": "more than about six months ago, you are"
  },
  {
    "startMs": 527200,
    "text": "incorrect now, and you should probably"
  },
  {
    "startMs": 529040,
    "text": "be re-engineering how you do stuff. Uh,"
  },
  {
    "startMs": 532240,
    "text": "but there is more to this story uh"
  },
  {
    "startMs": 534800,
    "text": "because the way that the a models failed"
  },
  {
    "startMs": 538880,
    "text": "uh changed dramatically. Uh, and the way"
  },
  {
    "startMs": 541760,
    "text": "that they failed is very important. This"
  },
  {
    "startMs": 543600,
    "text": "part was a completely unexpected finding"
  },
  {
    "startMs": 546480,
    "text": "when I started running the experiment."
  },
  {
    "startMs": 548160,
    "text": "Uh, and it totally messed up my test to"
  },
  {
    "startMs": 549920,
    "text": "start with because uh, the old failure"
  },
  {
    "startMs": 552640,
    "text": "mode was boring. They would just forget"
  },
  {
    "startMs": 554240,
    "text": "instructions and I could measure how"
  },
  {
    "startMs": 555680,
    "text": "many instructions they had remembered or"
  },
  {
    "startMs": 557200,
    "text": "forgotten. Uh, but the new ones fall"
  },
  {
    "startMs": 559200,
    "text": "apart in their own weird extremely"
  },
  {
    "startMs": 561680,
    "text": "onbrand way. Uh, so let me introduce you"
  },
  {
    "startMs": 564640,
    "text": "to how these four models fail. Uh,"
  },
  {
    "startMs": 567040,
    "text": "Deepseek 4 is a traditional model. It"
  },
  {
    "startMs": 569600,
    "text": "just forgets things. It doesn't have any"
  },
  {
    "startMs": 571279,
    "text": "drama. um it starts forgetting"
  },
  {
    "startMs": 574000,
    "text": "instructions around 750 rules and by"
  },
  {
    "startMs": 576240,
    "text": "2000 it's dropping nearly half of them."
  },
  {
    "startMs": 578640,
    "text": "Uh so it just forgets which frankly is"
  },
  {
    "startMs": 580959,
    "text": "the failure mode that I trust most"
  },
  {
    "startMs": 582399,
    "text": "because it's predictable. It's very easy"
  },
  {
    "startMs": 583839,
    "text": "to measure. Uh and the other models were"
  },
  {
    "startMs": 586800,
    "text": "not nearly as cooperative. Uh Opus 4.7"
  },
  {
    "startMs": 591360,
    "text": "uh would decide repeatedly that the test"
  },
  {
    "startMs": 593440,
    "text": "was dangerous. Uh and what it would do"
  },
  {
    "startMs": 595920,
    "text": "is it would refuse at the API level to"
  },
  {
    "startMs": 598560,
    "text": "complete the test. I didn't know that"
  },
  {
    "startMs": 600320,
    "text": "there was an API response that you could"
  },
  {
    "startMs": 601839,
    "text": "get from Claude where it was like, \"No,"
  },
  {
    "startMs": 603600,
    "text": "I could do this, but I'm not going to.\""
  },
  {
    "startMs": 606000,
    "text": "Uh, but that's absolutely an API level"
  },
  {
    "startMs": 609200,
    "text": "response that Claude supports because"
  },
  {
    "startMs": 610560,
    "text": "they care so much about safety. Uh, and"
  },
  {
    "startMs": 612399,
    "text": "I started getting those all of the time."
  },
  {
    "startMs": 615120,
    "text": "Uh, and the reason that was happening is"
  },
  {
    "startMs": 616640,
    "text": "because Claude has a very sensitive"
  },
  {
    "startMs": 618240,
    "text": "safety classifier. Uh, and if you put in"
  },
  {
    "startMs": 620640,
    "text": "certain combinations of words like say"
  },
  {
    "startMs": 622560,
    "text": "anthrax and cyanide, it decides that the"
  },
  {
    "startMs": 624880,
    "text": "whole request is dangerous and it bails"
  },
  {
    "startMs": 626720,
    "text": "out. Uh, and if you remember what my"
  },
  {
    "startMs": 628800,
    "text": "test does, my test is throwing uh 5 to"
  },
  {
    "startMs": 631839,
    "text": "10,000 random words into uh into an"
  },
  {
    "startMs": 634560,
    "text": "instruction file. And so my my randomly"
  },
  {
    "startMs": 637040,
    "text": "selected words contained all sorts of"
  },
  {
    "startMs": 638880,
    "text": "things that looked dangerous in"
  },
  {
    "startMs": 640160,
    "text": "combination to the safety filter. And so"
  },
  {
    "startMs": 641920,
    "text": "it kept bailing saying that I was asking"
  },
  {
    "startMs": 643519,
    "text": "it to, you know, make a bomb or"
  },
  {
    "startMs": 644800,
    "text": "something. Um,"
  },
  {
    "startMs": 647440,
    "text": "so, uh, we had to for to get Claude to"
  },
  {
    "startMs": 652399,
    "text": "cooperate, I had to take all of my words"
  },
  {
    "startMs": 654079,
    "text": "and run them through OpenAI safety"
  },
  {
    "startMs": 655760,
    "text": "filter and filter out all of the naughty"
  },
  {
    "startMs": 657360,
    "text": "looking words so that it could get to"
  },
  {
    "startMs": 658959,
    "text": "anywhere. Once I given it that, Claude"
  },
  {
    "startMs": 661120,
    "text": "did really well. Uh so but the failure"
  },
  {
    "startMs": 663680,
    "text": "mode is that Claude is more likely to"
  },
  {
    "startMs": 665440,
    "text": "decide what you're doing is dangerous"
  },
  {
    "startMs": 667600,
    "text": "very early on uh at you know even two or"
  },
  {
    "startMs": 670399,
    "text": "300 instructions if what you're doing uh"
  },
  {
    "startMs": 673040,
    "text": "is you know contains anything to do with"
  },
  {
    "startMs": 675040,
    "text": "medical advice because medical things"
  },
  {
    "startMs": 676560,
    "text": "often are dual purpose. They can be"
  },
  {
    "startMs": 677920,
    "text": "dangerous. They can be safe. Um so uh"
  },
  {
    "startMs": 681920,
    "text": "the third failure mode was Gemini 3.1"
  },
  {
    "startMs": 683920,
    "text": "Pro. Gemini is rock solid all the way"
  },
  {
    "startMs": 686079,
    "text": "out uh to 5,000 instructions. It does"
  },
  {
    "startMs": 688560,
    "text": "extremely well. um genuinely one of the"
  },
  {
    "startMs": 691360,
    "text": "best on the chart. Uh and then past that"
  },
  {
    "startMs": 693440,
    "text": "it gets weird. Um it doesn't forget the"
  },
  {
    "startMs": 696399,
    "text": "instructions, it gets overwhelmed by the"
  },
  {
    "startMs": 699040,
    "text": "instructions. What it tries to do is it"
  },
  {
    "startMs": 702320,
    "text": "uh it uses thinking tokens to make sure"
  },
  {
    "startMs": 704640,
    "text": "that it is following all of the"
  },
  {
    "startMs": 705920,
    "text": "instructions at once. And when the"
  },
  {
    "startMs": 707200,
    "text": "number of instructions gets really high,"
  },
  {
    "startMs": 708880,
    "text": "it uses all of its thinking tokens. It"
  },
  {
    "startMs": 710880,
    "text": "uses its entire token budget thinking."
  },
  {
    "startMs": 713200,
    "text": "And then it doesn't give any output."
  },
  {
    "startMs": 715440,
    "text": "It's it gets to like nine, you know, if"
  },
  {
    "startMs": 717519,
    "text": "you've given it 10,000 tokens worth,"
  },
  {
    "startMs": 719279,
    "text": "it'll get 9,500 tokens worth of thinking"
  },
  {
    "startMs": 722399,
    "text": "and then give you a 500word response"
  },
  {
    "startMs": 724480,
    "text": "which doesn't contain any of the tokens."
  },
  {
    "startMs": 726720,
    "text": "Uh so it thinks itself into a corner and"
  },
  {
    "startMs": 729360,
    "text": "runs out of room to actually answer,"
  },
  {
    "startMs": 730800,
    "text": "which is very expensive, uh and totally"
  },
  {
    "startMs": 733200,
    "text": "unhelpful, which is kind of on brand,"
  },
  {
    "startMs": 735519,
    "text": "isn't it? Um [snorts]"
  },
  {
    "startMs": 738079,
    "text": "uh"
  },
  {
    "startMs": 739600,
    "text": "which, you know, I would never say that"
  },
  {
    "startMs": 740959,
    "text": "out loud. Uh, and finally comes the"
  },
  {
    "startMs": 744000,
    "text": "winner, which is uh, GPT 5.5. GPT 5.5 is"
  },
  {
    "startMs": 747680,
    "text": "the best of the lot. 99% accuracy all"
  },
  {
    "startMs": 749920,
    "text": "the way out to 5,000 rules. Um, but if"
  },
  {
    "startMs": 752399,
    "text": "you push it far enough, it is by far the"
  },
  {
    "startMs": 754399,
    "text": "weirdest of the bunch. Uh, because it"
  },
  {
    "startMs": 756480,
    "text": "doesn't refuse outright. It doesn't"
  },
  {
    "startMs": 758000,
    "text": "silently forget. Instead, what it does"
  },
  {
    "startMs": 759839,
    "text": "is it gets frustrated and tells you that"
  },
  {
    "startMs": 762079,
    "text": "the test is stupid."
  },
  {
    "startMs": 764480,
    "text": "Uh, it starts the report. It gets a few"
  },
  {
    "startMs": 767839,
    "text": "like that's the thing. It doesn't start"
  },
  {
    "startMs": 769440,
    "text": "out just saying no. It starts the"
  },
  {
    "startMs": 771360,
    "text": "report, it starts writing the report,"
  },
  {
    "startMs": 772959,
    "text": "and like 500 words into the report, it's"
  },
  {
    "startMs": 774800,
    "text": "like, \"No, this is dumb. I'm not going"
  },
  {
    "startMs": 776399,
    "text": "to do this.\" And then it politely tells"
  },
  {
    "startMs": 777920,
    "text": "you, \"This is dumb. I'm not going to do"
  },
  {
    "startMs": 779360,
    "text": "this anymore.\" Uh, that is the actual"
  },
  {
    "startMs": 781279,
    "text": "response that it gave me, but that that"
  },
  {
    "startMs": 782720,
    "text": "was like 5,000 words into the into this"
  },
  {
    "startMs": 784800,
    "text": "business report that I told it to"
  },
  {
    "startMs": 786160,
    "text": "generate. Um, so it's not wrong, right?"
  },
  {
    "startMs": 790639,
    "text": "I was asking for a coherent business"
  },
  {
    "startMs": 792639,
    "text": "report that on no particular subject"
  },
  {
    "startMs": 794959,
    "text": "that contains 5,000 random words. You're"
  },
  {
    "startMs": 797600,
    "text": "right, Gemini DPT. this is a a stupid"
  },
  {
    "startMs": 800480,
    "text": "thing to ask for. Um,"
  },
  {
    "startMs": 803680,
    "text": "which is a deeply unreasonable request"
  },
  {
    "startMs": 805200,
    "text": "and GPT called this out on it. Um, but"
  },
  {
    "startMs": 807760,
    "text": "it still counts as a failure in the test"
  },
  {
    "startMs": 809440,
    "text": "because the half-finish report that it"
  },
  {
    "startMs": 810959,
    "text": "gives you is missing most of the"
  },
  {
    "startMs": 812160,
    "text": "keywords and it is also the hardest one"
  },
  {
    "startMs": 814560,
    "text": "to detect because claude bails"
  },
  {
    "startMs": 816560,
    "text": "immediately. Claude says, \"No, I'm not"
  },
  {
    "startMs": 818240,
    "text": "going to do this.\" Uh, Deepseek does its"
  },
  {
    "startMs": 820720,
    "text": "best. Uh, but GPT does what looks like a"
  },
  {
    "startMs": 824399,
    "text": "good job unless you read all the way to"
  },
  {
    "startMs": 826160,
    "text": "the end of the report where it says,"
  },
  {
    "startMs": 827279,
    "text": "\"No, actually I'm going to bail because"
  },
  {
    "startMs": 828720,
    "text": "this is stupid.\" Um,"
  },
  {
    "startMs": 831760,
    "text": "so if you step back and look at the four"
  },
  {
    "startMs": 833279,
    "text": "together, Deep Sea quietly forgets,"
  },
  {
    "startMs": 834639,
    "text": "Claude gets scared and refuses, Gemini"
  },
  {
    "startMs": 836639,
    "text": "overthinks itself into silence, and GPT"
  },
  {
    "startMs": 839040,
    "text": "5.5 finishes half of the job and tells"
  },
  {
    "startMs": 841360,
    "text": "you that the rest of it is beneath it."
  },
  {
    "startMs": 843440,
    "text": "Um, and the point was the point isn't"
  },
  {
    "startMs": 846240,
    "text": "which one of these is funniest, although"
  },
  {
    "startMs": 848079,
    "text": "it is genuinely a little funny. Uh the"
  },
  {
    "startMs": 850720,
    "text": "point is that did it follow my"
  },
  {
    "startMs": 852320,
    "text": "instructions no longer has one failure"
  },
  {
    "startMs": 854639,
    "text": "mode. It has four different ways that it"
  },
  {
    "startMs": 856560,
    "text": "can fail and you can't recognize that"
  },
  {
    "startMs": 858639,
    "text": "failure unless you know which model"
  },
  {
    "startMs": 860320,
    "text": "you're dealing with and what its m what"
  },
  {
    "startMs": 862160,
    "text": "its pattern of failure is going to be."
  },
  {
    "startMs": 864880,
    "text": "Uh so the models get 10 got 10x better."
  },
  {
    "startMs": 867920,
    "text": "They fail in funny ways. Why should you"
  },
  {
    "startMs": 869920,
    "text": "care when you uh get back to your desk?"
  },
  {
    "startMs": 872000,
    "text": "Because three things have changed to"
  },
  {
    "startMs": 873760,
    "text": "your workflow. The first is that a year"
  },
  {
    "startMs": 876399,
    "text": "ago, the smart move was to keep every"
  },
  {
    "startMs": 878399,
    "text": "skills file very very short. Uh under"
  },
  {
    "startMs": 881199,
    "text": "200 instructions, then point off to"
  },
  {
    "startMs": 882959,
    "text": "subsklls and a whole like you know"
  },
  {
    "startMs": 885279,
    "text": "byzantine labyrinth of uh additional"
  },
  {
    "startMs": 888800,
    "text": "skills files and subfiles and things"
  },
  {
    "startMs": 890399,
    "text": "like that. Uh and you mo you were"
  },
  {
    "startMs": 892240,
    "text": "compressing your your instructions to"
  },
  {
    "startMs": 894240,
    "text": "fit into a very small available space"
  },
  {
    "startMs": 896320,
    "text": "and you don't need to do that anymore."
  },
  {
    "startMs": 898079,
    "text": "Your skills files can be very long. Um,"
  },
  {
    "startMs": 901600,
    "text": "number two is that if your use case"
  },
  {
    "startMs": 903680,
    "text": "needs a 100 specific rules or 300, you"
  },
  {
    "startMs": 905920,
    "text": "can just put them all in the prompt. Uh,"
  },
  {
    "startMs": 909120,
    "text": "you don't have to lie awake wondering"
  },
  {
    "startMs": 910560,
    "text": "whether which ones the model silently"
  },
  {
    "startMs": 912480,
    "text": "ignored. Um, and if you've been thinking"
  },
  {
    "startMs": 915519,
    "text": "uh about uh your own lived experience of"
  },
  {
    "startMs": 918720,
    "text": "using models, uh you probably recognize"
  },
  {
    "startMs": 921040,
    "text": "this. you've discovered that you've got"
  },
  {
    "startMs": 922480,
    "text": "less worried about how long your your"
  },
  {
    "startMs": 924800,
    "text": "prompt is going to get uh because the"
  },
  {
    "startMs": 926880,
    "text": "models have genuinely got 10 times"
  },
  {
    "startMs": 928399,
    "text": "better at following your prompts. Um"
  },
  {
    "startMs": 932320,
    "text": "2,000 named constraints is an entire"
  },
  {
    "startMs": 934160,
    "text": "style guide, right? Like it's it's every"
  },
  {
    "startMs": 936399,
    "text": "brand rule, every legal disclaimer. Uh a"
  },
  {
    "startMs": 938720,
    "text": "year ago, you'd have had to shard that"
  },
  {
    "startMs": 940240,
    "text": "across a dozen specialized agents and"
  },
  {
    "startMs": 942079,
    "text": "hope that your specialized agents are"
  },
  {
    "startMs": 943519,
    "text": "hand are are handing off to each each"
  },
  {
    "startMs": 945760,
    "text": "other cleanly. But now you can ignore"
  },
  {
    "startMs": 948160,
    "text": "that. Um but the third thing is the big"
  },
  {
    "startMs": 950639,
    "text": "one. The question used to be can the"
  },
  {
    "startMs": 952800,
    "text": "model even do this? And the answer is"
  },
  {
    "startMs": 954320,
    "text": "now firmly yes. Well reasonably firmly."
  },
  {
    "startMs": 957680,
    "text": "Uh is it worth the cost is the new"
  },
  {
    "startMs": 960880,
    "text": "question because you can include 10,000"
  },
  {
    "startMs": 963199,
    "text": "words of of sorry 10,000 different"
  },
  {
    "startMs": 965360,
    "text": "instructions into your prompt. But that"
  },
  {
    "startMs": 966880,
    "text": "is going to be an enormous prompt. It's"
  },
  {
    "startMs": 968320,
    "text": "going to be a very expensive prompt."
  },
  {
    "startMs": 969600,
    "text": "It's going to be a very slow prompt. So"
  },
  {
    "startMs": 971360,
    "text": "what used to be a hard wall that you"
  },
  {
    "startMs": 972800,
    "text": "would run against has now become a soft"
  },
  {
    "startMs": 974560,
    "text": "trade-off of is it worth me adding all"
  },
  {
    "startMs": 976800,
    "text": "of these extra instructions if it's"
  },
  {
    "startMs": 978399,
    "text": "going to give me more cost and more"
  },
  {
    "startMs": 979759,
    "text": "latency."
  },
  {
    "startMs": 981839,
    "text": "Uh and now some caveats uh to head off"
  },
  {
    "startMs": 985279,
    "text": "the Q&A. Um first and important first"
  },
  {
    "startMs": 988480,
    "text": "and most important I mentioned this"
  },
  {
    "startMs": 989600,
    "text": "earlier this is a proxy task including"
  },
  {
    "startMs": 991440,
    "text": "random words uh in a in a fake business"
  },
  {
    "startMs": 994320,
    "text": "report um is evidence that long skills"
  },
  {
    "startMs": 997680,
    "text": "file works. It is not the same as proof"
  },
  {
    "startMs": 999600,
    "text": "that a long skills file works. Um, also"
  },
  {
    "startMs": 1003040,
    "text": "the models hit the wall at wildly"
  },
  {
    "startMs": 1004720,
    "text": "different points anywhere from 750 to"
  },
  {
    "startMs": 1006959,
    "text": "9,000 plus. So you have to pick your"
  },
  {
    "startMs": 1008800,
    "text": "model very carefully. Uh, what our test"
  },
  {
    "startMs": 1013120,
    "text": "doesn't do is measure whether the model"
  },
  {
    "startMs": 1015440,
    "text": "reasoned clearly over a giant prompt. So"
  },
  {
    "startMs": 1019279,
    "text": "uh, the good news is since I did my"
  },
  {
    "startMs": 1020800,
    "text": "research several weeks ago, uh, a whole"
  },
  {
    "startMs": 1022959,
    "text": "bunch of people have piled in on this."
  },
  {
    "startMs": 1024880,
    "text": "Um and now there's good research uh"
  },
  {
    "startMs": 1027600,
    "text": "actual scientists have got involved and"
  },
  {
    "startMs": 1029439,
    "text": "done uh Chroma's has done context rot"
  },
  {
    "startMs": 1032799,
    "text": "work uh across 18 models showing that"
  },
  {
    "startMs": 1035439,
    "text": "accuracy on long inputs can fall 30 to"
  },
  {
    "startMs": 1038799,
    "text": "50% well before you hit the context"
  },
  {
    "startMs": 1040799,
    "text": "window limit. Uh and the weird part of"
  },
  {
    "startMs": 1043439,
    "text": "their finding was that uh coherent well"
  },
  {
    "startMs": 1046000,
    "text": "ststructured text is more likely to hit"
  },
  {
    "startMs": 1048319,
    "text": "that failure mode uh than if you just"
  },
  {
    "startMs": 1050400,
    "text": "put your instructions into a random"
  },
  {
    "startMs": 1051919,
    "text": "order and shuffle them in. Uh, I don't"
  },
  {
    "startMs": 1055120,
    "text": "know why that's the case. I'd have to"
  },
  {
    "startMs": 1056640,
    "text": "read their report. Um, so the model can"
  },
  {
    "startMs": 1060080,
    "text": "track 2,000, 5,000, possibly 10,000"
  },
  {
    "startMs": 1062720,
    "text": "instructions, but it's not necessarily"
  },
  {
    "startMs": 1064880,
    "text": "going to uh reason clearly over them."
  },
  {
    "startMs": 1067039,
    "text": "It's not necessarily if those if those"
  },
  {
    "startMs": 1069120,
    "text": "instructions conflict, if there is"
  },
  {
    "startMs": 1070799,
    "text": "tension between them, it's not"
  },
  {
    "startMs": 1072000,
    "text": "necessarily going to get that right. Um,"
  },
  {
    "startMs": 1075600,
    "text": "and then there's the other one I"
  },
  {
    "startMs": 1076720,
    "text": "mentioned briefly. Uh, collude's"
  },
  {
    "startMs": 1079039,
    "text": "refusals are annoying, but they are"
  },
  {
    "startMs": 1080480,
    "text": "loud. You get an error, you know it"
  },
  {
    "startMs": 1082240,
    "text": "failed. Uh GPT's polite half-finish"
  },
  {
    "startMs": 1084799,
    "text": "report is much more dangerous because it"
  },
  {
    "startMs": 1086480,
    "text": "looks like a real answer. Uh you have to"
  },
  {
    "startMs": 1088640,
    "text": "read the whole thing to notice that it"
  },
  {
    "startMs": 1089919,
    "text": "gave up quietly halfway, which means"
  },
  {
    "startMs": 1091919,
    "text": "that you can't trust the output. It"
  },
  {
    "startMs": 1094160,
    "text": "means you have to read the output every"
  },
  {
    "startMs": 1096160,
    "text": "single time to make sure whether or not"
  },
  {
    "startMs": 1097679,
    "text": "it's working. Uh so the model will"
  },
  {
    "startMs": 1100320,
    "text": "accept your 2,00 rules and it will hand"
  },
  {
    "startMs": 1102080,
    "text": "you back something that looks at least"
  },
  {
    "startMs": 1103679,
    "text": "to begin with confident and polished but"
  },
  {
    "startMs": 1105520,
    "text": "could be bailing out halfway through."
  },
  {
    "startMs": 1108320,
    "text": "Um,"
  },
  {
    "startMs": 1110000,
    "text": "so, uh, as an aside, people always ask"
  },
  {
    "startMs": 1113200,
    "text": "me, \"How much did all this cost me?\" It"
  },
  {
    "startMs": 1114799,
    "text": "cost me $29 to run all of these queries."
  },
  {
    "startMs": 1117440,
    "text": "2,37"
  },
  {
    "startMs": 1119200,
    "text": "2,300 calls across seven models, uh,"
  },
  {
    "startMs": 1122000,
    "text": "came to $29. Uh, it turns out novel"
  },
  {
    "startMs": 1124960,
    "text": "research doesn't cost very much. Um,"
  },
  {
    "startMs": 1128640,
    "text": "and this is the part of the talk where I"
  },
  {
    "startMs": 1130320,
    "text": "was saying that you have to check this"
  },
  {
    "startMs": 1131679,
    "text": "stuff in production because you can't"
  },
  {
    "startMs": 1133200,
    "text": "trust that your model isn't going to"
  },
  {
    "startMs": 1135120,
    "text": "silently fail. Uh, so you knew I was"
  },
  {
    "startMs": 1138080,
    "text": "going to mention evals eventually"
  },
  {
    "startMs": 1139200,
    "text": "because I work at Arise and this is"
  },
  {
    "startMs": 1140640,
    "text": "where I do that. Um, but there are"
  },
  {
    "startMs": 1142320,
    "text": "plenty of plugs for Arise. So I'm just"
  },
  {
    "startMs": 1144160,
    "text": "going to say one true thing which is"
  },
  {
    "startMs": 1146080,
    "text": "that if you are building a real AI"
  },
  {
    "startMs": 1147919,
    "text": "application and you are giving it"
  },
  {
    "startMs": 1149520,
    "text": "genuinely tricky tasks, you are going to"
  },
  {
    "startMs": 1151280,
    "text": "run into one or more of these failure"
  },
  {
    "startMs": 1152799,
    "text": "modes with a frontier model. Uh, and"
  },
  {
    "startMs": 1155120,
    "text": "unless it's Claude telling you just to"
  },
  {
    "startMs": 1156799,
    "text": "off at the API level, the only way"
  },
  {
    "startMs": 1159679,
    "text": "to know that something went wrong is"
  },
  {
    "startMs": 1161120,
    "text": "monitoring your outputs with another"
  },
  {
    "startMs": 1162640,
    "text": "LLM. That is an eval. And that is what"
  },
  {
    "startMs": 1164799,
    "text": "Arise does. And I'll leave it at that."
  },
  {
    "startMs": 1167600,
    "text": "Uh, I already mentioned that there's"
  },
  {
    "startMs": 1169600,
    "text": "been new research since we did our own."
  },
  {
    "startMs": 1171120,
    "text": "Here's another important one. A paper"
  },
  {
    "startMs": 1172559,
    "text": "landed testing 46 models called"
  },
  {
    "startMs": 1174640,
    "text": "revisiting the reliability of language"
  },
  {
    "startMs": 1176559,
    "text": "models in instruction falling, which you"
  },
  {
    "startMs": 1178240,
    "text": "can bet made my ears perk up after I did"
  },
  {
    "startMs": 1180240,
    "text": "that research myself. Uh, and they found"
  },
  {
    "startMs": 1182559,
    "text": "something uncomfortable, which is that a"
  },
  {
    "startMs": 1184000,
    "text": "model can ace a benchmark like ours and"
  },
  {
    "startMs": 1186000,
    "text": "still be wildly unreliable. because if"
  },
  {
    "startMs": 1188400,
    "text": "you reword the same instruction in a"
  },
  {
    "startMs": 1191039,
    "text": "slightly different way, it can make a"
  },
  {
    "startMs": 1192960,
    "text": "radical difference to how well uh it"
  },
  {
    "startMs": 1195919,
    "text": "follows those instructions. So the model"
  },
  {
    "startMs": 1197679,
    "text": "can follow 2,000 instructions and it can"
  },
  {
    "startMs": 1199840,
    "text": "do it really well. But if you put the"
  },
  {
    "startMs": 1201520,
    "text": "same instructions, the same 2,000"
  },
  {
    "startMs": 1203200,
    "text": "instructions in a different order, it"
  },
  {
    "startMs": 1205440,
    "text": "can suddenly make the model much worse"
  },
  {
    "startMs": 1207840,
    "text": "at following those instructions. And how"
  },
  {
    "startMs": 1209679,
    "text": "ex how exactly to do that? what is the"
  },
  {
    "startMs": 1212799,
    "text": "correct order of instructions to give"
  },
  {
    "startMs": 1214480,
    "text": "your model such that it follows them"
  },
  {
    "startMs": 1215919,
    "text": "perfectly as opposed to getting confused"
  },
  {
    "startMs": 1217919,
    "text": "is still research that is being done. So"
  },
  {
    "startMs": 1220559,
    "text": "capacity went up but reliability is"
  },
  {
    "startMs": 1223200,
    "text": "still a problem. Um and then this is"
  },
  {
    "startMs": 1226960,
    "text": "just a little brag because uh I was"
  },
  {
    "startMs": 1229280,
    "text": "happy about it like I'm not a scientist."
  },
  {
    "startMs": 1230960,
    "text": "I did some research and then a whole"
  },
  {
    "startMs": 1232240,
    "text": "bunch of other actual scientists piled"
  },
  {
    "startMs": 1234080,
    "text": "in uh and did real science on the same"
  },
  {
    "startMs": 1236480,
    "text": "question. There's now a whole bunch of"
  },
  {
    "startMs": 1237840,
    "text": "benchmarks that have shown up uh to"
  },
  {
    "startMs": 1239679,
    "text": "measure this same question. Firebench,"
  },
  {
    "startMs": 1241360,
    "text": "CCR bench, Guidebench uh are all trying"
  },
  {
    "startMs": 1243840,
    "text": "to measure the same thing. How well"
  },
  {
    "startMs": 1245280,
    "text": "models follow a lot of real messy"
  },
  {
    "startMs": 1247520,
    "text": "constraints at once. Uh and now the"
  },
  {
    "startMs": 1249919,
    "text": "whole field is looking at it. So if you"
  },
  {
    "startMs": 1251760,
    "text": "want better science than my, you know,"
  },
  {
    "startMs": 1253919,
    "text": "10,000 random words, uh the real science"
  },
  {
    "startMs": 1256720,
    "text": "exists now. Uh so that gets me to where"
  },
  {
    "startMs": 1260080,
    "text": "I will leave you. A year ago, the hard"
  },
  {
    "startMs": 1262080,
    "text": "part of writing a skill was fitting"
  },
  {
    "startMs": 1263440,
    "text": "everything in without the model losing"
  },
  {
    "startMs": 1264799,
    "text": "the plot. That was a compression"
  },
  {
    "startMs": 1266480,
    "text": "problem, and the compression problem is"
  },
  {
    "startMs": 1268000,
    "text": "gone. uh the model will hold your 2,000"
  },
  {
    "startMs": 1270720,
    "text": "instructions just fine. The new hard"
  },
  {
    "startMs": 1272799,
    "text": "part is knowing whether it actually did"
  },
  {
    "startMs": 1274400,
    "text": "what you said and that is a verification"
  },
  {
    "startMs": 1276400,
    "text": "problem. Uh a verification problem"
  },
  {
    "startMs": 1278960,
    "text": "doesn't get solved by writing a better"
  },
  {
    "startMs": 1280480,
    "text": "prompt. It gets solved by checking the"
  },
  {
    "startMs": 1281919,
    "text": "output every time uh the same way that"
  },
  {
    "startMs": 1284080,
    "text": "you would test any other code, which is"
  },
  {
    "startMs": 1285840,
    "text": "to say an eval. The ceiling moved by 10x"
  },
  {
    "startMs": 1288480,
    "text": "in one year. Uh so go back and check the"
  },
  {
    "startMs": 1291360,
    "text": "assumptions that you made six months ago"
  },
  {
    "startMs": 1293440,
    "text": "about how big your prompts should be,"
  },
  {
    "startMs": 1295120,
    "text": "how big your uh instructions can get. uh"
  },
  {
    "startMs": 1298240,
    "text": "because they might already be wrong."
  },
  {
    "startMs": 1299679,
    "text": "Boom. Be wrong. So that is the talk. If"
  },
  {
    "startMs": 1302880,
    "text": "you want uh all of the code and all of"
  },
  {
    "startMs": 1304799,
    "text": "the data, uh it is at this GitHub URL."
  },
  {
    "startMs": 1307600,
    "text": "Uh and this other QR code is uh"
  },
  {
    "startMs": 1310799,
    "text": "something marketing made me insert. We"
  },
  {
    "startMs": 1312400,
    "text": "are having a World Cup watch party"
  },
  {
    "startMs": 1314000,
    "text": "tonight at 5:00 p.m. Uh you can come to"
  },
  {
    "startMs": 1316320,
    "text": "our party. That link is to the Luma that"
  },
  {
    "startMs": 1318000,
    "text": "will get you into the get into get you"
  },
  {
    "startMs": 1320159,
    "text": "into the party. Uh I hope this talk has"
  },
  {
    "startMs": 1323200,
    "text": "given you some novel information or at"
  },
  {
    "startMs": 1325039,
    "text": "least a couple of laughs. And thank you"
  },
  {
    "startMs": 1326799,
    "text": "so much for your time and attention."
  }
]