[
  {
    "startMs": 12639,
    "text": "Yep. Awesome. Thanks everyone for being"
  },
  {
    "startMs": 15440,
    "text": "here. My name is Kevin Madura. I'm from"
  },
  {
    "startMs": 17520,
    "text": "a company called Alex Partners. We're"
  },
  {
    "startMs": 19199,
    "text": "we're a consulting firm. Um I'm here to"
  },
  {
    "startMs": 21439,
    "text": "talk to you today about RLMs. Just"
  },
  {
    "startMs": 23920,
    "text": "curious, show of hands, who here is"
  },
  {
    "startMs": 26480,
    "text": "familiar with RLMs? So we know how much"
  },
  {
    "startMs": 29039,
    "text": "time to spend on it. Okay. So not many."
  },
  {
    "startMs": 31199,
    "text": "All right. Well, that's good. So we'll"
  },
  {
    "startMs": 33440,
    "text": "start with what an RLM is and and why"
  },
  {
    "startMs": 35600,
    "text": "it's different. So RLM is recursive"
  },
  {
    "startMs": 38160,
    "text": "language model. And really the the key"
  },
  {
    "startMs": 40320,
    "text": "difference here is that it treats the"
  },
  {
    "startMs": 43600,
    "text": "context as an object that it can"
  },
  {
    "startMs": 46079,
    "text": "interact with symbolically in its"
  },
  {
    "startMs": 47840,
    "text": "environment. So it differs from a tool"
  },
  {
    "startMs": 50800,
    "text": "call in the sense that typically when"
  },
  {
    "startMs": 52320,
    "text": "you do a tool call it's JSON or some"
  },
  {
    "startMs": 55039,
    "text": "type of string that's being sent being"
  },
  {
    "startMs": 58320,
    "text": "interpreted elsewhere maybe by some"
  },
  {
    "startMs": 60480,
    "text": "other program and that's that's"
  },
  {
    "startMs": 62000,
    "text": "returning effectively as a string. The"
  },
  {
    "startMs": 64159,
    "text": "key difference here is that it's"
  },
  {
    "startMs": 66000,
    "text": "interacting with a symbolic environment."
  },
  {
    "startMs": 68320,
    "text": "So typically that's a ripple Python"
  },
  {
    "startMs": 70240,
    "text": "ripple. Um so that's key difference"
  },
  {
    "startMs": 73520,
    "text": "number one. Key difference number two is"
  },
  {
    "startMs": 75439,
    "text": "that it's it has the ability to delegate"
  },
  {
    "startMs": 78000,
    "text": "to another LM often to itself. You can"
  },
  {
    "startMs": 80560,
    "text": "specify whether it's the same model or a"
  },
  {
    "startMs": 83200,
    "text": "different model, but fundamentally"
  },
  {
    "startMs": 84799,
    "text": "because it lives in this environment,"
  },
  {
    "startMs": 86720,
    "text": "you can offload or or make a sub call to"
  },
  {
    "startMs": 89280,
    "text": "another LM with particular parameters"
  },
  {
    "startMs": 91920,
    "text": "that also lives in that ripple"
  },
  {
    "startMs": 93360,
    "text": "environment. And so you get this ability"
  },
  {
    "startMs": 96560,
    "text": "to recursively decompose problems and"
  },
  {
    "startMs": 100400,
    "text": "apply and have the LLM basically decide"
  },
  {
    "startMs": 102799,
    "text": "how to apply certain logic or certain"
  },
  {
    "startMs": 105520,
    "text": "interpretations or write its own code to"
  },
  {
    "startMs": 108079,
    "text": "solve those problems and then that"
  },
  {
    "startMs": 110640,
    "text": "recurses down. So the subLM can do the"
  },
  {
    "startMs": 112880,
    "text": "same sort of thing in terms of"
  },
  {
    "startMs": 113920,
    "text": "understanding and interpreting what it"
  },
  {
    "startMs": 115840,
    "text": "thinks it needs to do. And I added this"
  },
  {
    "startMs": 118399,
    "text": "last one here. It's largely bitter"
  },
  {
    "startMs": 120960,
    "text": "lesson pilled in my opinion, right? is"
  },
  {
    "startMs": 123520,
    "text": "and and shared by Alex and the rest of"
  },
  {
    "startMs": 125360,
    "text": "the the creators of it. But as models"
  },
  {
    "startMs": 128239,
    "text": "get better, you should be able to defer"
  },
  {
    "startMs": 130479,
    "text": "more and more to the model for it to"
  },
  {
    "startMs": 133760,
    "text": "kind of figure out on its own what it"
  },
  {
    "startMs": 135200,
    "text": "needs to do."
  },
  {
    "startMs": 137440,
    "text": "So the I don't know if this is the"
  },
  {
    "startMs": 139040,
    "text": "actual kind of starting point for RLMs."
  },
  {
    "startMs": 142720,
    "text": "This is one that that I consider to be"
  },
  {
    "startMs": 144720,
    "text": "one of the the first kind of inklings of"
  },
  {
    "startMs": 146160,
    "text": "it. This is a tweet from Omar who is"
  },
  {
    "startMs": 148560,
    "text": "Alex's adviser um for for RLMs. And this"
  },
  {
    "startMs": 152160,
    "text": "was a concept that he had come up with"
  },
  {
    "startMs": 154000,
    "text": "where it was basically an ability to use"
  },
  {
    "startMs": 157280,
    "text": "DSPI and some other techniques to take"
  },
  {
    "startMs": 159760,
    "text": "in arbitrary"
  },
  {
    "startMs": 161680,
    "text": "um arbitrary length inputs. And"
  },
  {
    "startMs": 163519,
    "text": "basically the use case here would be"
  },
  {
    "startMs": 165040,
    "text": "summarizing an arbitrarily long document"
  },
  {
    "startMs": 167680,
    "text": "and coming up with a table of contents"
  },
  {
    "startMs": 169680,
    "text": "and some summary of of that content. But"
  },
  {
    "startMs": 173120,
    "text": "at least to me, this is kind of the"
  },
  {
    "startMs": 174400,
    "text": "first inkling of okay, context windows"
  },
  {
    "startMs": 176959,
    "text": "might not be something you need to"
  },
  {
    "startMs": 179120,
    "text": "deliberately manage. Although there's of"
  },
  {
    "startMs": 180640,
    "text": "course benefits to doing so, uh there"
  },
  {
    "startMs": 182959,
    "text": "there could be ways to um to exceed the"
  },
  {
    "startMs": 185760,
    "text": "the the context windows using some of"
  },
  {
    "startMs": 187519,
    "text": "these clever techniques."
  },
  {
    "startMs": 190159,
    "text": "Uh and so if you read the paper and some"
  },
  {
    "startMs": 192159,
    "text": "of the blog posts that are out there"
  },
  {
    "startMs": 193200,
    "text": "from Alex and Omar, I mean it it it has"
  },
  {
    "startMs": 196080,
    "text": "demonstrabably better performance on"
  },
  {
    "startMs": 198480,
    "text": "some of these long context tasks. So,"
  },
  {
    "startMs": 201040,
    "text": "ulong is one benchmark where um the the"
  },
  {
    "startMs": 205280,
    "text": "intent of the benchmark it's is to"
  },
  {
    "startMs": 206720,
    "text": "measure model performance on answering"
  },
  {
    "startMs": 208400,
    "text": "questions about excessively long"
  },
  {
    "startMs": 209840,
    "text": "context. This another one browse comp"
  },
  {
    "startMs": 212159,
    "text": "where it needs to iterate through a"
  },
  {
    "startMs": 214959,
    "text": "large a large body and corpus of text"
  },
  {
    "startMs": 216959,
    "text": "and answer particular questions about"
  },
  {
    "startMs": 218400,
    "text": "it. You can see the blue line at the top"
  },
  {
    "startMs": 220239,
    "text": "there is the RLM. It's very good"
  },
  {
    "startMs": 222959,
    "text": "performance as compared to some of these"
  },
  {
    "startMs": 224480,
    "text": "other models. And even on on the price"
  },
  {
    "startMs": 226560,
    "text": "curve, the the purple is actually just"
  },
  {
    "startMs": 229519,
    "text": "using tool calling um with GBT5 calling"
  },
  {
    "startMs": 232560,
    "text": "a BM25"
  },
  {
    "startMs": 234560,
    "text": "tool. And that's actually even more"
  },
  {
    "startMs": 236319,
    "text": "expensive for worse performance than"
  },
  {
    "startMs": 238560,
    "text": "than an RLM. So it's worth reading into"
  },
  {
    "startMs": 241439,
    "text": "if you're interested in in some of the"
  },
  {
    "startMs": 242879,
    "text": "benchmarks and how RLMs perform. Um but"
  },
  {
    "startMs": 245760,
    "text": "fundamentally uh an RLM again takes in"
  },
  {
    "startMs": 249280,
    "text": "your input and you're kind of deferring"
  },
  {
    "startMs": 250799,
    "text": "to the model about how to decompose the"
  },
  {
    "startMs": 253280,
    "text": "process what code it needs to write and"
  },
  {
    "startMs": 256400,
    "text": "it is very tightly integrated with the"
  },
  {
    "startMs": 258320,
    "text": "ripple itself. So it it by itself"
  },
  {
    "startMs": 260720,
    "text": "defines what it needs to do. And so I"
  },
  {
    "startMs": 262560,
    "text": "kind of had this mental model in terms"
  },
  {
    "startMs": 264320,
    "text": "of and I'm very DSP pipel if if you"
  },
  {
    "startMs": 267120,
    "text": "couldn't tell by now basically a student"
  },
  {
    "startMs": 269360,
    "text": "of of Omar and the rest of the group"
  },
  {
    "startMs": 271040,
    "text": "there where you have this relatively"
  },
  {
    "startMs": 273919,
    "text": "deterministic shell of what you want to"
  },
  {
    "startMs": 276000,
    "text": "do like what what is your intent what is"
  },
  {
    "startMs": 278560,
    "text": "your actual task that you're trying to"
  },
  {
    "startMs": 280080,
    "text": "accomplish. You define that in terms of"
  },
  {
    "startMs": 282479,
    "text": "your inputs and your outputs and some"
  },
  {
    "startMs": 284800,
    "text": "type of guidance or prompt or what have"
  },
  {
    "startMs": 287199,
    "text": "you to the model to say this is"
  },
  {
    "startMs": 289280,
    "text": "generally what I want to achieve. go off"
  },
  {
    "startMs": 291120,
    "text": "and do it. Here's the things that you"
  },
  {
    "startMs": 293040,
    "text": "can expect as your input. Here's what I"
  },
  {
    "startMs": 294880,
    "text": "want out of it. Go figure out the rest."
  },
  {
    "startMs": 297120,
    "text": "And so this applies for using something"
  },
  {
    "startMs": 299040,
    "text": "like DSPI, but I think it applies to"
  },
  {
    "startMs": 300639,
    "text": "RLMs as well because you don't have to"
  },
  {
    "startMs": 303360,
    "text": "worry as much now about how the actual"
  },
  {
    "startMs": 307199,
    "text": "implementation works in the middle. You"
  },
  {
    "startMs": 310240,
    "text": "can just have some guarantees about the"
  },
  {
    "startMs": 311759,
    "text": "inputs and the outputs and you can let"
  },
  {
    "startMs": 312960,
    "text": "the model figure out the rest of of that"
  },
  {
    "startMs": 315680,
    "text": "part of it."
  },
  {
    "startMs": 317120,
    "text": "So, a lot of this comes down to if if"
  },
  {
    "startMs": 318960,
    "text": "you were at um I think it was code in"
  },
  {
    "startMs": 321600,
    "text": "November in uh in New York City, Dex had"
  },
  {
    "startMs": 324000,
    "text": "this great talk about just broader"
  },
  {
    "startMs": 325919,
    "text": "context engineering and he he coined"
  },
  {
    "startMs": 327600,
    "text": "something like the dumb zone which it's"
  },
  {
    "startMs": 330400,
    "text": "kind of grayed out at the bottom there."
  },
  {
    "startMs": 331680,
    "text": "But the point is that we all know that"
  },
  {
    "startMs": 333840,
    "text": "there's context rot, right? Once you"
  },
  {
    "startMs": 335440,
    "text": "fill up the context window to a certain"
  },
  {
    "startMs": 337360,
    "text": "degree, performance starts to degrade."
  },
  {
    "startMs": 339919,
    "text": "And so RLMs somewhat get around this"
  },
  {
    "startMs": 342960,
    "text": "problem because the context itself"
  },
  {
    "startMs": 346639,
    "text": "doesn't fill up as quickly because"
  },
  {
    "startMs": 347919,
    "text": "you're deferring a lot of the subtasks"
  },
  {
    "startMs": 349520,
    "text": "to the subm models and it's the full"
  },
  {
    "startMs": 353280,
    "text": "kind of context and the inputs aren't"
  },
  {
    "startMs": 355840,
    "text": "exposed to the context window itself. It"
  },
  {
    "startMs": 358479,
    "text": "lives as a variable in the ripple and so"
  },
  {
    "startMs": 360880,
    "text": "the main LM can choose how to um how to"
  },
  {
    "startMs": 364800,
    "text": "access that. It can offload some of"
  },
  {
    "startMs": 366720,
    "text": "these subtasks to to sublim. And really"
  },
  {
    "startMs": 369280,
    "text": "the only context that it gets back are"
  },
  {
    "startMs": 371280,
    "text": "the things that actually matter. So in"
  },
  {
    "startMs": 373919,
    "text": "terms of how it's meaningfully"
  },
  {
    "startMs": 375039,
    "text": "different, rag of course you kind of"
  },
  {
    "startMs": 376960,
    "text": "just stuff the context window. You want"
  },
  {
    "startMs": 378560,
    "text": "it to limit there. Agents are largely"
  },
  {
    "startMs": 381759,
    "text": "just bringing strings back and you don't"
  },
  {
    "startMs": 384479,
    "text": "have this tight coupling between the"
  },
  {
    "startMs": 386960,
    "text": "logic, the execution and the results."
  },
  {
    "startMs": 389919,
    "text": "And so you still run into the same sort"
  },
  {
    "startMs": 391600,
    "text": "of problem there. Same thing with tool"
  },
  {
    "startMs": 393759,
    "text": "calling and and codec. And then RLMs as"
  },
  {
    "startMs": 397440,
    "text": "I mentioned you're act the the LLM is"
  },
  {
    "startMs": 400560,
    "text": "actually just interacting with the"
  },
  {
    "startMs": 402560,
    "text": "context the results as variables in the"
  },
  {
    "startMs": 404720,
    "text": "ripple so that it can do additional"
  },
  {
    "startMs": 406560,
    "text": "computation on versus it trying to"
  },
  {
    "startMs": 409360,
    "text": "attend to all these different tokens in"
  },
  {
    "startMs": 410800,
    "text": "this in its context window. It's it's a"
  },
  {
    "startMs": 412400,
    "text": "meaningfully different way of of the LLM"
  },
  {
    "startMs": 414720,
    "text": "interacting with the uh the actual"
  },
  {
    "startMs": 417280,
    "text": "content itself."
  },
  {
    "startMs": 419120,
    "text": "And so people always say, okay, what's"
  },
  {
    "startMs": 420960,
    "text": "the difference between that and and"
  },
  {
    "startMs": 422000,
    "text": "encoding agents? Um,"
  },
  {
    "startMs": 425280,
    "text": "in my mind, the largest difference is"
  },
  {
    "startMs": 427039,
    "text": "that the way that tool calls are tool"
  },
  {
    "startMs": 429520,
    "text": "calls calls are done is passing strings"
  },
  {
    "startMs": 432319,
    "text": "back and forth. Um but you can see with"
  },
  {
    "startMs": 434639,
    "text": "the release recently of workflows that"
  },
  {
    "startMs": 438400,
    "text": "uh Anthropic is doing something fairly"
  },
  {
    "startMs": 440400,
    "text": "similar and they um at the CIS"
  },
  {
    "startMs": 443440,
    "text": "conference I think it was Tar or someone"
  },
  {
    "startMs": 445599,
    "text": "similar um mentioned the RLM paper as a"
  },
  {
    "startMs": 450479,
    "text": "key driver of workflows and how they're"
  },
  {
    "startMs": 453039,
    "text": "how they've implemented it. And you can"
  },
  {
    "startMs": 455360,
    "text": "see here the intermediate results for"
  },
  {
    "startMs": 458080,
    "text": "workflows live in script variables i.e."
  },
  {
    "startMs": 462639,
    "text": "a variable in the context. So, it's"
  },
  {
    "startMs": 464639,
    "text": "driving some of these these"
  },
  {
    "startMs": 466000,
    "text": "breakthroughs and some of these"
  },
  {
    "startMs": 467120,
    "text": "techniques from the from the labs as"
  },
  {
    "startMs": 468479,
    "text": "well."
  },
  {
    "startMs": 473199,
    "text": "I'll skip through this a bit just"
  },
  {
    "startMs": 474560,
    "text": "because I have about 10 minutes left,"
  },
  {
    "startMs": 476560,
    "text": "but generally speaking, when you want to"
  },
  {
    "startMs": 478319,
    "text": "use it, it's obviously for large or"
  },
  {
    "startMs": 480240,
    "text": "dense input context."
  },
  {
    "startMs": 482879,
    "text": "An underexplored area is outputs as"
  },
  {
    "startMs": 485440,
    "text": "well. So if you have a some type of task"
  },
  {
    "startMs": 489440,
    "text": "where you need to generate hundreds of"
  },
  {
    "startMs": 491199,
    "text": "thousands of lines or whatever it might"
  },
  {
    "startMs": 492720,
    "text": "be, RLMs I think would be a good"
  },
  {
    "startMs": 494800,
    "text": "candidate for that as well. Obviously"
  },
  {
    "startMs": 497120,
    "text": "tasks that are imunable to some type of"
  },
  {
    "startMs": 498879,
    "text": "decomposition. So if you want to look"
  },
  {
    "startMs": 501120,
    "text": "through the entire I don't know the"
  },
  {
    "startMs": 503120,
    "text": "whole tax code as an example and try and"
  },
  {
    "startMs": 505840,
    "text": "find loopholes or something. You can't"
  },
  {
    "startMs": 507840,
    "text": "obviously put all of that into context"
  },
  {
    "startMs": 509360,
    "text": "at once. You could use an LLM to crunch"
  },
  {
    "startMs": 512560,
    "text": "through all of that and iteratively"
  },
  {
    "startMs": 514159,
    "text": "explore and use sub agents to explore"
  },
  {
    "startMs": 516719,
    "text": "interesting areas of this of the tax"
  },
  {
    "startMs": 518320,
    "text": "code. Bring back those sections and then"
  },
  {
    "startMs": 520399,
    "text": "reason over that and then just generally"
  },
  {
    "startMs": 522640,
    "text": "for for longer horizon sessions. And"
  },
  {
    "startMs": 525040,
    "text": "when you want to skip it, of course, it'"
  },
  {
    "startMs": 526560,
    "text": "be something that fits in context. You"
  },
  {
    "startMs": 528560,
    "text": "want something that's low latency or the"
  },
  {
    "startMs": 530720,
    "text": "the model itself is as strong of a"
  },
  {
    "startMs": 532480,
    "text": "coder."
  },
  {
    "startMs": 534480,
    "text": "And uh our friend Raymond here did some"
  },
  {
    "startMs": 536399,
    "text": "great perfor performance testing on the"
  },
  {
    "startMs": 538640,
    "text": "long chain of thought benchmark. Um I'll"
  },
  {
    "startMs": 541760,
    "text": "leave this link as a as a leave behind"
  },
  {
    "startMs": 543120,
    "text": "after, but just to give you a sense of"
  },
  {
    "startMs": 544880,
    "text": "how well it performs on some of these"
  },
  {
    "startMs": 546480,
    "text": "tasks. It's a meaningful jump overall"
  },
  {
    "startMs": 549279,
    "text": "from from 2.6 to 45.4%"
  },
  {
    "startMs": 552800,
    "text": "um accuracy on many of these tasks. And"
  },
  {
    "startMs": 554640,
    "text": "you can see it performs really well on"
  },
  {
    "startMs": 556800,
    "text": "things that are amanable to code. So"
  },
  {
    "startMs": 558959,
    "text": "logic puzzles and and chess and"
  },
  {
    "startMs": 560720,
    "text": "chemistry and things like that where it"
  },
  {
    "startMs": 563120,
    "text": "can dynamically write code bring in only"
  },
  {
    "startMs": 566800,
    "text": "the relevant part of the context compute"
  },
  {
    "startMs": 569040,
    "text": "that and then return the result where"
  },
  {
    "startMs": 571040,
    "text": "the main model is really just harvesting"
  },
  {
    "startMs": 572720,
    "text": "the results from from the subLM and try"
  },
  {
    "startMs": 574640,
    "text": "instead of trying to do that by itself."
  },
  {
    "startMs": 578160,
    "text": "Um I put together a few just super"
  },
  {
    "startMs": 580320,
    "text": "simple examples. I mean, these are kind"
  },
  {
    "startMs": 582080,
    "text": "of they're somewhat unfair, I suppose,"
  },
  {
    "startMs": 584000,
    "text": "to to the base model, but it it makes"
  },
  {
    "startMs": 586880,
    "text": "the point that there are certain tasks"
  },
  {
    "startMs": 588480,
    "text": "that base models just aren't really fit"
  },
  {
    "startMs": 591440,
    "text": "to do themselves because they because"
  },
  {
    "startMs": 593519,
    "text": "they have to attend all these to"
  },
  {
    "startMs": 594800,
    "text": "different tokens at once in the context"
  },
  {
    "startMs": 596320,
    "text": "window where you need or want to use"
  },
  {
    "startMs": 599120,
    "text": "some type of coding approach to that. So"
  },
  {
    "startMs": 602240,
    "text": "in this random example, summing 12"
  },
  {
    "startMs": 604000,
    "text": "numbers that are buried across 30,000"
  },
  {
    "startMs": 605839,
    "text": "tokens, the LLM trying to figure all"
  },
  {
    "startMs": 609120,
    "text": "that out by itself and give you the"
  },
  {
    "startMs": 610640,
    "text": "answer isn't always going to work as"
  },
  {
    "startMs": 612640,
    "text": "well as something that you can write reg"
  },
  {
    "startMs": 614880,
    "text": "x4 or something similar. And then the"
  },
  {
    "startMs": 617279,
    "text": "same same sort of thing particularly for"
  },
  {
    "startMs": 619040,
    "text": "data frames and we'll walk through a"
  },
  {
    "startMs": 620320,
    "text": "brief example here where because the LLM"
  },
  {
    "startMs": 623360,
    "text": "can"
  },
  {
    "startMs": 625040,
    "text": "interact with the data frame within the"
  },
  {
    "startMs": 627600,
    "text": "ripple. It just has a much better"
  },
  {
    "startMs": 629920,
    "text": "understanding of the content and can"
  },
  {
    "startMs": 631200,
    "text": "iterate through that much more quickly"
  },
  {
    "startMs": 632880,
    "text": "than having to pass tool calls back and"
  },
  {
    "startMs": 635360,
    "text": "forth in terms of like JSON strings and"
  },
  {
    "startMs": 637040,
    "text": "and that sort of thing."
  },
  {
    "startMs": 639360,
    "text": "Um, and then I threw this in there in"
  },
  {
    "startMs": 641440,
    "text": "terms of running the same experiments"
  },
  {
    "startMs": 643040,
    "text": "with a coding agent. Now, I didn't look"
  },
  {
    "startMs": 645680,
    "text": "into this too deeply. There's probably"
  },
  {
    "startMs": 647200,
    "text": "some unfair math going on here, but you"
  },
  {
    "startMs": 649440,
    "text": "can see that it was totally bloated in"
  },
  {
    "startMs": 651680,
    "text": "terms of the way that cloud code tried"
  },
  {
    "startMs": 653680,
    "text": "to um tried to solve these tasks. So,"
  },
  {
    "startMs": 657839,
    "text": "there's more work to be done there, of"
  },
  {
    "startMs": 659200,
    "text": "course, in terms of like running"
  },
  {
    "startMs": 661839,
    "text": "experiments to compare base models"
  },
  {
    "startMs": 663839,
    "text": "versus RLMs versus something like a"
  },
  {
    "startMs": 666079,
    "text": "coding agent. But there's just for"
  },
  {
    "startMs": 667600,
    "text": "certain tasks for like production"
  },
  {
    "startMs": 669120,
    "text": "workloads. My sense is you probably"
  },
  {
    "startMs": 671680,
    "text": "don't want to just do cloud-p"
  },
  {
    "startMs": 674320,
    "text": "your prompt and like hope for a good"
  },
  {
    "startMs": 676079,
    "text": "result. Like you want more of a"
  },
  {
    "startMs": 677360,
    "text": "structured approach to your inputs, your"
  },
  {
    "startMs": 680079,
    "text": "outputs and you want a defined pipeline"
  },
  {
    "startMs": 681760,
    "text": "for doing so which reduces your cost, it"
  },
  {
    "startMs": 684399,
    "text": "reduces your complexity, reduces your"
  },
  {
    "startMs": 685920,
    "text": "bloat, all that sort of thing. Um where"
  },
  {
    "startMs": 688720,
    "text": "RLMs can can shine."
  },
  {
    "startMs": 692079,
    "text": "So in the real world there are a bunch"
  },
  {
    "startMs": 694959,
    "text": "of different open source libraries that"
  },
  {
    "startMs": 696560,
    "text": "implement RLMs at some level. Some of"
  },
  {
    "startMs": 698880,
    "text": "them are more RLM focused uh like a"
  },
  {
    "startMs": 702800,
    "text": "predict RLM would be a good example of"
  },
  {
    "startMs": 704399,
    "text": "that versus others are kind of just"
  },
  {
    "startMs": 706480,
    "text": "integrating it into the broader approach"
  },
  {
    "startMs": 708800,
    "text": "or the broader framework. DSPI obviously"
  },
  {
    "startMs": 711920,
    "text": "uh there's axe which is really"
  },
  {
    "startMs": 713360,
    "text": "interesting work uh that's being done"
  },
  {
    "startMs": 715040,
    "text": "there. Predict RLM is more focused on"
  },
  {
    "startMs": 717760,
    "text": "like knowledge work. So it works with"
  },
  {
    "startMs": 719440,
    "text": "spreadsheets and PDFs and that sort of"
  },
  {
    "startMs": 721600,
    "text": "thing and then fast RLM. And then"
  },
  {
    "startMs": 724320,
    "text": "there's a tweet yesterday from this guy"
  },
  {
    "startMs": 726000,
    "text": "Sam Hogan"
  },
  {
    "startMs": 727839,
    "text": "um where who runs inference.net. He's"
  },
  {
    "startMs": 729920,
    "text": "using an RLM to basically run and"
  },
  {
    "startMs": 733120,
    "text": "extract um insights from your particular"
  },
  {
    "startMs": 736880,
    "text": "um production workload traces so that"
  },
  {
    "startMs": 739760,
    "text": "they can see what makes sense to defer"
  },
  {
    "startMs": 742000,
    "text": "off to something like a GLM 5.2 too and"
  },
  {
    "startMs": 745040,
    "text": "do that iteratively and automatically as"
  },
  {
    "startMs": 746959,
    "text": "your traffic goes through. So point"
  },
  {
    "startMs": 749200,
    "text": "being, you don't have to worry about"
  },
  {
    "startMs": 752000,
    "text": "context engineering. You can kind of"
  },
  {
    "startMs": 753519,
    "text": "just throw the RLM at it and have it"
  },
  {
    "startMs": 755200,
    "text": "figure it out. Um, I only have five"
  },
  {
    "startMs": 758240,
    "text": "minutes left, so we won't go through"
  },
  {
    "startMs": 759360,
    "text": "this whole example and I'll I'll skip to"
  },
  {
    "startMs": 761120,
    "text": "some of the traces because that's"
  },
  {
    "startMs": 762160,
    "text": "probably the most interesting. Um, but"
  },
  {
    "startMs": 764320,
    "text": "this is all you would really need to do"
  },
  {
    "startMs": 766880,
    "text": "in terms of a simple, in this case it's"
  },
  {
    "startMs": 769839,
    "text": "like a a cohort retention analysis,"
  },
  {
    "startMs": 771839,
    "text": "something that you might give to a data"
  },
  {
    "startMs": 773120,
    "text": "scientist. But this concept of applying"
  },
  {
    "startMs": 776720,
    "text": "an RLM to a complex data structure like"
  },
  {
    "startMs": 779839,
    "text": "a data frame becomes very easy to do."
  },
  {
    "startMs": 782240,
    "text": "This is all the code you need to do it"
  },
  {
    "startMs": 784000,
    "text": "where I'm feeding in three different"
  },
  {
    "startMs": 786000,
    "text": "data frames. I'm saying these are the"
  },
  {
    "startMs": 788079,
    "text": "sorts of things you need to look for."
  },
  {
    "startMs": 789600,
    "text": "These are the output types that I want."
  },
  {
    "startMs": 791839,
    "text": "and then just let the RLM go on it. And"
  },
  {
    "startMs": 795120,
    "text": "I'll I'll show you some of the traces."
  },
  {
    "startMs": 798079,
    "text": "Um, and so it has its own ripple where"
  },
  {
    "startMs": 801600,
    "text": "it can interact with those data frames."
  },
  {
    "startMs": 803600,
    "text": "And you can see it reasoning through."
  },
  {
    "startMs": 805040,
    "text": "Okay, first I need to do this. It's"
  },
  {
    "startMs": 806720,
    "text": "writing the code. And because it's li"
  },
  {
    "startMs": 808880,
    "text": "it's living in the ripple with the dataf"
  },
  {
    "startMs": 810959,
    "text": "frame, you don't have this additional"
  },
  {
    "startMs": 812959,
    "text": "bloat of the tool calls back and forth."
  },
  {
    "startMs": 814959,
    "text": "It's actually interacting directly with"
  },
  {
    "startMs": 816959,
    "text": "the dataf frame as if it was typing in"
  },
  {
    "startMs": 818639,
    "text": "its own Jupyter notebook. And there"
  },
  {
    "startMs": 820639,
    "text": "there's significant advantages for for"
  },
  {
    "startMs": 823120,
    "text": "doing so. And so you can see the sorts"
  },
  {
    "startMs": 825680,
    "text": "of outputs that it gets as a result. And"
  },
  {
    "startMs": 828399,
    "text": "it by itself will iterate. And in this"
  },
  {
    "startMs": 831360,
    "text": "case it didn't, but it has the option to"
  },
  {
    "startMs": 833680,
    "text": "defer to subLM to do okay. And now I"
  },
  {
    "startMs": 836079,
    "text": "have this big whatever this big subset"
  },
  {
    "startMs": 838480,
    "text": "of the data sublm go off and do this"
  },
  {
    "startMs": 840800,
    "text": "analysis give me the result and it can"
  },
  {
    "startMs": 842880,
    "text": "do that iteratively over time. Uh but"
  },
  {
    "startMs": 845519,
    "text": "the point is that the LM is directly"
  },
  {
    "startMs": 849120,
    "text": "interacting with the data frame in its"
  },
  {
    "startMs": 850959,
    "text": "ripple"
  },
  {
    "startMs": 852560,
    "text": "um and kind of iterating through the"
  },
  {
    "startMs": 854560,
    "text": "results. And so this u this platform"
  },
  {
    "startMs": 857839,
    "text": "compound is RLM and DSPI native. So it"
  },
  {
    "startMs": 861199,
    "text": "gives you this really nice breakdown of"
  },
  {
    "startMs": 863199,
    "text": "the reasoning. It separates out the code"
  },
  {
    "startMs": 865040,
    "text": "that's being generated and ultimately"
  },
  {
    "startMs": 867199,
    "text": "you can see"
  },
  {
    "startMs": 869920,
    "text": "uh the final output which is here where"
  },
  {
    "startMs": 873440,
    "text": "it's formatting. Okay, here are the key"
  },
  {
    "startMs": 875040,
    "text": "findings that I have. Here are the"
  },
  {
    "startMs": 876320,
    "text": "recommendations. And then you have this"
  },
  {
    "startMs": 878399,
    "text": "final submit which is the final answer"
  },
  {
    "startMs": 881360,
    "text": "that gives you the the typed um outputs"
  },
  {
    "startMs": 883839,
    "text": "that you had defined up front. And the"
  },
  {
    "startMs": 886160,
    "text": "key thing here is that the LLM itself is"
  },
  {
    "startMs": 888560,
    "text": "deciding when to stop. So you have this"
  },
  {
    "startMs": 891440,
    "text": "you have a variable of max iteration. So"
  },
  {
    "startMs": 893680,
    "text": "you can just you can decide whether you"
  },
  {
    "startMs": 895199,
    "text": "want it to have a maximum of 10 or 100"
  },
  {
    "startMs": 897600,
    "text": "or whatever it is. But it will by itself"
  },
  {
    "startMs": 900240,
    "text": "explore the data, understand what needs"
  },
  {
    "startMs": 902639,
    "text": "to happen and then when it it itself is"
  },
  {
    "startMs": 905040,
    "text": "comfortable, it can run submit and give"
  },
  {
    "startMs": 907360,
    "text": "you the final output. Again, being"
  },
  {
    "startMs": 910079,
    "text": "bitter or less impilled, this will get"
  },
  {
    "startMs": 911600,
    "text": "better over time. You can kind of just"
  },
  {
    "startMs": 913040,
    "text": "defer everything and it will figure out"
  },
  {
    "startMs": 915440,
    "text": "what to do. And so the hope would be you"
  },
  {
    "startMs": 918160,
    "text": "don't have to I mean we're already you"
  },
  {
    "startMs": 920240,
    "text": "know whatever this is 20 lines of code"
  },
  {
    "startMs": 922160,
    "text": "or something. Um, but you can you can"
  },
  {
    "startMs": 925040,
    "text": "see a world where you can continue to go"
  },
  {
    "startMs": 927839,
    "text": "up levels of abstraction. As long as you"
  },
  {
    "startMs": 929839,
    "text": "can define what your objective is and"
  },
  {
    "startMs": 931440,
    "text": "what you want it to do, the the model"
  },
  {
    "startMs": 933680,
    "text": "will kind of figure out the rest. Uh, so"
  },
  {
    "startMs": 936399,
    "text": "we just walk through a bunch of this,"
  },
  {
    "startMs": 937839,
    "text": "but um these are the different steps"
  },
  {
    "startMs": 939920,
    "text": "that it took in this example in the code"
  },
  {
    "startMs": 942000,
    "text": "that it wrote. Um, and then I'll just"
  },
  {
    "startMs": 945040,
    "text": "breeze through a few real world case"
  },
  {
    "startMs": 947279,
    "text": "studies and where it's actually being"
  },
  {
    "startMs": 948639,
    "text": "used. So I mentioned predict rm before."
  },
  {
    "startMs": 951120,
    "text": "So the company trampoline AI I think it"
  },
  {
    "startMs": 954240,
    "text": "is they're doing really interesting work"
  },
  {
    "startMs": 956240,
    "text": "in applying RLMs u like I mentioned"
  },
  {
    "startMs": 958880,
    "text": "before for different pieces of knowledge"
  },
  {
    "startMs": 961199,
    "text": "work. So natively interacting with PDFs"
  },
  {
    "startMs": 964320,
    "text": "and and spreadsheets and that sort of"
  },
  {
    "startMs": 966000,
    "text": "thing. So in this relatively simple"
  },
  {
    "startMs": 968639,
    "text": "example okay I have a bunch of I have a"
  },
  {
    "startMs": 971199,
    "text": "directory of invoices that I need to"
  },
  {
    "startMs": 973600,
    "text": "create one consolidated inventory out"
  },
  {
    "startMs": 975519,
    "text": "of."
  },
  {
    "startMs": 977120,
    "text": "As we all know invoices can be"
  },
  {
    "startMs": 979199,
    "text": "complicated. that can be very long, that"
  },
  {
    "startMs": 981680,
    "text": "can kind of be all over the place. To do"
  },
  {
    "startMs": 984320,
    "text": "that today without RLMs or this sort of"
  },
  {
    "startMs": 987759,
    "text": "like framework gets very complicated"
  },
  {
    "startMs": 990240,
    "text": "very quickly. I have a lot of battle"
  },
  {
    "startMs": 992079,
    "text": "scars to to prove it. Um, but with"
  },
  {
    "startMs": 994560,
    "text": "something like an RLM, you you don't"
  },
  {
    "startMs": 995920,
    "text": "need to worry as much about, okay, if I"
  },
  {
    "startMs": 997600,
    "text": "have a 200page invoice or contract or"
  },
  {
    "startMs": 1000720,
    "text": "whatever it is, you can let the RLM just"
  },
  {
    "startMs": 1003199,
    "text": "churn through all of that and give you"
  },
  {
    "startMs": 1004639,
    "text": "the result instead of having to worry"
  },
  {
    "startMs": 1006160,
    "text": "about chunking and embedding maybe and"
  },
  {
    "startMs": 1009680,
    "text": "doing all these different strategies to"
  },
  {
    "startMs": 1011040,
    "text": "try and get around the context window"
  },
  {
    "startMs": 1013440,
    "text": "management that we've all had to to do"
  },
  {
    "startMs": 1015199,
    "text": "previously."
  },
  {
    "startMs": 1017360,
    "text": "Um, so it allows you, the point there is"
  },
  {
    "startMs": 1019440,
    "text": "that you can focus on the abstractions"
  },
  {
    "startMs": 1022000,
    "text": "and what you actually want to do instead"
  },
  {
    "startMs": 1023759,
    "text": "of the context engineering itself, which"
  },
  {
    "startMs": 1025839,
    "text": "I think is a really helpful um, helpful"
  },
  {
    "startMs": 1028799,
    "text": "output of all of this. And an"
  },
  {
    "startMs": 1030559,
    "text": "interesting tidbit for all the DSPI fans"
  },
  {
    "startMs": 1032319,
    "text": "in the room, predict RLM uses DSPI to"
  },
  {
    "startMs": 1036720,
    "text": "determine the schemas between the main"
  },
  {
    "startMs": 1039760,
    "text": "LM and the subLM calls, which I"
  },
  {
    "startMs": 1042640,
    "text": "personally think is is a nice feature"
  },
  {
    "startMs": 1044400,
    "text": "because you have a lot more readability"
  },
  {
    "startMs": 1046559,
    "text": "and maintainability. So you understand"
  },
  {
    "startMs": 1049039,
    "text": "exactly what the model is trying to"
  },
  {
    "startMs": 1051360,
    "text": "achieve and the model can be much more"
  },
  {
    "startMs": 1052880,
    "text": "precise and prescriptive about the types"
  },
  {
    "startMs": 1055120,
    "text": "of data that it's looking for from the"
  },
  {
    "startMs": 1057120,
    "text": "subLM. And I would want to do some"
  },
  {
    "startMs": 1060720,
    "text": "experiments to test this out, but I"
  },
  {
    "startMs": 1062080,
    "text": "would think that this would improve"
  },
  {
    "startMs": 1063200,
    "text": "performance for cheaper models like a a"
  },
  {
    "startMs": 1066000,
    "text": "Quinn or some of the other ones because"
  },
  {
    "startMs": 1068000,
    "text": "you're specifying the inputs and outputs"
  },
  {
    "startMs": 1069520,
    "text": "and you're enforcing those types coming"
  },
  {
    "startMs": 1071200,
    "text": "back. And so you get all the benefits of"
  },
  {
    "startMs": 1074160,
    "text": "the RLM being able to churn through all"
  },
  {
    "startMs": 1076160,
    "text": "this information, but you have a lot"
  },
  {
    "startMs": 1078240,
    "text": "more of the structure in between where"
  },
  {
    "startMs": 1081039,
    "text": "when it's handing off to a sublm, it it"
  },
  {
    "startMs": 1083520,
    "text": "enforces some of the uh some of those"
  },
  {
    "startMs": 1085200,
    "text": "schemas."
  },
  {
    "startMs": 1087120,
    "text": "This is an example from um an AWS"
  },
  {
    "startMs": 1090960,
    "text": "engineer from a couple days ago. We were"
  },
  {
    "startMs": 1093280,
    "text": "just kind of playing around with it, but"
  },
  {
    "startMs": 1094960,
    "text": "I just thought it was a nice example of"
  },
  {
    "startMs": 1097440,
    "text": "you can kind of just throw arbitrary"
  },
  {
    "startMs": 1100160,
    "text": "data at RLM. In this case, it was a"
  },
  {
    "startMs": 1103120,
    "text": "bunch of log data um to surface some"
  },
  {
    "startMs": 1106799,
    "text": "interesting uh results and he he found"
  },
  {
    "startMs": 1108799,
    "text": "it useful. Um there's a a project called"
  },
  {
    "startMs": 1112240,
    "text": "Halo which uses an RLM to look at traces"
  },
  {
    "startMs": 1116160,
    "text": "of um of different t uh agent tasks."
  },
  {
    "startMs": 1121440,
    "text": "And basically the promise of Halo is"
  },
  {
    "startMs": 1123360,
    "text": "that instead of optimizing a particular"
  },
  {
    "startMs": 1125919,
    "text": "like workflow or DSPI uh or or other"
  },
  {
    "startMs": 1129520,
    "text": "framework"
  },
  {
    "startMs": 1131200,
    "text": "uh like structure itself, it's it's"
  },
  {
    "startMs": 1133679,
    "text": "actually iterating on the harness. So"
  },
  {
    "startMs": 1136080,
    "text": "it's like a meta abstraction almost or"
  },
  {
    "startMs": 1138559,
    "text": "meta optimization of the harness itself"
  },
  {
    "startMs": 1141200,
    "text": "and it uses an RLM because as we all"
  },
  {
    "startMs": 1143520,
    "text": "know tracing can get very long and and"
  },
  {
    "startMs": 1145679,
    "text": "complicated."
  },
  {
    "startMs": 1147280,
    "text": "So the RLM can not only take in all that"
  },
  {
    "startMs": 1149840,
    "text": "context but also leverage the um the"
  },
  {
    "startMs": 1153039,
    "text": "structure of those traces to to"
  },
  {
    "startMs": 1155520,
    "text": "recommend a better um a better harness."
  },
  {
    "startMs": 1158880,
    "text": "And then this last one uh this is all"
  },
  {
    "startMs": 1161120,
    "text": "the code you need. I ran this little"
  },
  {
    "startMs": 1163280,
    "text": "experiment. There's a an intentionally"
  },
  {
    "startMs": 1165360,
    "text": "vulnerable application called uh it's"
  },
  {
    "startMs": 1167760,
    "text": "from OASP, but basically there's a it's"
  },
  {
    "startMs": 1170240,
    "text": "a web app with a bunch of"
  },
  {
    "startMs": 1171200,
    "text": "vulnerabilities in it. This is all the"
  },
  {
    "startMs": 1173440,
    "text": "code you need on the right hand side to"
  },
  {
    "startMs": 1175440,
    "text": "run basically an agent to run through"
  },
  {
    "startMs": 1178320,
    "text": "whatever it is 500,000 lines of code to"
  },
  {
    "startMs": 1181120,
    "text": "generate some type of security report."
  },
  {
    "startMs": 1182799,
    "text": "That's just an arbitrary example, but"
  },
  {
    "startMs": 1184640,
    "text": "the point is you don't need a lot of"
  },
  {
    "startMs": 1187520,
    "text": "context engineering. You don't need a"
  },
  {
    "startMs": 1188880,
    "text": "lot of structure around it to achieve"
  },
  {
    "startMs": 1191039,
    "text": "what you want to do. And so you can feed"
  },
  {
    "startMs": 1193120,
    "text": "in an arbitrary uh size codebase into"
  },
  {
    "startMs": 1196400,
    "text": "this and get some type of insights out."
  },
  {
    "startMs": 1198880,
    "text": "So you can imagine that being applied to"
  },
  {
    "startMs": 1200480,
    "text": "other areas as well. Um so I know I I"
  },
  {
    "startMs": 1204000,
    "text": "rushed through everything a little bit"
  },
  {
    "startMs": 1205200,
    "text": "but I'm happy to answer questions"
  },
  {
    "startMs": 1206559,
    "text": "afterwards. Uh the I'll leave you leave"
  },
  {
    "startMs": 1209360,
    "text": "you with this. The biggest promise I see"
  },
  {
    "startMs": 1211440,
    "text": "here is just imagine a world where the"
  },
  {
    "startMs": 1213840,
    "text": "models are actually post-trained and"
  },
  {
    "startMs": 1216240,
    "text": "actually like RLM aware. I think things"
  },
  {
    "startMs": 1218559,
    "text": "will get pretty crazy pretty quick when"
  },
  {
    "startMs": 1221919,
    "text": "they actually know how to use and kind"
  },
  {
    "startMs": 1224799,
    "text": "of take advantage of the RLM methodology"
  },
  {
    "startMs": 1227200,
    "text": "natively. So, thank you so much for your"
  },
  {
    "startMs": 1229440,
    "text": "time."
  }
]