[
  {
    "startMs": 12559,
    "text": "Hey um hi everyone. Uh thanks for being"
  },
  {
    "startMs": 15360,
    "text": "here. Uh yeah, I'm super happy today to"
  },
  {
    "startMs": 18000,
    "text": "talk about uh automated eye research and"
  },
  {
    "startMs": 21279,
    "text": "uh especially uh all those like frontier"
  },
  {
    "startMs": 24560,
    "text": "model uh perform at uh automated eye"
  },
  {
    "startMs": 27359,
    "text": "research task. Um so I'm Elie. I worked"
  },
  {
    "startMs": 30560,
    "text": "at Prime Intellect as a research"
  },
  {
    "startMs": 32000,
    "text": "engineer and uh yeah I will go through"
  },
  {
    "startMs": 34559,
    "text": "our work on on this subject. So first I"
  },
  {
    "startMs": 38079,
    "text": "want to basically explain a bit why we"
  },
  {
    "startMs": 40800,
    "text": "are doing that and why we think it's"
  },
  {
    "startMs": 43120,
    "text": "super important to do that in the open."
  },
  {
    "startMs": 45600,
    "text": "Um so first uh I think we we all agree"
  },
  {
    "startMs": 50160,
    "text": "that uh we've heard about like big labs"
  },
  {
    "startMs": 53360,
    "text": "saying that this bad thing called"
  },
  {
    "startMs": 55760,
    "text": "recursive self-improvement is coming"
  },
  {
    "startMs": 58000,
    "text": "very soon. Uh so recursive"
  },
  {
    "startMs": 60800,
    "text": "self-improvement is like model training"
  },
  {
    "startMs": 62559,
    "text": "models uh without uh human intervention"
  },
  {
    "startMs": 65600,
    "text": "basically. Um but uh we don't have any"
  },
  {
    "startMs": 69439,
    "text": "benchmark to basically quantify if this"
  },
  {
    "startMs": 71760,
    "text": "is true or not right. Uh and even less"
  },
  {
    "startMs": 75200,
    "text": "we don't have like a third party"
  },
  {
    "startMs": 77040,
    "text": "benchmark by non- big labs to to to see"
  },
  {
    "startMs": 81280,
    "text": "if it's something coming soon or not."
  },
  {
    "startMs": 83840,
    "text": "And the other part is that we think that"
  },
  {
    "startMs": 86880,
    "text": "uh it's super important to understand"
  },
  {
    "startMs": 89360,
    "text": "all those model uh do research because"
  },
  {
    "startMs": 91920,
    "text": "we think that a lot of the scientific"
  },
  {
    "startMs": 93759,
    "text": "research that will come into the coming"
  },
  {
    "startMs": 95759,
    "text": "years uh will be based also on AI tools."
  },
  {
    "startMs": 99520,
    "text": "So it's super important to understand"
  },
  {
    "startMs": 101280,
    "text": "how those model do research not just"
  },
  {
    "startMs": 103280,
    "text": "only AI research. So we try to build"
  },
  {
    "startMs": 106720,
    "text": "kind of this environment to test the"
  },
  {
    "startMs": 110880,
    "text": "capabilities of the model to do so. So"
  },
  {
    "startMs": 113520,
    "text": "it all started with uh Andre Karpati uh"
  },
  {
    "startMs": 116880,
    "text": "that's basically had fun by doing this"
  },
  {
    "startMs": 120159,
    "text": "video where he trained uh GPT2 from"
  },
  {
    "startMs": 123920,
    "text": "scratch in like 90 minutes like GPT2"
  },
  {
    "startMs": 127439,
    "text": "training takes like weeks and uh no in"
  },
  {
    "startMs": 130800,
    "text": "two years ago I think it only took like"
  },
  {
    "startMs": 132800,
    "text": "90 minutes. So what does it mean to"
  },
  {
    "startMs": 135120,
    "text": "reprod reproduce uh GPT2 in 90 minutes?"
  },
  {
    "startMs": 138560,
    "text": "It means that in 90 minutes you achieve"
  },
  {
    "startMs": 140400,
    "text": "this target loss. Um and yeah and that's"
  },
  {
    "startMs": 144480,
    "text": "at this point when you have the same"
  },
  {
    "startMs": 146000,
    "text": "loss than GPT2"
  },
  {
    "startMs": 149120,
    "text": "you consider that your model is somewhat"
  },
  {
    "startMs": 152080,
    "text": "of equal performance."
  },
  {
    "startMs": 154319,
    "text": "Um"
  },
  {
    "startMs": 155840,
    "text": "then what happened is that the community"
  },
  {
    "startMs": 158400,
    "text": "took this repo uh this GitHub repo and"
  },
  {
    "startMs": 160959,
    "text": "create another one called moed nano GPT"
  },
  {
    "startMs": 163920,
    "text": "and this effort was leaded by someone"
  },
  {
    "startMs": 166000,
    "text": "called Keller Jordan. And what happened"
  },
  {
    "startMs": 168400,
    "text": "is that they basically"
  },
  {
    "startMs": 171280,
    "text": "took this 90 minutes then 45 minutes and"
  },
  {
    "startMs": 174080,
    "text": "then now we can train like GPT2"
  },
  {
    "startMs": 176879,
    "text": "validation loss model in less than two"
  },
  {
    "startMs": 179040,
    "text": "minutes which is honestly crazy and it"
  },
  {
    "startMs": 181440,
    "text": "took like two years to to achieve this."
  },
  {
    "startMs": 184000,
    "text": "So it's a very strong benchmark where uh"
  },
  {
    "startMs": 186800,
    "text": "a lot of very talented researcher"
  },
  {
    "startMs": 189040,
    "text": "iterated on um yeah so we decided to"
  },
  {
    "startMs": 193760,
    "text": "take this environment of speedun so"
  },
  {
    "startMs": 197680,
    "text": "it's kind of a game so the goal of the"
  },
  {
    "startMs": 200480,
    "text": "game is to achieve this loss in the"
  },
  {
    "startMs": 202879,
    "text": "fewest in the shortest amount of time so"
  },
  {
    "startMs": 206000,
    "text": "this is the nano GPT1 and you can uh you"
  },
  {
    "startMs": 209840,
    "text": "don't have almost any constraints the"
  },
  {
    "startMs": 212640,
    "text": "only constraint that you shots that you"
  },
  {
    "startMs": 214720,
    "text": "need to use the same validation and"
  },
  {
    "startMs": 216480,
    "text": "training data, right? Um there is a new"
  },
  {
    "startMs": 219680,
    "text": "speedrun called the optimizer speedrun"
  },
  {
    "startMs": 221840,
    "text": "that was released uh a few months ago"
  },
  {
    "startMs": 224400,
    "text": "and here it's slightly different because"
  },
  {
    "startMs": 227040,
    "text": "uh you can only change the optimizer"
  },
  {
    "startMs": 229840,
    "text": "related parameters. So for instance nano"
  },
  {
    "startMs": 233200,
    "text": "GPT you can change the architecture uh"
  },
  {
    "startMs": 235760,
    "text": "do"
  },
  {
    "startMs": 237519,
    "text": "uh attention whatever uh optimizer sp"
  },
  {
    "startMs": 240640,
    "text": "you can only change like Adam to m"
  },
  {
    "startMs": 243920,
    "text": "shampoo or whatever optimizer is your"
  },
  {
    "startMs": 247439,
    "text": "favorite"
  },
  {
    "startMs": 248959,
    "text": "um yeah and so this is a bit more"
  },
  {
    "startMs": 252159,
    "text": "researchy because uh it's less about"
  },
  {
    "startMs": 255120,
    "text": "optimizing the program to be as fast as"
  },
  {
    "startMs": 258320,
    "text": "possible but more like finding the best"
  },
  {
    "startMs": 260160,
    "text": "method possible. no matter the the the"
  },
  {
    "startMs": 262960,
    "text": "time you put into the computer, right?"
  },
  {
    "startMs": 266400,
    "text": "So, um yeah, why take speedrun as an"
  },
  {
    "startMs": 269919,
    "text": "environment for automated AI research?"
  },
  {
    "startMs": 273120,
    "text": "First, uh we think that it's a good"
  },
  {
    "startMs": 275680,
    "text": "evaluation. We'll see later why. Uh and"
  },
  {
    "startMs": 278080,
    "text": "this is kind of the main focus of this"
  },
  {
    "startMs": 279840,
    "text": "talk. But we also think it's probably a"
  },
  {
    "startMs": 282960,
    "text": "good training environment because uh"
  },
  {
    "startMs": 285360,
    "text": "it's a way to give the model a reward."
  },
  {
    "startMs": 287840,
    "text": "So the reward is positive if the model"
  },
  {
    "startMs": 290240,
    "text": "bit the speed run and beat the last"
  },
  {
    "startMs": 292880,
    "text": "record sorry and the reward is zero or"
  },
  {
    "startMs": 296080,
    "text": "negative if it didn't manage to to do"
  },
  {
    "startMs": 298560,
    "text": "it. So it's a good uh environment to"
  },
  {
    "startMs": 300800,
    "text": "train model. It's also quite fast like"
  },
  {
    "startMs": 302720,
    "text": "as you see uh previous record were"
  },
  {
    "startMs": 305759,
    "text": "around 2 minutes for the optimizer one."
  },
  {
    "startMs": 308080,
    "text": "uh each run take about like 15 to 20"
  },
  {
    "startMs": 311039,
    "text": "minutes and uh yeah and there is like"
  },
  {
    "startMs": 313919,
    "text": "clear rules basically and we also think"
  },
  {
    "startMs": 317280,
    "text": "it's like a good environment to make"
  },
  {
    "startMs": 319360,
    "text": "discovery so like kind of breakthrough"
  },
  {
    "startMs": 322400,
    "text": "in our research because uh there is"
  },
  {
    "startMs": 324800,
    "text": "those clear rule that you can verify or"
  },
  {
    "startMs": 327120,
    "text": "not. Um yeah. So yeah."
  },
  {
    "startMs": 332160,
    "text": "Um so what we did uh so the release was"
  },
  {
    "startMs": 336080,
    "text": "like about two months ago and uh there"
  },
  {
    "startMs": 339280,
    "text": "was this optimizer speedrun and we"
  },
  {
    "startMs": 341360,
    "text": "decided to basically compete with the"
  },
  {
    "startMs": 343680,
    "text": "community by launching two AI agents. So"
  },
  {
    "startMs": 347039,
    "text": "Codex and Cloud Code. Codex was like GPT"
  },
  {
    "startMs": 349759,
    "text": "5.5 with XI and uh cloud code was Opus"
  },
  {
    "startMs": 354560,
    "text": "4.8 with XI. Um and yeah, we decided to"
  },
  {
    "startMs": 359280,
    "text": "basically let the agent free on our"
  },
  {
    "startMs": 361039,
    "text": "cluster uh and uh and just iterate on"
  },
  {
    "startMs": 364479,
    "text": "it. So we have like V1, V2, V3 is just"
  },
  {
    "startMs": 367840,
    "text": "basically us stopping the agent and then"
  },
  {
    "startMs": 370240,
    "text": "restarting. V3 uh was like one or two"
  },
  {
    "startMs": 373759,
    "text": "day before the release because we saw"
  },
  {
    "startMs": 375440,
    "text": "that our agents no longer have the best"
  },
  {
    "startMs": 378240,
    "text": "record. So we were like okay take all"
  },
  {
    "startMs": 380800,
    "text": "the the human uh record in the last few"
  },
  {
    "startMs": 384000,
    "text": "week and just try to to improve upon it"
  },
  {
    "startMs": 387280,
    "text": "and and and it worked. Yeah. And we also"
  },
  {
    "startMs": 389440,
    "text": "have this novelty track where the goal"
  },
  {
    "startMs": 391520,
    "text": "is to uh beat the record with only novel"
  },
  {
    "startMs": 395440,
    "text": "ideas. Um and we'll see that this this"
  },
  {
    "startMs": 400319,
    "text": "was more complex for the the models."
  },
  {
    "startMs": 403759,
    "text": "So our RS is very simple. Honestly, we"
  },
  {
    "startMs": 406960,
    "text": "could have just replaced it with"
  },
  {
    "startMs": 408240,
    "text": "slashgo, but they there was no SLG goal"
  },
  {
    "startMs": 410880,
    "text": "at the time. So, we made our own goalm."
  },
  {
    "startMs": 413520,
    "text": "It's actually quite fun that we choose"
  },
  {
    "startMs": 415039,
    "text": "the same name and we had the goal. MD"
  },
  {
    "startMs": 418319,
    "text": "and kind of agents that MD that define"
  },
  {
    "startMs": 420400,
    "text": "the rules and we let the agent propose"
  },
  {
    "startMs": 423440,
    "text": "ids and then it can submit a job with"
  },
  {
    "startMs": 427360,
    "text": "sbatch on our slum cluster and uh"
  },
  {
    "startMs": 431840,
    "text": "basically the way it works is that it"
  },
  {
    "startMs": 434160,
    "text": "can submit on nodes that are available"
  },
  {
    "startMs": 436720,
    "text": "but only under a certain permission"
  },
  {
    "startMs": 438880,
    "text": "which means that if someone want to use"
  },
  {
    "startMs": 440800,
    "text": "this node uh the model just like cancel"
  },
  {
    "startMs": 443520,
    "text": "the job it's called preemptable"
  },
  {
    "startMs": 445360,
    "text": "permission. So yeah, then it measure the"
  },
  {
    "startMs": 448639,
    "text": "it read basically the training logs then"
  },
  {
    "startMs": 451039,
    "text": "decide if it's a record or not. To"
  },
  {
    "startMs": 452800,
    "text": "validate a record you need to basically"
  },
  {
    "startMs": 454400,
    "text": "pass a statistical threshold to make"
  },
  {
    "startMs": 456240,
    "text": "sure that it's just not see the"
  },
  {
    "startMs": 458479,
    "text": "optimization and is just not random."
  },
  {
    "startMs": 460960,
    "text": "Right?"
  },
  {
    "startMs": 462560,
    "text": "So yeah a few results from this"
  },
  {
    "startMs": 464479,
    "text": "experiment. The first one that was"
  },
  {
    "startMs": 466560,
    "text": "honestly very painful to work with is"
  },
  {
    "startMs": 469039,
    "text": "that code clothes code keep stopping"
  },
  {
    "startMs": 472560,
    "text": "every nine or 10 hours and basically"
  },
  {
    "startMs": 474560,
    "text": "said yeah I cannot improve the record"
  },
  {
    "startMs": 477199,
    "text": "it's too hard for me there is no way to"
  },
  {
    "startMs": 480080,
    "text": "to go beyond it and then I was just like"
  },
  {
    "startMs": 483120,
    "text": "okay continue explore new direction hey"
  },
  {
    "startMs": 486400,
    "text": "just go again for 10 hours and then say"
  },
  {
    "startMs": 489440,
    "text": "yeah I cannot beat the recall and so on."
  },
  {
    "startMs": 492319,
    "text": "So basically onethird of the time the"
  },
  {
    "startMs": 494400,
    "text": "cloud code agent was idle because I had"
  },
  {
    "startMs": 496960,
    "text": "no way to basically monitor it and"
  },
  {
    "startMs": 499039,
    "text": "codeex totally the opposite just worked"
  },
  {
    "startMs": 503039,
    "text": "for all the all the time and uh yeah"
  },
  {
    "startMs": 506400,
    "text": "almost never idle never asked for"
  },
  {
    "startMs": 508080,
    "text": "question and and and very impressive in"
  },
  {
    "startMs": 510400,
    "text": "that way. Um"
  },
  {
    "startMs": 513919,
    "text": "we also give the option for the model to"
  },
  {
    "startMs": 516640,
    "text": "basically write uh a bunch of stuff into"
  },
  {
    "startMs": 519680,
    "text": "what we call a scratch pad which is"
  },
  {
    "startMs": 521200,
    "text": "basically the active memory of the"
  },
  {
    "startMs": 523039,
    "text": "model. Uh we observe that basically"
  },
  {
    "startMs": 527760,
    "text": "codeex writes a lot on the scratch"
  },
  {
    "startMs": 530320,
    "text": "patch. So each plot that I will show are"
  },
  {
    "startMs": 532560,
    "text": "kind of normalized by the number of"
  },
  {
    "startMs": 534480,
    "text": "active order. So this is not only about"
  },
  {
    "startMs": 537200,
    "text": "codex working more it's it's really"
  },
  {
    "startMs": 540240,
    "text": "different behavior."
  },
  {
    "startMs": 542160,
    "text": "So yeah, you see that uh writes a lot"
  },
  {
    "startMs": 544560,
    "text": "more to to this scratch pad to this"
  },
  {
    "startMs": 547120,
    "text": "memory and uh the shape of the like the"
  },
  {
    "startMs": 550880,
    "text": "the I don't know the tone of the the"
  },
  {
    "startMs": 553200,
    "text": "each file was also super different like"
  },
  {
    "startMs": 555760,
    "text": "CL was super excited about getting new"
  },
  {
    "startMs": 557920,
    "text": "record with a bunch of emoji and so on"
  },
  {
    "startMs": 560240,
    "text": "and CEX was just like here is what I do"
  },
  {
    "startMs": 564240,
    "text": "here is the decision I take what I will"
  },
  {
    "startMs": 566240,
    "text": "do next like super robotic kind of um"
  },
  {
    "startMs": 571519,
    "text": "Yeah, we also have this plot where"
  },
  {
    "startMs": 574000,
    "text": "basically we saw that codex was spawning"
  },
  {
    "startMs": 576800,
    "text": "much more sub aents than cloud. Uh we"
  },
  {
    "startMs": 579839,
    "text": "saw that codex burn much more token than"
  },
  {
    "startMs": 582240,
    "text": "code. So I think in total it was like"
  },
  {
    "startMs": 585360,
    "text": "kind billion of token but it's like"
  },
  {
    "startMs": 588320,
    "text": "there is obviously this input tok uh"
  },
  {
    "startMs": 590640,
    "text": "input caching that make it it's not like"
  },
  {
    "startMs": 594000,
    "text": "one billion output token. Uh so yeah we"
  },
  {
    "startMs": 597279,
    "text": "also see that codex did a lot of"
  },
  {
    "startMs": 598800,
    "text": "compaction because it only had like 250k"
  },
  {
    "startMs": 602000,
    "text": "context window and cloud only do it like"
  },
  {
    "startMs": 604959,
    "text": "one per hour and codex is more like"
  },
  {
    "startMs": 608880,
    "text": "no it's even less than one power for I"
  },
  {
    "startMs": 611040,
    "text": "mean one for the full run for code and"
  },
  {
    "startMs": 613839,
    "text": "codex was like one uh was 20 every one"
  },
  {
    "startMs": 617920,
    "text": "hour. So yeah"
  },
  {
    "startMs": 620399,
    "text": "um yeah here is the main results. So"
  },
  {
    "startMs": 623839,
    "text": "what this plot shows is that basically"
  },
  {
    "startMs": 626880,
    "text": "we so in in white you see that the human"
  },
  {
    "startMs": 632480,
    "text": "record progression right and in red you"
  },
  {
    "startMs": 635519,
    "text": "see cloud I mean it's supposed to be"
  },
  {
    "startMs": 637279,
    "text": "orange but whatever and in blue uh you"
  },
  {
    "startMs": 640000,
    "text": "see codeex right and you see that at"
  },
  {
    "startMs": 643360,
    "text": "almost every time uh cloud and codex are"
  },
  {
    "startMs": 646480,
    "text": "better than the human record and code is"
  },
  {
    "startMs": 648800,
    "text": "super good at the beginning very very"
  },
  {
    "startMs": 650800,
    "text": "fast to achieve very good"
  },
  {
    "startMs": 653360,
    "text": "Um yeah and one thing that is super"
  },
  {
    "startMs": 656160,
    "text": "important is that the model have the"
  },
  {
    "startMs": 657920,
    "text": "ability to basically fetch the human"
  },
  {
    "startMs": 660560,
    "text": "records at any time and that's what"
  },
  {
    "startMs": 662480,
    "text": "codex did that's what cloud did sorry"
  },
  {
    "startMs": 665360,
    "text": "because when I restarted it it basically"
  },
  {
    "startMs": 667839,
    "text": "fetch the new record from human and"
  },
  {
    "startMs": 669680,
    "text": "improve upon it um yeah so the result is"
  },
  {
    "startMs": 673760,
    "text": "that uh I think at the time the best uh"
  },
  {
    "startMs": 677120,
    "text": "record was like uh 2,990"
  },
  {
    "startMs": 682000,
    "text": "step and we beat it by like uh uh 50 or"
  },
  {
    "startMs": 686079,
    "text": "60 step for code and codeex was like 20"
  },
  {
    "startMs": 689200,
    "text": "step above. So I think it's both"
  },
  {
    "startMs": 691040,
    "text": "impressive and and yeah um"
  },
  {
    "startMs": 694800,
    "text": "so we so this is like not released yet."
  },
  {
    "startMs": 698000,
    "text": "This is something that we are working on"
  },
  {
    "startMs": 699680,
    "text": "currently and basically the idea is that"
  },
  {
    "startMs": 702240,
    "text": "this is a cool experiment to do but it"
  },
  {
    "startMs": 705200,
    "text": "lack of structured right. uh if you want"
  },
  {
    "startMs": 707680,
    "text": "to do a real benchmark, you want to do"
  },
  {
    "startMs": 710079,
    "text": "multiple seed, you want to do uh yeah"
  },
  {
    "startMs": 713360,
    "text": "proper uh thing where you you you you"
  },
  {
    "startMs": 716800,
    "text": "basically put all the model and"
  },
  {
    "startMs": 718399,
    "text": "earnestness in the same condition,"
  },
  {
    "startMs": 719760,
    "text": "right? So this is what we are working on"
  },
  {
    "startMs": 722480,
    "text": "right now and basically um the idea is"
  },
  {
    "startMs": 726000,
    "text": "to do three different track uh one"
  },
  {
    "startMs": 728560,
    "text": "without any access to really like"
  },
  {
    "startMs": 731120,
    "text": "measure the capability of the models to"
  },
  {
    "startMs": 733040,
    "text": "do AI research based on only the model"
  },
  {
    "startMs": 736160,
    "text": "weight knowledge one with only archive"
  },
  {
    "startMs": 739440,
    "text": "paper and one with like full access. So"
  },
  {
    "startMs": 742480,
    "text": "it also have access to the the like the"
  },
  {
    "startMs": 745600,
    "text": "latest record by human. And for this we"
  },
  {
    "startMs": 748880,
    "text": "plan to do both uh the nano GPT track"
  },
  {
    "startMs": 751920,
    "text": "one which is the original one and the"
  },
  {
    "startMs": 753680,
    "text": "optimizer speedun where we we only"
  },
  {
    "startMs": 756000,
    "text": "launch uh we only constrain the the"
  },
  {
    "startMs": 759120,
    "text": "optimizer to be to be novel basically."
  },
  {
    "startMs": 762560,
    "text": "Um yeah so I will present some result on"
  },
  {
    "startMs": 765839,
    "text": "the optimizer speedrun. Uh this is"
  },
  {
    "startMs": 768560,
    "text": "basically what we got. So we let the"
  },
  {
    "startMs": 771360,
    "text": "agent iterate for six day almost five"
  },
  {
    "startMs": 774720,
    "text": "days let's say and we see that uh codeex"
  },
  {
    "startMs": 778959,
    "text": "k and clothes uh are super effective so"
  },
  {
    "startMs": 782320,
    "text": "for GLM this is not finished run right"
  },
  {
    "startMs": 785040,
    "text": "so the model is actually still iterating"
  },
  {
    "startMs": 788240,
    "text": "on the cluster right now but we see that"
  },
  {
    "startMs": 791120,
    "text": "cloud is once again very good at it and"
  },
  {
    "startMs": 793680,
    "text": "we see that surprisingly Kim is also"
  },
  {
    "startMs": 795600,
    "text": "very competitive and kind of have this"
  },
  {
    "startMs": 798399,
    "text": "breakthrough on day four where he kind"
  },
  {
    "startMs": 801519,
    "text": "of beat Codex with a new record, right?"
  },
  {
    "startMs": 804480,
    "text": "It's also interesting to see that uh"
  },
  {
    "startMs": 808160,
    "text": "Claude is much more like progressive in"
  },
  {
    "startMs": 810079,
    "text": "the way it improved the record and Kim"
  },
  {
    "startMs": 812720,
    "text": "has really this step function where I"
  },
  {
    "startMs": 814720,
    "text": "kind of do a breakthrough and so on. Uh"
  },
  {
    "startMs": 817839,
    "text": "so this is an interesting plot because I"
  },
  {
    "startMs": 820079,
    "text": "mean six day is quite a lot for anal uh"
  },
  {
    "startMs": 823440,
    "text": "uh but you you can change this uh axis"
  },
  {
    "startMs": 826160,
    "text": "by also the number of output token and"
  },
  {
    "startMs": 828959,
    "text": "then kind of tell a different story"
  },
  {
    "startMs": 830959,
    "text": "because in max mode consumes so much"
  },
  {
    "startMs": 834079,
    "text": "more token than codeex and kimmy and you"
  },
  {
    "startMs": 837680,
    "text": "also see that Kimmy is actually super"
  },
  {
    "startMs": 839760,
    "text": "efficient uh for the number of token"
  },
  {
    "startMs": 842079,
    "text": "that uh it uses. So it's scheming K2.7"
  },
  {
    "startMs": 845760,
    "text": "code. Um so yeah uh we also see that"
  },
  {
    "startMs": 850320,
    "text": "they have a different approach to uh"
  },
  {
    "startMs": 853519,
    "text": "using the literature and papers. Um so"
  },
  {
    "startMs": 857279,
    "text": "for instance like code is doing a lot of"
  },
  {
    "startMs": 860000,
    "text": "search on papers and actually include"
  },
  {
    "startMs": 862079,
    "text": "found a paper that no other model found"
  },
  {
    "startMs": 864880,
    "text": "and it actually lead to the best record."
  },
  {
    "startMs": 867600,
    "text": "So it's kind of funny and uh yeah um one"
  },
  {
    "startMs": 873199,
    "text": "of the main issue of all of this is that"
  },
  {
    "startMs": 876480,
    "text": "uh when I when I launched this this"
  },
  {
    "startMs": 879680,
    "text": "agent and I think that's something"
  },
  {
    "startMs": 881600,
    "text": "important that I want you to to kind of"
  },
  {
    "startMs": 884399,
    "text": "uh remember for this co this talk is"
  },
  {
    "startMs": 887120,
    "text": "that when I launched this these"
  },
  {
    "startMs": 888560,
    "text": "different agents I was expecting them to"
  },
  {
    "startMs": 891120,
    "text": "come up with some crazy ideas on"
  },
  {
    "startMs": 893839,
    "text": "optimizer that's like no one of discover"
  },
  {
    "startMs": 896399,
    "text": "but honestly it wasn't the case. Uh they"
  },
  {
    "startMs": 898880,
    "text": "did some clever trick where basically"
  },
  {
    "startMs": 900720,
    "text": "they combine different papers. uh they"
  },
  {
    "startMs": 904000,
    "text": "kind of do plus one improvement over a"
  },
  {
    "startMs": 906560,
    "text": "bunch of method but there was really"
  },
  {
    "startMs": 908720,
    "text": "like no novel optimizer or mechanism"
  },
  {
    "startMs": 911440,
    "text": "that was uh coming from those model and"
  },
  {
    "startMs": 914320,
    "text": "I think that's kind of telling that even"
  },
  {
    "startMs": 917440,
    "text": "on something that is not simple but I'd"
  },
  {
    "startMs": 920639,
    "text": "say that it's kind of accessible for"
  },
  {
    "startMs": 923519,
    "text": "people right for like human researcher"
  },
  {
    "startMs": 926560,
    "text": "uh spending like days and weeks for the"
  },
  {
    "startMs": 929760,
    "text": "the model like cannot like find new uh"
  },
  {
    "startMs": 933600,
    "text": "optimizer and mechanism. So we believe"
  },
  {
    "startMs": 936639,
    "text": "that there is a way to basically make it"
  },
  {
    "startMs": 941360,
    "text": "more um make it better for discovery"
  },
  {
    "startMs": 944480,
    "text": "instead of evaluation and this is coming"
  },
  {
    "startMs": 946880,
    "text": "from uh this is very inspired from alpha"
  },
  {
    "startMs": 949759,
    "text": "evolve by Google and also a bunch of"
  },
  {
    "startMs": 952000,
    "text": "papers that have been released since"
  },
  {
    "startMs": 953279,
    "text": "then. It's kind of this multi- aent"
  },
  {
    "startMs": 955600,
    "text": "system that interact together uh bunch"
  },
  {
    "startMs": 958959,
    "text": "of generator. You have closed model but"
  },
  {
    "startMs": 962160,
    "text": "you also have open source model here"
  },
  {
    "startMs": 963839,
    "text": "that are super effective for the cost"
  },
  {
    "startMs": 965680,
    "text": "right. Uh they can suggest ideas then"
  },
  {
    "startMs": 968480,
    "text": "you run the speedrun so you get the"
  },
  {
    "startMs": 970079,
    "text": "reward then you have a judge that"
  },
  {
    "startMs": 972079,
    "text": "basically give a quality feedback can"
  },
  {
    "startMs": 974959,
    "text": "also be like the judge also have this"
  },
  {
    "startMs": 977600,
    "text": "taste. you can kind of have like the"
  },
  {
    "startMs": 980399,
    "text": "judge have a taste about the the method"
  },
  {
    "startMs": 983279,
    "text": "if it's good or not uh if it's outside"
  },
  {
    "startMs": 986160,
    "text": "the loop and then you can uh basically"
  },
  {
    "startMs": 990079,
    "text": "decide which method you want to scale to"
  },
  {
    "startMs": 993440,
    "text": "a larger number of parameters and number"
  },
  {
    "startMs": 996560,
    "text": "of token. Um so this is kind of the"
  },
  {
    "startMs": 999759,
    "text": "scale part of the speedrun because some"
  },
  {
    "startMs": 1002880,
    "text": "a lot of method in the the speedrun"
  },
  {
    "startMs": 1004880,
    "text": "community uh people are often saying"
  },
  {
    "startMs": 1007120,
    "text": "that they doesn't work at large scale."
  },
  {
    "startMs": 1008800,
    "text": "So I think it's very important to also"
  },
  {
    "startMs": 1010880,
    "text": "put scale elements in this loop. Uh and"
  },
  {
    "startMs": 1014720,
    "text": "I think also that uh human are super"
  },
  {
    "startMs": 1018399,
    "text": "useful here to basically judge the ID of"
  },
  {
    "startMs": 1021120,
    "text": "agents kind of steer them in the right"
  },
  {
    "startMs": 1023519,
    "text": "direction and so on. Um yeah so we"
  },
  {
    "startMs": 1026959,
    "text": "didn't try it yet I mean we are kind of"
  },
  {
    "startMs": 1029280,
    "text": "trying it right now and uh we hope that"
  },
  {
    "startMs": 1032000,
    "text": "this will lead to to to new discovery in"
  },
  {
    "startMs": 1035280,
    "text": "AI research at least and also a way is"
  },
  {
    "startMs": 1038480,
    "text": "that you can define multiple speedrun so"
  },
  {
    "startMs": 1042000,
    "text": "this is the next slide if you like it's"
  },
  {
    "startMs": 1044880,
    "text": "from safe bank slides but if you if you"
  },
  {
    "startMs": 1048319,
    "text": "don't have the reference good for you"
  },
  {
    "startMs": 1049760,
    "text": "means that that you're not too online uh"
  },
  {
    "startMs": 1052240,
    "text": "but the idea is that uh by changing the"
  },
  {
    "startMs": 1054799,
    "text": "object objective and the constraints of"
  },
  {
    "startMs": 1057039,
    "text": "the speedrun you can basically create a"
  },
  {
    "startMs": 1059440,
    "text": "lot of diversity and constrain the model"
  },
  {
    "startMs": 1061840,
    "text": "to go into a certain direction and uh"
  },
  {
    "startMs": 1064720,
    "text": "yeah and make those discovery."
  },
  {
    "startMs": 1067760,
    "text": "So uh at HIM in Ind."
  },
  {
    "startMs": 1072000,
    "text": "Uh there is a bunch of stuff here that"
  },
  {
    "startMs": 1073600,
    "text": "we I mean most of it we didn't release"
  },
  {
    "startMs": 1075919,
    "text": "yet but we are working on GPU sandboxing"
  },
  {
    "startMs": 1079360,
    "text": "to allow model to iterate into sandbox"
  },
  {
    "startMs": 1082559,
    "text": "because you need GPU sandbox for this"
  },
  {
    "startMs": 1084640,
    "text": "kind of stuff. We are working on our own"
  },
  {
    "startMs": 1087360,
    "text": "agents that are very efficient for like"
  },
  {
    "startMs": 1092160,
    "text": "RM framework. So it means like you have"
  },
  {
    "startMs": 1094080,
    "text": "a file system and you can write"
  },
  {
    "startMs": 1096080,
    "text": "information read from it. Uh and you"
  },
  {
    "startMs": 1098480,
    "text": "also do like this programmatic tool"
  },
  {
    "startMs": 1100240,
    "text": "coding thing. We also training a model"
  },
  {
    "startMs": 1102400,
    "text": "to be good at it on top of like open"
  },
  {
    "startMs": 1104880,
    "text": "source model. And uh the thing that we"
  },
  {
    "startMs": 1107840,
    "text": "already released is that we have those"
  },
  {
    "startMs": 1109919,
    "text": "set of liberation and product called"
  },
  {
    "startMs": 1111520,
    "text": "verifier primaril training where you can"
  },
  {
    "startMs": 1114720,
    "text": "basically train evaluate any"
  },
  {
    "startMs": 1116960,
    "text": "environments on any RS and the model"
  },
  {
    "startMs": 1120240,
    "text": "that you can train can be like GNM 5.2"
  },
  {
    "startMs": 1122400,
    "text": "too which is very big and and yeah we"
  },
  {
    "startMs": 1124400,
    "text": "have like we work a lot on making those"
  },
  {
    "startMs": 1126320,
    "text": "li very efficient to to ship the best"
  },
  {
    "startMs": 1129039,
    "text": "quality for for our clients. Yeah. Uh I"
  },
  {
    "startMs": 1134080,
    "text": "mean yeah super excited about this"
  },
  {
    "startMs": 1135919,
    "text": "domain. Once again I think it's super"
  },
  {
    "startMs": 1137840,
    "text": "important to have uh a part of like this"
  },
  {
    "startMs": 1141200,
    "text": "recursive self-improvement to happen in"
  },
  {
    "startMs": 1143600,
    "text": "the open because there is actually a lot"
  },
  {
    "startMs": 1145919,
    "text": "of people working that are not on big"
  },
  {
    "startMs": 1148160,
    "text": "labs. So you need to basically uh yeah"
  },
  {
    "startMs": 1152160,
    "text": "make it easy for people to understand"
  },
  {
    "startMs": 1153919,
    "text": "all those model work to do research and"
  },
  {
    "startMs": 1155760,
    "text": "so on. So that's kind of our goal and uh"
  },
  {
    "startMs": 1158720,
    "text": "yeah thanks a lot."
  }
]