[
  {
    "startMs": 13599,
    "text": "Yeah, I guess uh let's get started. I'm"
  },
  {
    "startMs": 15679,
    "text": "I'm Armen. I'm I'm the co-founder and"
  },
  {
    "startMs": 18320,
    "text": "CEO of Perceptron. Um, we'll get a"
  },
  {
    "startMs": 20880,
    "text": "little bit into into what we do, but uh,"
  },
  {
    "startMs": 23519,
    "text": "primarily what I want to talk about"
  },
  {
    "startMs": 24880,
    "text": "today is kind of our research stance"
  },
  {
    "startMs": 27279,
    "text": "that we want to move away from"
  },
  {
    "startMs": 29199,
    "text": "distinctions between VLMs, VAS, world"
  },
  {
    "startMs": 32559,
    "text": "models, whatever you want to call it, to"
  },
  {
    "startMs": 34079,
    "text": "something that we call embodied"
  },
  {
    "startMs": 35360,
    "text": "foundation models."
  },
  {
    "startMs": 42559,
    "text": "So specifically what we what we do at"
  },
  {
    "startMs": 45040,
    "text": "Perceptron kind of our northstar really"
  },
  {
    "startMs": 46960,
    "text": "is we want to be able to build physical"
  },
  {
    "startMs": 49039,
    "text": "AI foundations that give us the ability"
  },
  {
    "startMs": 51039,
    "text": "to perceive, understand and interact"
  },
  {
    "startMs": 53199,
    "text": "with the physical world in real time."
  },
  {
    "startMs": 55600,
    "text": "And so kind of the the northstar mission"
  },
  {
    "startMs": 57520,
    "text": "is really to bridge the the physical and"
  },
  {
    "startMs": 59520,
    "text": "digital world uh worlds fundamentally"
  },
  {
    "startMs": 62000,
    "text": "meaning that our our goal is kind of"
  },
  {
    "startMs": 64080,
    "text": "wherever there is a um a device, an"
  },
  {
    "startMs": 67439,
    "text": "instrument, a robot, a camera, a sensor,"
  },
  {
    "startMs": 70400,
    "text": "uh we're essentially there providing"
  },
  {
    "startMs": 71760,
    "text": "intelligence to it."
  },
  {
    "startMs": 74880,
    "text": "And specifically when we talk about this"
  },
  {
    "startMs": 76640,
    "text": "paradigm of being able to perceive,"
  },
  {
    "startMs": 78400,
    "text": "being able to reason, being able to act,"
  },
  {
    "startMs": 80479,
    "text": "we view this as kind of a a unification"
  },
  {
    "startMs": 83040,
    "text": "of uh traditional multimodal modeling."
  },
  {
    "startMs": 85920,
    "text": "So I came from fair, I was there for six"
  },
  {
    "startMs": 88080,
    "text": "years and and and my target there was"
  },
  {
    "startMs": 90720,
    "text": "really to try to figure out how to scale"
  },
  {
    "startMs": 92320,
    "text": "up recipes for multimodal models. And so"
  },
  {
    "startMs": 95439,
    "text": "one of the early things that we we"
  },
  {
    "startMs": 96960,
    "text": "started working on and we've published a"
  },
  {
    "startMs": 98400,
    "text": "lot in this domain was around early"
  },
  {
    "startMs": 99920,
    "text": "fusion. So this concept that you want to"
  },
  {
    "startMs": 101759,
    "text": "bring in all these modalities as early"
  },
  {
    "startMs": 103360,
    "text": "as you can and and the real complexity"
  },
  {
    "startMs": 106079,
    "text": "there is trying to figure out what is"
  },
  {
    "startMs": 107439,
    "text": "the correct way to actually properly"
  },
  {
    "startMs": 108960,
    "text": "represent all the different modalities"
  },
  {
    "startMs": 111360,
    "text": "uh both on the input and on on the"
  },
  {
    "startMs": 113119,
    "text": "output that you want to be able to"
  },
  {
    "startMs": 115040,
    "text": "represent holistically and so VLMs have"
  },
  {
    "startMs": 117840,
    "text": "kind of became the when I talk about"
  },
  {
    "startMs": 119439,
    "text": "multimodal models um you probably think"
  },
  {
    "startMs": 121600,
    "text": "of VLM. So this is the ability to take"
  },
  {
    "startMs": 123600,
    "text": "in some image video and some text and be"
  },
  {
    "startMs": 125439,
    "text": "able to output some some text"
  },
  {
    "startMs": 127600,
    "text": "essentially. And there's there's"
  },
  {
    "startMs": 129200,
    "text": "variations of this. There's there's"
  },
  {
    "startMs": 130879,
    "text": "models like the ER models, the embodied"
  },
  {
    "startMs": 132959,
    "text": "reasoning models that are able to output"
  },
  {
    "startMs": 134959,
    "text": "maybe some grounding points um that are"
  },
  {
    "startMs": 137120,
    "text": "able to do spatial understanding or"
  },
  {
    "startMs": 138720,
    "text": "reasoning a little bit better. Then we"
  },
  {
    "startMs": 140800,
    "text": "have things like VALAS that extend kind"
  },
  {
    "startMs": 142640,
    "text": "of the output domain apart from just"
  },
  {
    "startMs": 144400,
    "text": "text to now actions. And these are"
  },
  {
    "startMs": 146800,
    "text": "traditionally of course continue to be"
  },
  {
    "startMs": 148560,
    "text": "built on by standard VLM backbones."
  },
  {
    "startMs": 150959,
    "text": "Although there's been some efforts to"
  },
  {
    "startMs": 152560,
    "text": "try to migrate away from uh from VLMs to"
  },
  {
    "startMs": 156319,
    "text": "things like world models or world action"
  },
  {
    "startMs": 158000,
    "text": "models, although nothing that's been"
  },
  {
    "startMs": 159760,
    "text": "super fruitful just yet. And then we get"
  },
  {
    "startMs": 162319,
    "text": "into kind of more interesting and"
  },
  {
    "startMs": 164160,
    "text": "complex uh variants of multimodal models"
  },
  {
    "startMs": 166800,
    "text": "like world models where you essentially"
  },
  {
    "startMs": 168560,
    "text": "are outputting video uh from some inputs"
  },
  {
    "startMs": 171360,
    "text": "and the inputs can be either image or"
  },
  {
    "startMs": 172959,
    "text": "video or honestly image video and and"
  },
  {
    "startMs": 175200,
    "text": "actions. Um and the last point that I'll"
  },
  {
    "startMs": 177840,
    "text": "talk about is something recent which we"
  },
  {
    "startMs": 179920,
    "text": "call semantic world models which is you"
  },
  {
    "startMs": 181760,
    "text": "don't really output anything but you"
  },
  {
    "startMs": 183200,
    "text": "learn some type of representation that"
  },
  {
    "startMs": 184720,
    "text": "you think is useful um in the future."
  },
  {
    "startMs": 187360,
    "text": "What we kind of view as an embodied"
  },
  {
    "startMs": 188959,
    "text": "foundation model is actually uh a"
  },
  {
    "startMs": 191360,
    "text": "framing that allows you to both to do"
  },
  {
    "startMs": 193040,
    "text": "the standard perception the embodied"
  },
  {
    "startMs": 195120,
    "text": "reasoning and then the northstar target"
  },
  {
    "startMs": 197360,
    "text": "of control all within one model. Um, so"
  },
  {
    "startMs": 200560,
    "text": "being able to reason across different"
  },
  {
    "startMs": 202159,
    "text": "sense of modalities on the input and"
  },
  {
    "startMs": 204000,
    "text": "being able to do kind of most of what I"
  },
  {
    "startMs": 206239,
    "text": "mentioned on the output all within one"
  },
  {
    "startMs": 208239,
    "text": "single unified model."
  },
  {
    "startMs": 211599,
    "text": "And so very quickly I'm going to talk"
  },
  {
    "startMs": 213200,
    "text": "about two challenges and these are kind"
  },
  {
    "startMs": 214640,
    "text": "of fundamental research challenges that"
  },
  {
    "startMs": 216319,
    "text": "we face and I'll talk about kind of how"
  },
  {
    "startMs": 218159,
    "text": "our company has approached this um and"
  },
  {
    "startMs": 220319,
    "text": "and and what other folks are doing in"
  },
  {
    "startMs": 222080,
    "text": "here as well. So the very first thing is"
  },
  {
    "startMs": 225440,
    "text": "think about purely if you're going to"
  },
  {
    "startMs": 226799,
    "text": "try to model something like video like"
  },
  {
    "startMs": 228720,
    "text": "one hour of video depending on the"
  },
  {
    "startMs": 230720,
    "text": "representation you might have something"
  },
  {
    "startMs": 232080,
    "text": "like 1 million visual tokens that are"
  },
  {
    "startMs": 233840,
    "text": "coming in. The truth is is that there's"
  },
  {
    "startMs": 235840,
    "text": "not actually any ground truth that you"
  },
  {
    "startMs": 238080,
    "text": "can use uh effectively, right? So you"
  },
  {
    "startMs": 240480,
    "text": "can do things like and people have done"
  },
  {
    "startMs": 242319,
    "text": "this of course like pull out the"
  },
  {
    "startMs": 243920,
    "text": "transcripts uh predict the transcripts"
  },
  {
    "startMs": 246159,
    "text": "from the video or or maybe synthetically"
  },
  {
    "startMs": 248400,
    "text": "label some frames, ask it some questions"
  },
  {
    "startMs": 250720,
    "text": "and it turns out that this is kind of a"
  },
  {
    "startMs": 252640,
    "text": "humongous waste, right? So if you think"
  },
  {
    "startMs": 254159,
    "text": "about what's going into your model, you"
  },
  {
    "startMs": 255439,
    "text": "have 1 million tokens going in and"
  },
  {
    "startMs": 257120,
    "text": "you're essentially calculating the loss"
  },
  {
    "startMs": 258639,
    "text": "on something like you know 2% of all the"
  },
  {
    "startMs": 261840,
    "text": "tokens that are going in. And so this is"
  },
  {
    "startMs": 264080,
    "text": "very fundamentally problematic and and"
  },
  {
    "startMs": 265759,
    "text": "and the truth is is any way you try to"
  },
  {
    "startMs": 267759,
    "text": "figure out how to fix this, you're"
  },
  {
    "startMs": 269120,
    "text": "essentially injecting a wrong training"
  },
  {
    "startMs": 270639,
    "text": "signal. Either the signal is too sparse,"
  },
  {
    "startMs": 273280,
    "text": "it's too synthetic, or it's too"
  },
  {
    "startMs": 275040,
    "text": "indiscriminate. And so approaches beyond"
  },
  {
    "startMs": 277919,
    "text": "just synthetic enrichment have been,"
  },
  {
    "startMs": 279680,
    "text": "well, let's predict every single pixel."
  },
  {
    "startMs": 281600,
    "text": "I mean, true, this is a very dense"
  },
  {
    "startMs": 283520,
    "text": "signal, but it actually very poorly"
  },
  {
    "startMs": 285120,
    "text": "allocates attention. there's, you know,"
  },
  {
    "startMs": 287360,
    "text": "you're comparing a, you're treating a"
  },
  {
    "startMs": 288880,
    "text": "background pixel with the same degree of"
  },
  {
    "startMs": 290479,
    "text": "importance as you're treating a gripper"
  },
  {
    "startMs": 292240,
    "text": "tip or the contact points or the"
  },
  {
    "startMs": 294320,
    "text": "specific failures or the physics. And so"
  },
  {
    "startMs": 296880,
    "text": "what we do kind of at Perceptron is, and"
  },
  {
    "startMs": 299280,
    "text": "this is kind of some of our core IP is"
  },
  {
    "startMs": 301280,
    "text": "we think about what does a natural"
  },
  {
    "startMs": 302800,
    "text": "perceptive objective look like? So"
  },
  {
    "startMs": 304960,
    "text": "specifically, how can I predict the"
  },
  {
    "startMs": 306880,
    "text": "percepts that we think will matter in in"
  },
  {
    "startMs": 308800,
    "text": "the future in a very very automatic way."
  },
  {
    "startMs": 311600,
    "text": "So, as an example, you might hardcode"
  },
  {
    "startMs": 313360,
    "text": "something like, well, if you have a"
  },
  {
    "startMs": 314479,
    "text": "robotic arm, well, the the the tip of"
  },
  {
    "startMs": 316320,
    "text": "the grippers turns out to be a very"
  },
  {
    "startMs": 317759,
    "text": "useful percept that you can predict into"
  },
  {
    "startMs": 319440,
    "text": "the future. And folks have started doing"
  },
  {
    "startMs": 321199,
    "text": "this. Um, like the Momo app folks from"
  },
  {
    "startMs": 323680,
    "text": "AI2 have done this. There are other VAS"
  },
  {
    "startMs": 325759,
    "text": "that have done this. But this is still a"
  },
  {
    "startMs": 327360,
    "text": "hard-coded percept. So, the question is,"
  },
  {
    "startMs": 329280,
    "text": "can you figure out an automatic way that"
  },
  {
    "startMs": 331039,
    "text": "the model semantically is able to learn"
  },
  {
    "startMs": 333840,
    "text": "this very unique objective? And the"
  },
  {
    "startMs": 336560,
    "text": "truth is, we have figured out a way. uh"
  },
  {
    "startMs": 338639,
    "text": "we're not going to share how we do it"
  },
  {
    "startMs": 339840,
    "text": "here, but this is just kind of hinting"
  },
  {
    "startMs": 341199,
    "text": "at how we approach the the problem of of"
  },
  {
    "startMs": 343759,
    "text": "of sparity."
  },
  {
    "startMs": 346400,
    "text": "The the second core problem that we've"
  },
  {
    "startMs": 348240,
    "text": "spent a lot of time focusing on is is"
  },
  {
    "startMs": 350080,
    "text": "context bloat. So, if you have always on"
  },
  {
    "startMs": 352560,
    "text": "cameras, if you have robots that don't"
  },
  {
    "startMs": 354240,
    "text": "necessarily wait, they don't stop,"
  },
  {
    "startMs": 355919,
    "text": "you're essentially having to reason over"
  },
  {
    "startMs": 357520,
    "text": "a very very long um very long amount of"
  },
  {
    "startMs": 360880,
    "text": "tokens. And and so text is relatively"
  },
  {
    "startMs": 362960,
    "text": "dense and video is relatively sparse."
  },
  {
    "startMs": 365440,
    "text": "And so the question is, are there"
  },
  {
    "startMs": 367919,
    "text": "architectural breakthroughs that are"
  },
  {
    "startMs": 369600,
    "text": "there in order for you to be able to"
  },
  {
    "startMs": 370960,
    "text": "deal with this problem problem natively"
  },
  {
    "startMs": 373039,
    "text": "rather than just trying to on some"
  },
  {
    "startMs": 375600,
    "text": "synthetic level figure out how to fix"
  },
  {
    "startMs": 377600,
    "text": "this imbalance?"
  },
  {
    "startMs": 379919,
    "text": "Um, and so there's a couple of things"
  },
  {
    "startMs": 381919,
    "text": "that you can do. One kind of core first"
  },
  {
    "startMs": 384880,
    "text": "principle is you need to start treating"
  },
  {
    "startMs": 386400,
    "text": "different uh modalities as completely"
  },
  {
    "startMs": 388960,
    "text": "different. So you can't treat text"
  },
  {
    "startMs": 390560,
    "text": "tokens as the same as image tokens as"
  },
  {
    "startMs": 392319,
    "text": "the same as audio or video tokens. So,"
  },
  {
    "startMs": 395199,
    "text": "one thing that you can start thinking"
  },
  {
    "startMs": 396319,
    "text": "about doing is focusing on spatial"
  },
  {
    "startMs": 398160,
    "text": "compression or token compression. And"
  },
  {
    "startMs": 400319,
    "text": "people do really dumb things and we"
  },
  {
    "startMs": 401919,
    "text": "started off doing the dumb things and it"
  },
  {
    "startMs": 403360,
    "text": "does work. You can start thinking about"
  },
  {
    "startMs": 404880,
    "text": "averaging, you know, patch-wise"
  },
  {
    "startMs": 407039,
    "text": "representations across a a video or an"
  },
  {
    "startMs": 409600,
    "text": "image. Um, and you can start getting"
  },
  {
    "startMs": 411360,
    "text": "some interesting compression rates, you"
  },
  {
    "startMs": 413039,
    "text": "know, up up up to 10x."
  },
  {
    "startMs": 415680,
    "text": "But still, this is relatively of a hack"
  },
  {
    "startMs": 417360,
    "text": "and there's there's not, you know,"
  },
  {
    "startMs": 419280,
    "text": "wellused architectural methods to"
  },
  {
    "startMs": 421039,
    "text": "actually solve this. Um, so what we've"
  },
  {
    "startMs": 424080,
    "text": "done in in in the last couple of months,"
  },
  {
    "startMs": 425599,
    "text": "we've released what we think is our"
  },
  {
    "startMs": 427199,
    "text": "approach to dealing with um varying"
  },
  {
    "startMs": 430240,
    "text": "degrees of sparity, which is just let"
  },
  {
    "startMs": 432160,
    "text": "the model figure out what tokens it"
  },
  {
    "startMs": 433759,
    "text": "should look at and what tokens it"
  },
  {
    "startMs": 435120,
    "text": "shouldn't look like. And so we released"
  },
  {
    "startMs": 437120,
    "text": "our data sparse mixture of experts"
  },
  {
    "startMs": 438960,
    "text": "paper, which essentially allows you to"
  },
  {
    "startMs": 440560,
    "text": "do this. Um, it allows the model, well,"
  },
  {
    "startMs": 442960,
    "text": "there's a router in the model allows it"
  },
  {
    "startMs": 444720,
    "text": "to actually predict what token I should"
  },
  {
    "startMs": 447039,
    "text": "input, what token I should skip. Um, and"
  },
  {
    "startMs": 449440,
    "text": "it allows us to do it kind of for free"
  },
  {
    "startMs": 451680,
    "text": "through all the different layers."
  },
  {
    "startMs": 454560,
    "text": "And it turns out that if you just let"
  },
  {
    "startMs": 456160,
    "text": "the model learn, if you're not actually"
  },
  {
    "startMs": 457599,
    "text": "hard coding any significant um"
  },
  {
    "startMs": 460000,
    "text": "architectural priors, um, the model"
  },
  {
    "startMs": 462880,
    "text": "actually does learn. So if you end up"
  },
  {
    "startMs": 464560,
    "text": "visualizing the data sparse compute uh"
  },
  {
    "startMs": 467120,
    "text": "that our models use, you'll actually see"
  },
  {
    "startMs": 468960,
    "text": "that our models innately learn to start"
  },
  {
    "startMs": 471039,
    "text": "focusing on very um high density"
  },
  {
    "startMs": 473599,
    "text": "information or task relevant"
  },
  {
    "startMs": 475039,
    "text": "information. Um so in in in the upper"
  },
  {
    "startMs": 477280,
    "text": "right you can kind of see that the model"
  },
  {
    "startMs": 478639,
    "text": "decides to focus in on the graph which"
  },
  {
    "startMs": 480479,
    "text": "actually you as a human would also kind"
  },
  {
    "startMs": 482319,
    "text": "of zoom into because this is likely uh"
  },
  {
    "startMs": 485120,
    "text": "what's interesting within the figure. Um"
  },
  {
    "startMs": 487680,
    "text": "and it turns out that even task"
  },
  {
    "startMs": 489680,
    "text": "dependent um uh task dependent"
  },
  {
    "startMs": 492639,
    "text": "allocation ends up happening as well. So"
  },
  {
    "startMs": 494479,
    "text": "if you look at the bottom left if you"
  },
  {
    "startMs": 495840,
    "text": "just ask you know a very very general"
  },
  {
    "startMs": 498000,
    "text": "question you're kind of going to see an"
  },
  {
    "startMs": 500080,
    "text": "attention graph that is throughout the"
  },
  {
    "startMs": 501599,
    "text": "whole image. So the model just doesn't"
  },
  {
    "startMs": 503360,
    "text": "know what the proper way to allocate"
  },
  {
    "startMs": 504800,
    "text": "compute is. At the same time, if you ask"
  },
  {
    "startMs": 506720,
    "text": "it to do something like, you know,"
  },
  {
    "startMs": 507840,
    "text": "segment out all the fruit, you can see"
  },
  {
    "startMs": 509680,
    "text": "that it's going to allocate more tokens"
  },
  {
    "startMs": 511199,
    "text": "to what it thinks are fruit tokens. And"
  },
  {
    "startMs": 513440,
    "text": "so this is a very nice and clever trick"
  },
  {
    "startMs": 515200,
    "text": "that that we use and we've published and"
  },
  {
    "startMs": 517039,
    "text": "and other folks are starting to use"
  },
  {
    "startMs": 518719,
    "text": "around embedding priors into the"
  },
  {
    "startMs": 521680,
    "text": "architecture that are useful to deal"
  },
  {
    "startMs": 524320,
    "text": "with the sparsity imbalances of your"
  },
  {
    "startMs": 526320,
    "text": "modalities but not too harsh to the"
  },
  {
    "startMs": 528640,
    "text": "point that the models aren't actually"
  },
  {
    "startMs": 530399,
    "text": "learning uh natively."
  },
  {
    "startMs": 534160,
    "text": "And so kind of we we put all this"
  },
  {
    "startMs": 535920,
    "text": "together and you guys might have seen"
  },
  {
    "startMs": 536959,
    "text": "the release, but we essentially released"
  },
  {
    "startMs": 538640,
    "text": "our our our model which was uh what we"
  },
  {
    "startMs": 541760,
    "text": "considered to be the first embodied"
  },
  {
    "startMs": 543600,
    "text": "foundation model a couple weeks ago. And"
  },
  {
    "startMs": 545760,
    "text": "this model is essentially frontier um"
  },
  {
    "startMs": 548560,
    "text": "with respect to Gemini 3.1 Pro. It's"
  },
  {
    "startMs": 550640,
    "text": "actually better than Gemini embodied"
  },
  {
    "startMs": 552160,
    "text": "reasoning. Um and it's something like 15"
  },
  {
    "startMs": 554399,
    "text": "times cheaper. And it's essentially"
  },
  {
    "startMs": 555920,
    "text": "trained on this one pedibyte data set"
  },
  {
    "startMs": 558160,
    "text": "that we've collected across uh literally"
  },
  {
    "startMs": 560640,
    "text": "everything. It's uh from from internet"
  },
  {
    "startMs": 563360,
    "text": "crawls to to our own custommade training"
  },
  {
    "startMs": 567200,
    "text": "recipes or synthetic data pipelines. We"
  },
  {
    "startMs": 569120,
    "text": "have this one one pabyte of data across"
  },
  {
    "startMs": 571440,
    "text": "text, images, videos, trajectories. And"
  },
  {
    "startMs": 573760,
    "text": "these trajectories can be very very"
  },
  {
    "startMs": 575279,
    "text": "general. They can be desktop use"
  },
  {
    "startMs": 577279,
    "text": "trajectories. It can be playing a video"
  },
  {
    "startMs": 579120,
    "text": "game trajectory."
  },
  {
    "startMs": 581839,
    "text": "And it turns out that once you start"
  },
  {
    "startMs": 583200,
    "text": "doing these things, very interesting"
  },
  {
    "startMs": 585200,
    "text": "properties end up emerging. And so the"
  },
  {
    "startMs": 586959,
    "text": "the biggest property that we kind of saw"
  },
  {
    "startMs": 588959,
    "text": "which is kind of obvious in retrospect"
  },
  {
    "startMs": 590959,
    "text": "is that you can actually start thinking"
  },
  {
    "startMs": 592720,
    "text": "of doing classical CV tasks as being an"
  },
  {
    "startMs": 596160,
    "text": "agentic task. And so in this case we"
  },
  {
    "startMs": 598640,
    "text": "essentially reframe detection as an"
  },
  {
    "startMs": 600160,
    "text": "agentic task. So our model can you know"
  },
  {
    "startMs": 602560,
    "text": "write code it can it can ask to zoom in"
  },
  {
    "startMs": 604880,
    "text": "into specific portions. Uh it can change"
  },
  {
    "startMs": 607920,
    "text": "the contrast. Um and you can actually"
  },
  {
    "startMs": 610160,
    "text": "see here this is a very hard problem. I"
  },
  {
    "startMs": 612480,
    "text": "think there's a whole Reddit subreddit"
  },
  {
    "startMs": 614160,
    "text": "of these problems of trying to find very"
  },
  {
    "startMs": 616079,
    "text": "hard objects and images. And our models"
  },
  {
    "startMs": 618399,
    "text": "essentially do this very well, but they"
  },
  {
    "startMs": 619760,
    "text": "do this in an agentic sense. So this"
  },
  {
    "startMs": 621440,
    "text": "isn't the this isn't a classical, you"
  },
  {
    "startMs": 623680,
    "text": "know, detect this one box. This is the"
  },
  {
    "startMs": 625839,
    "text": "model actually deciding that it needs to"
  },
  {
    "startMs": 627360,
    "text": "tile things up. It needs to change the"
  },
  {
    "startMs": 629040,
    "text": "contrast. It proposes a box here. I"
  },
  {
    "startMs": 631760,
    "text": "think here it yeah uh increases contrast"
  },
  {
    "startMs": 635120,
    "text": "and it can find the bird. So again, very"
  },
  {
    "startMs": 637760,
    "text": "hard to do if if like even for a human"
  },
  {
    "startMs": 640160,
    "text": "and humans are very good at perceptive"
  },
  {
    "startMs": 641680,
    "text": "tasks. This is a relatively tough thing"
  },
  {
    "startMs": 643600,
    "text": "to do. And this all comes from just"
  },
  {
    "startMs": 645680,
    "text": "having natively embodied models that"
  },
  {
    "startMs": 648399,
    "text": "actually understand how to look at"
  },
  {
    "startMs": 649839,
    "text": "different modalities."
  },
  {
    "startMs": 651600,
    "text": "And so following up kind of how does"
  },
  {
    "startMs": 654000,
    "text": "this relate to the general physical AI"
  },
  {
    "startMs": 656640,
    "text": "stance around robotics? So I kind of"
  },
  {
    "startMs": 658160,
    "text": "stole this slide from from GDM folks."
  },
  {
    "startMs": 660959,
    "text": "And so one thing we've we're starting to"
  },
  {
    "startMs": 662800,
    "text": "see from kind of robotics um u agentic"
  },
  {
    "startMs": 666800,
    "text": "systems is this separation between what"
  },
  {
    "startMs": 669440,
    "text": "we call kind of embodied reasoning"
  },
  {
    "startMs": 671040,
    "text": "models or orchestrators and tactile"
  },
  {
    "startMs": 673519,
    "text": "policy models. You can think of problems"
  },
  {
    "startMs": 675519,
    "text": "as like you know if I have a if if I'm"
  },
  {
    "startMs": 679200,
    "text": "making coffee and that takes me three"
  },
  {
    "startMs": 680560,
    "text": "minutes to do I mean one thing I can do"
  },
  {
    "startMs": 681839,
    "text": "is try to force my whole you know VLA to"
  },
  {
    "startMs": 684079,
    "text": "try to figure out how to do this"
  },
  {
    "startMs": 685440,
    "text": "individual task or what I can do is I"
  },
  {
    "startMs": 687360,
    "text": "can have an orchestrator model that"
  },
  {
    "startMs": 688800,
    "text": "breaks up this tasks into subtasks and"
  },
  {
    "startMs": 691120,
    "text": "there's a tactile control policy that's"
  },
  {
    "startMs": 692959,
    "text": "running on top and there's kind of a"
  },
  {
    "startMs": 694480,
    "text": "full spectrum between you know full VA"
  },
  {
    "startMs": 696720,
    "text": "only all the way to this kind of agentic"
  },
  {
    "startMs": 698720,
    "text": "system. Uh but the main thing I'm trying"
  },
  {
    "startMs": 700480,
    "text": "to highlight is that embodied reasoning"
  },
  {
    "startMs": 702079,
    "text": "is actually very interesting and complex"
  },
  {
    "startMs": 704480,
    "text": "problem that is yet to be solved. Um"
  },
  {
    "startMs": 706560,
    "text": "that being said, our models continue to"
  },
  {
    "startMs": 708240,
    "text": "be frontier on embodied reasoning. And"
  },
  {
    "startMs": 711120,
    "text": "because they're frontier, we start"
  },
  {
    "startMs": 712640,
    "text": "seeing really cool things that we"
  },
  {
    "startMs": 713839,
    "text": "haven't seen before. Here's a concrete"
  },
  {
    "startMs": 716560,
    "text": "example of doing very complex egocentric"
  },
  {
    "startMs": 719120,
    "text": "or not egocentric, but this is robotic"
  },
  {
    "startMs": 721279,
    "text": "data annotation. And you can kind of see"
  },
  {
    "startMs": 723200,
    "text": "here the model is jumping around looking"
  },
  {
    "startMs": 725040,
    "text": "at different portions of the video, uh,"
  },
  {
    "startMs": 727360,
    "text": "you know, clipping it, figuring out"
  },
  {
    "startMs": 728880,
    "text": "whether or not the captions are correct,"
  },
  {
    "startMs": 730560,
    "text": "self-ver uh, self-verifying. And we can"
  },
  {
    "startMs": 733519,
    "text": "all do this because AR models are fast."
  },
  {
    "startMs": 735519,
    "text": "They're significantly cheaper than"
  },
  {
    "startMs": 737360,
    "text": "anything else that's out there. So, if"
  },
  {
    "startMs": 739040,
    "text": "you try to do this with Gemini, this"
  },
  {
    "startMs": 740399,
    "text": "video would probably cost you a couple"
  },
  {
    "startMs": 741920,
    "text": "of dollars where for us, it's probably"
  },
  {
    "startMs": 743440,
    "text": "in the sense. Um, and these all kind of"
  },
  {
    "startMs": 745839,
    "text": "emerged from being able to have these"
  },
  {
    "startMs": 747680,
    "text": "frontier embodied reasoning capabilities"
  },
  {
    "startMs": 749839,
    "text": "that we just previously have not seen"
  },
  {
    "startMs": 751519,
    "text": "from other models."
  },
  {
    "startMs": 755600,
    "text": "Um,"
  },
  {
    "startMs": 757279,
    "text": "okay. Probably going to share the the"
  },
  {
    "startMs": 759040,
    "text": "biggest research breakthrough that we've"
  },
  {
    "startMs": 760639,
    "text": "had and I think we'll we'll share more"
  },
  {
    "startMs": 762959,
    "text": "of this uh in the upcoming weeks"
  },
  {
    "startMs": 765040,
    "text": "probably on Twitter. Uh, but one thing"
  },
  {
    "startMs": 766959,
    "text": "that we found is we've we've we've"
  },
  {
    "startMs": 768639,
    "text": "discovered new scaling laws for embodied"
  },
  {
    "startMs": 770959,
    "text": "foundation models. So these are models"
  },
  {
    "startMs": 772800,
    "text": "again that you can jointly do"
  },
  {
    "startMs": 774800,
    "text": "control-based training, you can do"
  },
  {
    "startMs": 776560,
    "text": "trajectory training, you can do"
  },
  {
    "startMs": 778720,
    "text": "perceptive training, you can do embodied"
  },
  {
    "startMs": 780639,
    "text": "reasoning training. If you just figure"
  },
  {
    "startMs": 782240,
    "text": "out what the right way to mix this all"
  },
  {
    "startMs": 783920,
    "text": "together is and the right objectives to"
  },
  {
    "startMs": 785600,
    "text": "use, you actually start seeing very"
  },
  {
    "startMs": 787440,
    "text": "interesting levers that you maybe"
  },
  {
    "startMs": 788880,
    "text": "previously haven't been able to see"
  },
  {
    "startMs": 790240,
    "text": "before. So the concrete lever that I'll"
  },
  {
    "startMs": 793040,
    "text": "talk about is this ability to trade um"
  },
  {
    "startMs": 796959,
    "text": "this ability to trade general video uh"
  },
  {
    "startMs": 800079,
    "text": "pre-training data for teleop data. So"
  },
  {
    "startMs": 802639,
    "text": "kind of as we know teleop data is very"
  },
  {
    "startMs": 804320,
    "text": "expensive. It's on the orders of you"
  },
  {
    "startMs": 806079,
    "text": "know $100 per hour of data for a hundred"
  },
  {
    "startMs": 809440,
    "text": "you know dollars I can collect"
  },
  {
    "startMs": 810880,
    "text": "significantly more video pre-training"
  },
  {
    "startMs": 812639,
    "text": "data. And so what this graph is showing"
  },
  {
    "startMs": 814800,
    "text": "is that um if you're just training pure"
  },
  {
    "startMs": 817839,
    "text": "VA's pure policies there's this band"
  },
  {
    "startMs": 819920,
    "text": "that you have. So you do actually still"
  },
  {
    "startMs": 821519,
    "text": "have scaling loss. So you do get"
  },
  {
    "startMs": 823200,
    "text": "benefits from more and more teop data."
  },
  {
    "startMs": 825360,
    "text": "That being said, the benefits are not as"
  },
  {
    "startMs": 827040,
    "text": "substantial as if you are really"
  },
  {
    "startMs": 828399,
    "text": "training these unified embodied"
  },
  {
    "startMs": 830399,
    "text": "foundation models. And so this is what"
  },
  {
    "startMs": 832959,
    "text": "kind of the bottom bottom half of the"
  },
  {
    "startMs": 834720,
    "text": "graph is. Uh and the really cool kind of"
  },
  {
    "startMs": 838079,
    "text": "uh lever that we get is you can"
  },
  {
    "startMs": 840320,
    "text": "essentially trade 10x uh less teleop"
  },
  {
    "startMs": 843199,
    "text": "data if you have 10x more video"
  },
  {
    "startMs": 845199,
    "text": "pre-training data. And so far this is"
  },
  {
    "startMs": 847680,
    "text": "kind of held for the amount of compute"
  },
  {
    "startMs": 849360,
    "text": "that our company has. Um and it will be"
  },
  {
    "startMs": 851680,
    "text": "continuing to to kind of push the fold"
  },
  {
    "startMs": 854160,
    "text": "on uh how far you can push these"
  },
  {
    "startMs": 856560,
    "text": "embodied foundation models."
  },
  {
    "startMs": 862800,
    "text": "So here's a couple of videos of of a"
  },
  {
    "startMs": 865360,
    "text": "policy that hopefully will will um we'll"
  },
  {
    "startMs": 868160,
    "text": "open source one of the smaller models in"
  },
  {
    "startMs": 869680,
    "text": "a couple of weeks. Uh but this is all"
  },
  {
    "startMs": 871839,
    "text": "running natively within a single model"
  },
  {
    "startMs": 873760,
    "text": "that is capable of doing the embodied"
  },
  {
    "startMs": 876000,
    "text": "reasoning in order to figure out the"
  },
  {
    "startMs": 877920,
    "text": "task actually is outputting control"
  },
  {
    "startMs": 880000,
    "text": "tokens. Um you can see it's a little bit"
  },
  {
    "startMs": 882320,
    "text": "jittery but that's okay. Hopefully it'll"
  },
  {
    "startMs": 884240,
    "text": "be figured out at scale and you can"
  },
  {
    "startMs": 886079,
    "text": "actually see very complex tasks that"
  },
  {
    "startMs": 888240,
    "text": "previously I think would be really tough"
  },
  {
    "startMs": 889760,
    "text": "for pure VA to do. So I think if you"
  },
  {
    "startMs": 892160,
    "text": "look at the right hand video, this is"
  },
  {
    "startMs": 894399,
    "text": "requiring them all to actually read the"
  },
  {
    "startMs": 896079,
    "text": "title of the book, have the knowledge"
  },
  {
    "startMs": 897920,
    "text": "about what type of book this is and then"
  },
  {
    "startMs": 900079,
    "text": "properly allocate it within one of the"
  },
  {
    "startMs": 902000,
    "text": "bins. Um so this is actually a"
  },
  {
    "startMs": 903760,
    "text": "multi-step task between perception and"
  },
  {
    "startMs": 906000,
    "text": "control that is really really tough to"
  },
  {
    "startMs": 907920,
    "text": "do if you have a kind of a pure"
  },
  {
    "startMs": 909440,
    "text": "endto-end control model that is not"
  },
  {
    "startMs": 911120,
    "text": "aware of the different perceptive tasks"
  },
  {
    "startMs": 914240,
    "text": "that it needs to accomplish in order to"
  },
  {
    "startMs": 915760,
    "text": "do this task."
  },
  {
    "startMs": 923519,
    "text": "Yeah,"
  },
  {
    "startMs": 925199,
    "text": "it's pretty cool. It also works zero"
  },
  {
    "startMs": 927199,
    "text": "shot relatively well out of the box. So,"
  },
  {
    "startMs": 929600,
    "text": "we're excited to get this in the hands"
  },
  {
    "startMs": 931040,
    "text": "of folks in in a couple of weeks"
  },
  {
    "startMs": 933120,
    "text": "sometime in July."
  },
  {
    "startMs": 938160,
    "text": "Um, yeah, going to leave a couple of"
  },
  {
    "startMs": 940240,
    "text": "minutes for for for general questions,"
  },
  {
    "startMs": 942320,
    "text": "but if you guys are interested, let's uh"
  },
  {
    "startMs": 945519,
    "text": "let's connect. So, one cool thing that"
  },
  {
    "startMs": 947199,
    "text": "we do with our company is we actually"
  },
  {
    "startMs": 949040,
    "text": "for a limited set of partners give"
  },
  {
    "startMs": 950480,
    "text": "access to our Mark1 weights. We give"
  },
  {
    "startMs": 952240,
    "text": "access to our larger embodied foundation"
  },
  {
    "startMs": 954480,
    "text": "models weights. So, email me, DM me on"
  },
  {
    "startMs": 957040,
    "text": "Twitter, uh, whatever is easier. Um, and"
  },
  {
    "startMs": 959759,
    "text": "then yeah, I'll open up. There's a"
  },
  {
    "startMs": 961120,
    "text": "couple minutes left for for questions."
  },
  {
    "startMs": 977759,
    "text": "these models"
  },
  {
    "startMs": 982399,
    "text": "and like when I saw"
  },
  {
    "startMs": 991839,
    "text": "a lot of contss"
  },
  {
    "startMs": 999360,
    "text": "So can you talk a little bit on that?"
  },
  {
    "startMs": 1001199,
    "text": ">> Yeah, I think it's it's so yeah. So the"
  },
  {
    "startMs": 1003360,
    "text": "question is how are we able to nail"
  },
  {
    "startMs": 1004560,
    "text": "temporal and uh understanding to this"
  },
  {
    "startMs": 1007279,
    "text": "degree? Um it's a good question. I mean"
  },
  {
    "startMs": 1009680,
    "text": "to be honest it's not nailed. So there's"
  },
  {
    "startMs": 1011519,
    "text": "still a lot of work to actually get it"
  },
  {
    "startMs": 1012800,
    "text": "to a place where you can reliably deploy"
  },
  {
    "startMs": 1014399,
    "text": "it. The the core thing is how do you"
  },
  {
    "startMs": 1016959,
    "text": "think about context management? So you"
  },
  {
    "startMs": 1018959,
    "text": "have relatively limited context. So I"
  },
  {
    "startMs": 1021519,
    "text": "think the the the models here have 1"
  },
  {
    "startMs": 1023279,
    "text": "million context, but that's relatively"
  },
  {
    "startMs": 1024880,
    "text": "easy to fit in with the, you know, high"
  },
  {
    "startMs": 1027039,
    "text": "FPS video. And so you have to start"
  },
  {
    "startMs": 1029039,
    "text": "thinking about, are there interesting"
  },
  {
    "startMs": 1030400,
    "text": "things that you can do? I'll throw"
  },
  {
    "startMs": 1032559,
    "text": "something out there. We we used to do"
  },
  {
    "startMs": 1034000,
    "text": "this, but we got past this, but like how"
  },
  {
    "startMs": 1035760,
    "text": "do I think about like uh uh key frames"
  },
  {
    "startMs": 1038559,
    "text": "versus delta frames? How can I manage my"
  },
  {
    "startMs": 1040559,
    "text": "context by training these two off? Um,"
  },
  {
    "startMs": 1043600,
    "text": "and then you start thinking about during"
  },
  {
    "startMs": 1045280,
    "text": "your pre-training objective, how can I"
  },
  {
    "startMs": 1046959,
    "text": "start kind of natively ingesting things"
  },
  {
    "startMs": 1050080,
    "text": "that I think will be useful for the"
  },
  {
    "startMs": 1051600,
    "text": "robotics tasks. Um, so for example, a"
  },
  {
    "startMs": 1054400,
    "text": "very basic thing that even kind of the"
  },
  {
    "startMs": 1055840,
    "text": "the Gemini models used to struggle at, I"
  },
  {
    "startMs": 1058080,
    "text": "think the new ones are pretty good, but"
  },
  {
    "startMs": 1059440,
    "text": "being able to tell cardalities. So like"
  },
  {
    "startMs": 1061360,
    "text": "left and right is very hard to tell if"
  },
  {
    "startMs": 1063120,
    "text": "you do internet scale crawls because no"
  },
  {
    "startMs": 1065280,
    "text": "one on the internet is necessarily"
  },
  {
    "startMs": 1066720,
    "text": "labeling things as, you know, this"
  },
  {
    "startMs": 1068400,
    "text": "object is to the left of this object,"
  },
  {
    "startMs": 1070240,
    "text": "it's below this object. So really"
  },
  {
    "startMs": 1072240,
    "text": "thinking about data distributions early"
  },
  {
    "startMs": 1073840,
    "text": "on gives you this ability relatively"
  },
  {
    "startMs": 1075840,
    "text": "quickly. And it's also like er this type"
  },
  {
    "startMs": 1078240,
    "text": "of embodied reasoning was a very"
  },
  {
    "startMs": 1079600,
    "text": "concrete focus with this which is why I"
  },
  {
    "startMs": 1082240,
    "text": "think we were able to kind of surpass"
  },
  {
    "startMs": 1083919,
    "text": "Gemini ER with relatively less compute."
  },
  {
    "startMs": 1092480,
    "text": ">> One issue with"
  },
  {
    "startMs": 1096640,
    "text": "sometimes I was wondering whether"
  },
  {
    "startMs": 1100480,
    "text": "you looked"
  },
  {
    "startMs": 1102000,
    "text": "your models are more robust than"
  },
  {
    "startMs": 1108160,
    "text": ">> Yeah. So, probably the coolest"
  },
  {
    "startMs": 1109840,
    "text": "robustness that we've seen is that uh"
  },
  {
    "startMs": 1112559,
    "text": "for VA models specifically, like if you"
  },
  {
    "startMs": 1114559,
    "text": "go and you take one of the Chinese ones"
  },
  {
    "startMs": 1117200,
    "text": "um and you try to fine-tune it for a"
  },
  {
    "startMs": 1118640,
    "text": "specific policy, if you just change the"
  },
  {
    "startMs": 1120640,
    "text": "background of the I don't know, you can"
  },
  {
    "startMs": 1123200,
    "text": "like in the table uh if you change the"
  },
  {
    "startMs": 1125200,
    "text": "the background of the table, the policy"
  },
  {
    "startMs": 1127280,
    "text": "will actually fail. What's really"
  },
  {
    "startMs": 1129039,
    "text": "interesting, if you do this type of"
  },
  {
    "startMs": 1130240,
    "text": "joint perceptive and control modeling,"
  },
  {
    "startMs": 1132000,
    "text": "you're much more robust to these types"
  },
  {
    "startMs": 1133760,
    "text": "of errors or if the light is hitting it"
  },
  {
    "startMs": 1136000,
    "text": "a slightly different way. And I think we"
  },
  {
    "startMs": 1138080,
    "text": "primarily view this as robustness to"
  },
  {
    "startMs": 1139919,
    "text": "background uh in a way that I think"
  },
  {
    "startMs": 1142160,
    "text": "traditional models don't necessarily"
  },
  {
    "startMs": 1143760,
    "text": "have. That being said, I'm not going to"
  },
  {
    "startMs": 1145360,
    "text": "overclaim like uh I mean like it's still"
  },
  {
    "startMs": 1147440,
    "text": "relatively hard. I think if I was going"
  },
  {
    "startMs": 1148799,
    "text": "to go and shine a flashlight into one of"
  },
  {
    "startMs": 1150559,
    "text": "the one of the arms, it's probably not"
  },
  {
    "startMs": 1152720,
    "text": "going to work. Uh but being able to"
  },
  {
    "startMs": 1154960,
    "text": "jointly model these things helps a"
  },
  {
    "startMs": 1157039,
    "text": "significant amount. We also do a lot of"
  },
  {
    "startMs": 1158720,
    "text": "online augmentations. So, so we do"
  },
  {
    "startMs": 1160799,
    "text": "actually, you know, fake um I don't know"
  },
  {
    "startMs": 1163760,
    "text": "um one of the one of the arm cameras"
  },
  {
    "startMs": 1166960,
    "text": "being off, right? We fake uh you know,"
  },
  {
    "startMs": 1169919,
    "text": "sunlight coming in from a certain"
  },
  {
    "startMs": 1171440,
    "text": "direction, right? So, we do these things"
  },
  {
    "startMs": 1173600,
    "text": "during training to improve robustness."
  },
  {
    "startMs": 1175679,
    "text": "Uh but the big gains come from taking"
  },
  {
    "startMs": 1177919,
    "text": "this early fusion paradigm and then"
  },
  {
    "startMs": 1179679,
    "text": "moving it into the robotics domain."
  },
  {
    "startMs": 1184880,
    "text": ">> Cool."
  },
  {
    "startMs": 1187039,
    "text": "uh create knowledge base using the K1"
  },
  {
    "startMs": 1190080,
    "text": "model."
  },
  {
    "startMs": 1191600,
    "text": ">> Uh oh, knowledge bases. Um it's useful"
  },
  {
    "startMs": 1193840,
    "text": "for like I I mean if you want to caption"
  },
  {
    "startMs": 1195919,
    "text": "images, videos, it's relatively well. We"
  },
  {
    "startMs": 1198080,
    "text": "work with robotics partners for like"
  },
  {
    "startMs": 1199840,
    "text": "very complex egocentric annotation like"
  },
  {
    "startMs": 1201919,
    "text": "this. Um so it's not necessarily"
  },
  {
    "startMs": 1203679,
    "text": "building out an ontology, but being able"
  },
  {
    "startMs": 1205440,
    "text": "to do kind of very deep structured"
  },
  {
    "startMs": 1207679,
    "text": "extraction, I think our models are are"
  },
  {
    "startMs": 1209440,
    "text": "very good at. Yeah. By the way,"
  },
  {
    "startMs": 1211760,
    "text": "everything that I kind of showed here is"
  },
  {
    "startMs": 1213280,
    "text": "kind of public APIs, so you can go play"
  },
  {
    "startMs": 1215280,
    "text": "around with it. The benchmarks are"
  },
  {
    "startMs": 1216880,
    "text": "public. Um, I think I'm out of time."
  },
  {
    "startMs": 1219760,
    "text": "They're cutting me off. So, uh, I can"
  },
  {
    "startMs": 1222080,
    "text": "talk with folks outside, but thank you"
  },
  {
    "startMs": 1223840,
    "text": "guys."
  }
]