Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
8:29:11Hey, guys.
8:29:13Very excited to be here.
8:29:14Thank you for staying for the last talk.
8:29:17Welcome to my talk about the matrix and how we're living through it.
8:29:24I have been working on Runway for almost a decade now.
8:29:27Started here in New York, maybe around 10 years ago.
8:29:31I want to work through some of the learnings we have seen through building the company,
8:29:35building this research lab here in New York, and where do we see a lot of the work that
8:29:40we do in video, in pixel generation, really moving forward.
8:29:43So I don't exaggerate and I joke that this might be a simulation or not.
8:29:49Maybe for perspective, I will walk you through a piece of history.
8:29:53This is like watching prehistoric ancient ruins.
8:29:58is our first AI film we
8:30:00ever made. It was done in 2019. It's called The Orange because, as you can tell very clearly, it's an orange that falls from the tree and walks through different parts of the world. This is done, this is a video model, an image model that we had back then that was creating pixels in 128 pixels by 128 pixels that we then forcefully up-sample to this kind of resolution.
8:30:24I think of this as art now.
8:30:26This is like looking at a film from the 1900s.
8:30:29You know, it's beautiful.
8:30:30There's a lot of history behind it.
8:30:32But this is 2019.
8:30:34We started the company in 2018, 2017.
8:30:37And so we're trying to figure out if this had any lags.
8:30:40We're trying to convince ourselves that if you scale this,
8:30:43it will naturally improve to the point where we can get somewhere close to this.
8:30:48This is a state-of-the-art video today.
8:30:50This is all generated.
8:30:52None of these characters are real.
8:30:53None of those locations are real.
8:30:55I'm sure if you're probably here, you know this.
8:30:58By now, you probably have seen this online, on Twitter.
8:31:01The models are just incredibly good.
8:31:03And the most interesting thing is that when I look at stuff like this, I still believe
8:31:08this will look like the orange in just a couple more years.
8:31:11So I want to walk you through how we're thinking not only through what we're doing at Runway
8:31:16today but also in the next couple of years and the research that's shaping that.
8:31:22We are a company that's at the intersection of both art and science.
8:31:26We have deep roots in artistic and creative communities, and we have some of the best researchers in the world working on diffusion models and large video models.
8:31:35And I like that combination because it allows us to bring the best of both worlds.
8:31:41We're a team of scientists and artists thinking about the problem of scaling video and world models.
8:31:47but in a way the core idea of the company was already discovered by cinema and by moving pictures
8:31:52120 130 years ago which is if you think about video and maybe you remember this from uh from
8:31:59high school video is an optical illusion video is the illusion of movement that if you do if you
8:32:04stitch frames fast enough you will create movement you will create the illusion of movement you
8:32:08create motion pictures and so at run we were doing kind of something very similar we're taking that
8:32:13idea to the next level, how do we do next frame prediction fast enough, scale enough,
8:32:20in such a way that we can hopefully do more than just pretty and nice creative videos.
8:32:26And so, Aloto is powered by, of course, scaling laws.
8:32:31I think we believed in scaling laws not only applied for language models but also for video.
8:32:37And this has been true.
8:32:38We're now really ramping up the amount of compute that we put behind the models that
8:32:43you already have seen.
8:32:44And I think the interesting thing about seeing the models scale over time is the emerging
8:32:49properties that appear, the intelligence that appears when you start taking this very simple
8:32:54idea that cinema found itself figuring out 120 years ago, which is next frame prediction,
8:33:01and taking next frame prediction really to the next level.
8:33:03So if I take an image like this, which is not real, those two people are not real, and
8:33:07And I just keep on predicting what the next frame will look like based on the previous
8:33:11frames.
8:33:12And if I do it fast enough, and if I do it with a network that's big enough, I'll get
8:33:17movement.
8:33:19I'll get delusion of movement.
8:33:20I'll get something that looks so real to us that we humans can distinguish.
8:33:24I look at that video and I've been doing this for 10 years.
8:33:27I can't tell you if that's real or not.
8:33:30And we measure that.
8:33:31We've actually measured and tried to quantify if people can distinguish between real and
8:33:36I think that Turing test was already passed.
8:33:39People are not able to distinguish.
8:33:40But the most interesting thing is here is that really what's happening as you predict
8:33:44next frames is you're learning a lot more than just pretty frames.
8:33:48You're learning about physics.
8:33:50You're learning about object permanence.
8:33:51You're learning about gravity.
8:33:52You're learning about occlusion.
8:33:54You're learning about movements.
8:33:55You're learning about how the objects work in the real world.
8:33:59And you're doing that all through just video, through images, through next frame prediction.
8:34:04So if you look at a complex thing like this, a good video model for it to understand how
8:34:11to make this a consistent video will need to naturally understand what all these subjects
8:34:17are, how each of them relate to one another, how a camel might probably stand on a ball
8:34:22like that, what water in a container looks like.
8:34:25And if I do all of that together and I do it fast enough, I'll get, hopefully, a consistent
8:34:29video that looks real to us.
8:34:31So, a lot of the benchmarks that we use to evaluate our video models are not only creative,
8:34:39it basically means do you like it or do you don't, but are actually based on physics,
8:34:45on geometries, on how objects behave in the real world and how those objects behave, and
8:34:50also simulator or the real world.
8:34:53So, you take that idea of next-frame prediction and you do it fast enough and you get, of
8:34:57And of course, now something that looks more like what we should expect from a video application
8:35:01or a game application.
8:35:02Again, this is just video, right?
8:35:04And you take that idea to a next level.
8:35:07What happens if we start this idea of next-round prediction, but do it in the context of a 3D
8:35:12environment or something that looks similar to a game?
8:35:14You can start actually getting closer to a physics engine, a simulation system.
8:35:18But the simulation system has no physics laws.
8:35:21We didn't describe what the behavior of the object should be.
8:35:25The model just knows how to do that by poor force of scale.
8:35:32And this gets kind of interesting because you can only think about playing worlds.
8:35:36This is not GTA.
8:35:38This is someone playing someone else in an environment and a software that doesn't access.
8:35:45It's all simulation of the simulation, which gets really meta, right?
8:35:49And so I'm not kidding when I said welcome to my metric simulation.
8:35:53But the most interesting thing for me is that everything you saw before was not real-time,
8:35:59that it took a couple of minutes or seconds for those videos to be made.
8:36:03A lot of what we're seeing right now, when we think about the next three, four years
8:36:07of video models and world models, is to deeply think about what happens when you start not
8:36:14only scaling the models and making them much, much, much bigger, with much more emerging
8:36:20properties but also what happens when you start making them interactive.
8:36:23So take the same idea of next frame prediction, you take a frame, you predict the next frame,
8:36:27then X, then X, then X, but then it can act at any point in time in that frame prediction,
8:36:32which means I can simulate any possible divergence or action in that world.
8:36:37So in this world, I could play the video, but I can also come and touch every part,
8:36:43any aspect of that image, and make it interactive to affect the consequences of that world.
8:36:50And it works.
8:36:53And of course, this is incredibly useful for a variety of different use cases.
8:36:58As you can imagine, we're now spinning an entire robotics lab in Runway.
8:37:03So if you come to our office in New York, half of it is what looks like a Hollywood
8:37:07studio, half of it is a bunch of robots walking through and moving through the office.
8:37:11And so we're able to make these models or start making these models the brains of how
8:37:17autonomous and embodiment systems walk and reason about the world.
8:37:21But there's a couple of more interesting use cases that I think go beyond the too obvious
8:37:26one, which is video and robotics, which is what happens when you think about anything,
8:37:31anything, any pixel on a screen as a video model.
8:37:35This is a great example.
8:37:37This is a video model.
8:37:38What you see right now, well, this is an image, first of all.
8:37:40That image could be real or not, but I could run one of our video models in real time and
8:37:45make this an interactive application.
8:37:46So let's say I want to turn the lights of the car.
8:37:50Maybe I want to turn the car around.
8:37:52I can just click through it.
8:37:54I can turn it around again, maybe open the hood and see what's inside the car.
8:37:58Now, everything you just saw here has no physical 3D representation.
8:38:03There's no Unity.
8:38:05There's no Unreal.
8:38:06There's no Blender.
8:38:07It's just a video model.
8:38:09A video model has been trained to understand actions and clicks in an interface.
8:38:14We call this a world interface model.
8:38:19And the world interface model, I feel you guys understand now, it's an interface and
8:38:24it's a world model, world interface model, allows you to do very cool stuff like this.
8:38:28And so here's another application, another example of how you could think about using
8:38:33a model like this in real time.
8:38:35I can take an image, a representation, move objects around and expect them to move as
8:38:41I was actually in that room.
8:38:44Again, remember, there's no 3D representation of this.
8:38:48This environment might not actually have been modeled before.
8:38:51It's just an image and a model that understands how objects move in a space because of this
8:38:55very simple idea that next frame prediction scale well enough can become a world model
8:39:00or a simulation engine.
8:39:03So with the eight minutes I have left, I want to show you this idea taken to another extreme
8:39:11we're thinking not only about what happens if you can control the school applications and the school
8:39:15interfaces but what happens if you keep extrapolating and i think by the way extrapolating
8:39:22may be one of the most important skills these days in ai in company building what happens over the
8:39:27next six months 12 months 18 months may be one of the most important things to to try to predict
8:39:32understand um and so we ask ourselves that question a lot at runway what happens what will
8:39:38happen if we keep scaling the models, if we keep adding more data, if we keep making them
8:39:42better simulation systems, if we make them universal simulation systems that can predict
8:39:46anything on a screen, literally anything on a screen.
8:39:49So we've been working on this idea that we call Continuum.
8:39:52And Continuum is the idea of real-time video models or role models taken really to
8:40:00the next degree. This is an operating system. The idea here is that you might open your computer at some point in the near future and what you see streaming to your screen might be coming from a video model.
8:40:10So I will show you next specific how this idea could shape. This is still very early research, but I want to show you this idea of combining video models as part of your operating system, as part of your daily work.
8:40:24So here's an example.
8:40:25I was researching San Francisco and the fog in San Francisco.
8:40:30I wanted to learn more about when and how it happens.
8:40:33This is structural data.
8:40:34This is actually a website.
8:40:35This is something you can take from the Internet.
8:40:39And now Continuum has this idea that I can select any part of content in that website
8:40:43and make it a portal.
8:40:44We call this a portal.
8:40:46And the portal is a real-time video.
8:40:47It's fully interactive and fully based on the content you're experiencing
8:40:51experiencing reading through, I probably won't read through all of that text, it's too much
8:40:55text for me, so I just want to make a video and I can make a video that's represented
8:40:58by the text you see there.
8:41:00These portals can be 3D realizations, it can be any content that you're watching or interacting
8:41:04with.
8:41:05On the right, you can have all the visualizations and portals you've experienced before, anything
8:41:09you've been trying to learn.
8:41:11Basically, it's a memory, it's the disk of the imaginations or the visualizations you've
8:41:16seen before.
8:41:16So here I was researching about batteries.
8:41:18And so I can go back to the portal I had around batteries before.
8:41:24And since there are real-time interactive applications, I can click them anywhere.
8:41:27I can move them around.
8:41:29I can start asking questions, and the videos and the content itself will change based on
8:41:34my interactions and the way I'm managing through it.
8:41:41Here's another very cool example.
8:41:42that the first time I saw it, when the team showed it to me, it blew my mind.
8:41:46Since it's a real-time video, you can make it full interactive with anything.
8:41:50Not only with clicks, but resizing the application should also change the behavior.
8:41:55So I can resize.
8:41:56I'm learning about photosynthesis here.
8:41:58If I resize the application, I get a different experience.
8:42:02And this is, again, full interactive application, so I can click through it, navigate through
8:42:07parts of the cell or how photosynthesis works to get a better understanding of what I'm trying
8:42:11to learn and do.
8:42:14The one thing that still feels to me kind of unsigned about this is I don't know how
8:42:18to call this.
8:42:19I don't know if this is ‑‑ it doesn't feel like a visualization, a video, a game,
8:42:24something in between.
8:42:25And I think we need to do the collective work of figuring out how we call these kind of
8:42:28interactions in the first place.
8:42:30And I'm sure you hear the other thing is that you can change the structure of the prompt
8:42:34or what's actually creating that interaction in the first place.
8:42:40Here is another example. I'm going to type something. And this is one of the ones that
8:42:50I feel has been the most under explored yet. We call this visual thinking. So we've connected
8:42:57Continuum to a coding agent. And what the coding agent can do is it can use Continuum to visualize
8:43:03through a problem. So imagine you ask your coding agent to create a 3D model for your bike,
8:43:09and instead of the model trying to just think through that via code, the model can spin a
8:43:13version of continuum to think through the problem visually. So if you ever think about someone,
8:43:18I'm sure you have friends or maybe a lot of people here may be visual thinkers, what a visual
8:43:23thinker is, is it thinks through problems and through ideas visually first. And that helps you
8:43:29ground and project some of the best things you want to do.
8:43:33This is somehow very similar.
8:43:35Agents have the ability to do visual thinking.
8:43:38Agents can spin up a sort of imagination that you can tap and interact with.
8:43:44So not only the agent can interact with it, I can also see it myself.
8:43:48And I can guide the agent in much better ways than I might have been able to guide it just
8:43:53via text alone.
8:43:54Which is for us a very powerful idea.
8:43:56It's a completely new paradigm.
8:44:00visual thinking. The last example I have is how you think about taking Continuum to become
8:44:12much more than just using or navigating interactive structural content in the way I've showed
8:44:19you before, but actually making them fully live experiences that are reactive. So in
8:44:26And as Continuum understands the intention of what I'm trying to search from, it also
8:44:32suggests interfaces that are specifically designed in real time just for that.
8:44:37So here it created for me this Mars outpost that actually should be accurate with the
8:44:42data about the actual robots that are in Mars right now or the ones that are projected maybe
8:44:47in 2030.
8:44:48And since this is fully interactive, I can interact with it.
8:44:50I can go to places.
8:44:51I can move around in the same way that I might move around in a 3D system or a 3D continuum.
8:44:59And it works.
8:45:01Just fascinating.
8:45:06So when I say everything will be simulated, I'm really trying to understand or help not
8:45:13really our team but the world get to the point where a lot of what we'll see today will found
8:45:18obvious hopefully in a few more years or months.
8:45:20We'll look back at Continuum
8:45:22and it might feel very close
8:45:23to the video of the Orance that we just showed you.
8:45:26Continuum and a lot of the things we're doing
8:45:27in video and in runway when it comes
8:45:30to simulation are still in very early days,
8:45:32but we're getting to a point where it actually
8:45:34works and it really scales.
8:45:35So what we're really excited about is that for the next
8:45:378 to 12 months, we're releasing most of what
8:45:39I showed here for everyone to use.
8:45:41Thank you, guys.
8:45:43Thank
