1
00:00:01,326 --> 00:00:01,986
Hey, guys.

2
00:00:03,266 --> 00:00:04,586
Very excited to be here.

3
00:00:04,966 --> 00:00:06,526
Thank you for staying for the last talk.

4
00:00:07,966 --> 00:00:11,326
Welcome to my talk about the matrix and how we're living through it.

5
00:00:14,386 --> 00:00:16,826
I have been working on Runway for almost a decade now.

6
00:00:17,106 --> 00:00:20,166
Started here in New York, maybe around 10 years ago.

7
00:00:21,846 --> 00:00:25,466
I want to work through some of the learnings we have seen through building the company,

8
00:00:25,666 --> 00:00:30,026
building this research lab here in New York, and where do we see a lot of the work that

9
00:00:30,026 --> 00:00:33,546
we do in video, in pixel generation, really moving forward.

10
00:00:33,766 --> 00:00:38,226
So I don't exaggerate and I joke that this might be a simulation or not.

11
00:00:39,786 --> 00:00:43,686
Maybe for perspective, I will walk you through a piece of history.

12
00:00:43,806 --> 00:00:47,306
This is like watching prehistoric ancient ruins.

13
00:00:48,586 --> 00:00:49,966
is our first AI film we

14
00:00:50,026 --> 00:01:13,726
ever made. It was done in 2019. It's called The Orange because, as you can tell very clearly, it's an orange that falls from the tree and walks through different parts of the world. This is done, this is a video model, an image model that we had back then that was creating pixels in 128 pixels by 128 pixels that we then forcefully up-sample to this kind of resolution.

15
00:01:14,806 --> 00:01:16,446
I think of this as art now.

16
00:01:16,686 --> 00:01:19,246
This is like looking at a film from the 1900s.

17
00:01:19,366 --> 00:01:20,186
You know, it's beautiful.

18
00:01:20,806 --> 00:01:22,306
There's a lot of history behind it.

19
00:01:22,566 --> 00:01:23,606
But this is 2019.

20
00:01:24,426 --> 00:01:26,766
We started the company in 2018, 2017.

21
00:01:27,186 --> 00:01:29,646
And so we're trying to figure out if this had any lags.

22
00:01:30,546 --> 00:01:33,186
We're trying to convince ourselves that if you scale this,

23
00:01:33,446 --> 00:01:37,826
it will naturally improve to the point where we can get somewhere close to this.

24
00:01:38,546 --> 00:01:40,126
This is a state-of-the-art video today.

25
00:01:40,426 --> 00:01:42,126
This is all generated.

26
00:01:42,406 --> 00:01:43,626
None of these characters are real.

27
00:01:43,726 --> 00:01:45,366
None of those locations are real.

28
00:01:45,506 --> 00:01:47,526
I'm sure if you're probably here, you know this.

29
00:01:48,086 --> 00:01:50,706
By now, you probably have seen this online, on Twitter.

30
00:01:51,386 --> 00:01:53,166
The models are just incredibly good.

31
00:01:53,586 --> 00:01:58,066
And the most interesting thing is that when I look at stuff like this, I still believe

32
00:01:58,066 --> 00:02:00,826
this will look like the orange in just a couple more years.

33
00:02:01,566 --> 00:02:06,346
So I want to walk you through how we're thinking not only through what we're doing at Runway

34
00:02:06,346 --> 00:02:10,826
today but also in the next couple of years and the research that's shaping that.

35
00:02:12,106 --> 00:02:15,986
We are a company that's at the intersection of both art and science.

36
00:02:16,266 --> 00:02:25,226
We have deep roots in artistic and creative communities, and we have some of the best researchers in the world working on diffusion models and large video models.

37
00:02:25,866 --> 00:02:30,646
And I like that combination because it allows us to bring the best of both worlds.

38
00:02:31,266 --> 00:02:36,426
We're a team of scientists and artists thinking about the problem of scaling video and world models.

39
00:02:37,026 --> 00:02:42,906
but in a way the core idea of the company was already discovered by cinema and by moving pictures

40
00:02:42,906 --> 00:02:49,306
120 130 years ago which is if you think about video and maybe you remember this from uh from

41
00:02:49,306 --> 00:02:54,806
high school video is an optical illusion video is the illusion of movement that if you do if you

42
00:02:54,806 --> 00:02:59,006
stitch frames fast enough you will create movement you will create the illusion of movement you

43
00:02:59,006 --> 00:03:03,226
create motion pictures and so at run we were doing kind of something very similar we're taking that

44
00:03:03,226 --> 00:03:09,286
idea to the next level, how do we do next frame prediction fast enough, scale enough,

45
00:03:10,166 --> 00:03:15,586
in such a way that we can hopefully do more than just pretty and nice creative videos.

46
00:03:16,346 --> 00:03:21,306
And so, Aloto is powered by, of course, scaling laws.

47
00:03:21,486 --> 00:03:27,266
I think we believed in scaling laws not only applied for language models but also for video.

48
00:03:27,466 --> 00:03:28,366
And this has been true.

49
00:03:28,366 --> 00:03:33,206
We're now really ramping up the amount of compute that we put behind the models that

50
00:03:33,206 --> 00:03:34,086
you already have seen.

51
00:03:34,706 --> 00:03:39,826
And I think the interesting thing about seeing the models scale over time is the emerging

52
00:03:39,826 --> 00:03:44,526
properties that appear, the intelligence that appears when you start taking this very simple

53
00:03:44,526 --> 00:03:50,726
idea that cinema found itself figuring out 120 years ago, which is next frame prediction,

54
00:03:51,066 --> 00:03:53,466
and taking next frame prediction really to the next level.

55
00:03:53,646 --> 00:03:57,986
So if I take an image like this, which is not real, those two people are not real, and

56
00:03:57,986 --> 00:04:02,006
And I just keep on predicting what the next frame will look like based on the previous

57
00:04:02,006 --> 00:04:02,366
frames.

58
00:04:02,686 --> 00:04:07,946
And if I do it fast enough, and if I do it with a network that's big enough, I'll get

59
00:04:07,946 --> 00:04:08,806
movement.

60
00:04:09,126 --> 00:04:09,906
I'll get delusion of movement.

61
00:04:10,066 --> 00:04:14,226
I'll get something that looks so real to us that we humans can distinguish.

62
00:04:14,426 --> 00:04:17,006
I look at that video and I've been doing this for 10 years.

63
00:04:17,406 --> 00:04:18,766
I can't tell you if that's real or not.

64
00:04:20,026 --> 00:04:21,066
And we measure that.

65
00:04:21,366 --> 00:04:26,146
We've actually measured and tried to quantify if people can distinguish between real and

66
00:04:26,146 --> 00:04:28,566
I think that Turing test was already passed.

67
00:04:29,246 --> 00:04:30,306
People are not able to distinguish.

68
00:04:30,526 --> 00:04:34,346
But the most interesting thing is here is that really what's happening as you predict

69
00:04:34,346 --> 00:04:37,986
next frames is you're learning a lot more than just pretty frames.

70
00:04:38,206 --> 00:04:39,906
You're learning about physics.

71
00:04:40,126 --> 00:04:41,626
You're learning about object permanence.

72
00:04:41,746 --> 00:04:42,686
You're learning about gravity.

73
00:04:42,926 --> 00:04:44,466
You're learning about occlusion.

74
00:04:44,626 --> 00:04:45,606
You're learning about movements.

75
00:04:45,866 --> 00:04:49,226
You're learning about how the objects work in the real world.

76
00:04:49,486 --> 00:04:54,446
And you're doing that all through just video, through images, through next frame prediction.

77
00:04:54,446 --> 00:05:01,366
So if you look at a complex thing like this, a good video model for it to understand how

78
00:05:01,366 --> 00:05:07,466
to make this a consistent video will need to naturally understand what all these subjects

79
00:05:07,466 --> 00:05:12,546
are, how each of them relate to one another, how a camel might probably stand on a ball

80
00:05:12,546 --> 00:05:14,826
like that, what water in a container looks like.

81
00:05:15,026 --> 00:05:19,866
And if I do all of that together and I do it fast enough, I'll get, hopefully, a consistent

82
00:05:19,866 --> 00:05:21,686
video that looks real to us.

83
00:05:21,686 --> 00:05:28,906
So, a lot of the benchmarks that we use to evaluate our video models are not only creative,

84
00:05:29,226 --> 00:05:34,826
it basically means do you like it or do you don't, but are actually based on physics,

85
00:05:35,226 --> 00:05:40,766
on geometries, on how objects behave in the real world and how those objects behave, and

86
00:05:40,766 --> 00:05:42,846
also simulator or the real world.

87
00:05:43,046 --> 00:05:47,146
So, you take that idea of next-frame prediction and you do it fast enough and you get, of

88
00:05:47,146 --> 00:05:51,346
And of course, now something that looks more like what we should expect from a video application

89
00:05:51,346 --> 00:05:52,266
or a game application.

90
00:05:52,706 --> 00:05:54,286
Again, this is just video, right?

91
00:05:54,886 --> 00:05:57,126
And you take that idea to a next level.

92
00:05:57,366 --> 00:06:02,066
What happens if we start this idea of next-round prediction, but do it in the context of a 3D

93
00:06:02,066 --> 00:06:04,006
environment or something that looks similar to a game?

94
00:06:04,166 --> 00:06:08,406
You can start actually getting closer to a physics engine, a simulation system.

95
00:06:08,566 --> 00:06:11,786
But the simulation system has no physics laws.

96
00:06:11,966 --> 00:06:15,846
We didn't describe what the behavior of the object should be.

97
00:06:15,846 --> 00:06:21,946
The model just knows how to do that by poor force of scale.

98
00:06:22,726 --> 00:06:26,726
And this gets kind of interesting because you can only think about playing worlds.

99
00:06:26,986 --> 00:06:27,886
This is not GTA.

100
00:06:28,426 --> 00:06:35,086
This is someone playing someone else in an environment and a software that doesn't access.

101
00:06:35,306 --> 00:06:38,906
It's all simulation of the simulation, which gets really meta, right?

102
00:06:39,466 --> 00:06:42,266
And so I'm not kidding when I said welcome to my metric simulation.

103
00:06:43,906 --> 00:06:48,566
But the most interesting thing for me is that everything you saw before was not real-time,

104
00:06:49,426 --> 00:06:52,886
that it took a couple of minutes or seconds for those videos to be made.

105
00:06:53,646 --> 00:06:57,786
A lot of what we're seeing right now, when we think about the next three, four years

106
00:06:57,786 --> 00:07:04,366
of video models and world models, is to deeply think about what happens when you start not

107
00:07:04,366 --> 00:07:10,606
only scaling the models and making them much, much, much bigger, with much more emerging

108
00:07:10,606 --> 00:07:13,286
properties but also what happens when you start making them interactive.

109
00:07:13,846 --> 00:07:17,606
So take the same idea of next frame prediction, you take a frame, you predict the next frame,

110
00:07:17,726 --> 00:07:22,686
then X, then X, then X, but then it can act at any point in time in that frame prediction,

111
00:07:22,926 --> 00:07:27,086
which means I can simulate any possible divergence or action in that world.

112
00:07:27,386 --> 00:07:33,786
So in this world, I could play the video, but I can also come and touch every part,

113
00:07:33,966 --> 00:07:39,506
any aspect of that image, and make it interactive to affect the consequences of that world.

114
00:07:40,406 --> 00:07:41,186
And it works.

115
00:07:43,686 --> 00:07:48,186
And of course, this is incredibly useful for a variety of different use cases.

116
00:07:48,726 --> 00:07:52,806
As you can imagine, we're now spinning an entire robotics lab in Runway.

117
00:07:53,106 --> 00:07:57,086
So if you come to our office in New York, half of it is what looks like a Hollywood

118
00:07:57,086 --> 00:08:00,926
studio, half of it is a bunch of robots walking through and moving through the office.

119
00:08:01,246 --> 00:08:07,266
And so we're able to make these models or start making these models the brains of how

120
00:08:07,266 --> 00:08:10,466
autonomous and embodiment systems walk and reason about the world.

121
00:08:11,966 --> 00:08:17,006
But there's a couple of more interesting use cases that I think go beyond the too obvious

122
00:08:17,006 --> 00:08:21,246
one, which is video and robotics, which is what happens when you think about anything,

123
00:08:21,366 --> 00:08:24,126
anything, any pixel on a screen as a video model.

124
00:08:25,506 --> 00:08:26,446
This is a great example.

125
00:08:27,026 --> 00:08:27,826
This is a video model.

126
00:08:28,446 --> 00:08:30,226
What you see right now, well, this is an image, first of all.

127
00:08:30,466 --> 00:08:35,066
That image could be real or not, but I could run one of our video models in real time and

128
00:08:35,066 --> 00:08:36,366
make this an interactive application.

129
00:08:36,366 --> 00:08:40,046
So let's say I want to turn the lights of the car.

130
00:08:40,286 --> 00:08:41,926
Maybe I want to turn the car around.

131
00:08:42,606 --> 00:08:43,866
I can just click through it.

132
00:08:44,246 --> 00:08:47,766
I can turn it around again, maybe open the hood and see what's inside the car.

133
00:08:48,166 --> 00:08:53,466
Now, everything you just saw here has no physical 3D representation.

134
00:08:53,826 --> 00:08:54,946
There's no Unity.

135
00:08:55,186 --> 00:08:56,106
There's no Unreal.

136
00:08:56,406 --> 00:08:57,246
There's no Blender.

137
00:08:57,646 --> 00:08:58,866
It's just a video model.

138
00:08:59,086 --> 00:09:04,246
A video model has been trained to understand actions and clicks in an interface.

139
00:09:04,246 --> 00:09:06,526
We call this a world interface model.

140
00:09:09,146 --> 00:09:14,206
And the world interface model, I feel you guys understand now, it's an interface and

141
00:09:14,206 --> 00:09:18,146
it's a world model, world interface model, allows you to do very cool stuff like this.

142
00:09:18,266 --> 00:09:23,486
And so here's another application, another example of how you could think about using

143
00:09:23,486 --> 00:09:24,746
a model like this in real time.

144
00:09:26,006 --> 00:09:31,446
I can take an image, a representation, move objects around and expect them to move as

145
00:09:31,446 --> 00:09:34,406
I was actually in that room.

146
00:09:34,686 --> 00:09:37,926
Again, remember, there's no 3D representation of this.

147
00:09:38,046 --> 00:09:41,186
This environment might not actually have been modeled before.

148
00:09:41,506 --> 00:09:45,886
It's just an image and a model that understands how objects move in a space because of this

149
00:09:45,886 --> 00:09:50,506
very simple idea that next frame prediction scale well enough can become a world model

150
00:09:50,506 --> 00:09:51,426
or a simulation engine.

151
00:09:53,826 --> 00:09:59,346
So with the eight minutes I have left, I want to show you this idea taken to another extreme

152
00:10:01,186 --> 00:10:05,986
we're thinking not only about what happens if you can control the school applications and the school

153
00:10:05,986 --> 00:10:12,066
interfaces but what happens if you keep extrapolating and i think by the way extrapolating

154
00:10:12,066 --> 00:10:17,906
may be one of the most important skills these days in ai in company building what happens over the

155
00:10:17,906 --> 00:10:22,806
next six months 12 months 18 months may be one of the most important things to to try to predict

156
00:10:22,806 --> 00:10:28,126
understand um and so we ask ourselves that question a lot at runway what happens what will

157
00:10:28,126 --> 00:10:32,466
happen if we keep scaling the models, if we keep adding more data, if we keep making them

158
00:10:32,466 --> 00:10:36,846
better simulation systems, if we make them universal simulation systems that can predict

159
00:10:36,846 --> 00:10:39,206
anything on a screen, literally anything on a screen.

160
00:10:39,346 --> 00:10:42,086
So we've been working on this idea that we call Continuum.

161
00:10:42,506 --> 00:10:49,946
And Continuum is the idea of real-time video models or role models taken really to

162
00:10:50,026 --> 00:10:59,626
the next degree. This is an operating system. The idea here is that you might open your computer at some point in the near future and what you see streaming to your screen might be coming from a video model.

163
00:11:00,386 --> 00:11:13,226
So I will show you next specific how this idea could shape. This is still very early research, but I want to show you this idea of combining video models as part of your operating system, as part of your daily work.

164
00:11:14,146 --> 00:11:15,246
So here's an example.

165
00:11:15,686 --> 00:11:20,106
I was researching San Francisco and the fog in San Francisco.

166
00:11:20,526 --> 00:11:22,946
I wanted to learn more about when and how it happens.

167
00:11:23,166 --> 00:11:24,246
This is structural data.

168
00:11:24,386 --> 00:11:25,286
This is actually a website.

169
00:11:25,606 --> 00:11:28,986
This is something you can take from the Internet.

170
00:11:29,546 --> 00:11:33,606
And now Continuum has this idea that I can select any part of content in that website

171
00:11:33,606 --> 00:11:34,586
and make it a portal.

172
00:11:34,786 --> 00:11:35,766
We call this a portal.

173
00:11:36,026 --> 00:11:37,446
And the portal is a real-time video.

174
00:11:37,826 --> 00:11:41,226
It's fully interactive and fully based on the content you're experiencing

175
00:11:41,226 --> 00:11:45,026
experiencing reading through, I probably won't read through all of that text, it's too much

176
00:11:45,026 --> 00:11:48,806
text for me, so I just want to make a video and I can make a video that's represented

177
00:11:48,806 --> 00:11:50,186
by the text you see there.

178
00:11:50,846 --> 00:11:54,826
These portals can be 3D realizations, it can be any content that you're watching or interacting

179
00:11:54,826 --> 00:11:55,146
with.

180
00:11:55,546 --> 00:11:59,746
On the right, you can have all the visualizations and portals you've experienced before, anything

181
00:11:59,746 --> 00:12:00,746
you've been trying to learn.

182
00:12:01,586 --> 00:12:06,106
Basically, it's a memory, it's the disk of the imaginations or the visualizations you've

183
00:12:06,106 --> 00:12:06,566
seen before.

184
00:12:06,566 --> 00:12:08,606
So here I was researching about batteries.

185
00:12:08,906 --> 00:12:13,826
And so I can go back to the portal I had around batteries before.

186
00:12:14,166 --> 00:12:17,746
And since there are real-time interactive applications, I can click them anywhere.

187
00:12:17,966 --> 00:12:18,826
I can move them around.

188
00:12:19,126 --> 00:12:24,286
I can start asking questions, and the videos and the content itself will change based on

189
00:12:24,286 --> 00:12:27,806
my interactions and the way I'm managing through it.

190
00:12:31,126 --> 00:12:32,986
Here's another very cool example.

191
00:12:32,986 --> 00:12:35,886
that the first time I saw it, when the team showed it to me, it blew my mind.

192
00:12:36,626 --> 00:12:39,846
Since it's a real-time video, you can make it full interactive with anything.

193
00:12:40,286 --> 00:12:44,626
Not only with clicks, but resizing the application should also change the behavior.

194
00:12:45,106 --> 00:12:46,506
So I can resize.

195
00:12:46,786 --> 00:12:48,386
I'm learning about photosynthesis here.

196
00:12:48,686 --> 00:12:51,246
If I resize the application, I get a different experience.

197
00:12:52,386 --> 00:12:57,306
And this is, again, full interactive application, so I can click through it, navigate through

198
00:12:57,306 --> 00:13:01,866
parts of the cell or how photosynthesis works to get a better understanding of what I'm trying

199
00:13:01,866 --> 00:13:02,706
to learn and do.

200
00:13:04,466 --> 00:13:08,766
The one thing that still feels to me kind of unsigned about this is I don't know how

201
00:13:08,766 --> 00:13:09,426
to call this.

202
00:13:09,586 --> 00:13:14,366
I don't know if this is ‑‑ it doesn't feel like a visualization, a video, a game,

203
00:13:14,886 --> 00:13:15,766
something in between.

204
00:13:15,986 --> 00:13:18,886
And I think we need to do the collective work of figuring out how we call these kind of

205
00:13:18,886 --> 00:13:19,966
interactions in the first place.

206
00:13:20,566 --> 00:13:24,466
And I'm sure you hear the other thing is that you can change the structure of the prompt

207
00:13:24,466 --> 00:13:26,886
or what's actually creating that interaction in the first place.

208
00:13:30,646 --> 00:13:40,306
Here is another example. I'm going to type something. And this is one of the ones that

209
00:13:40,306 --> 00:13:47,986
I feel has been the most under explored yet. We call this visual thinking. So we've connected

210
00:13:47,986 --> 00:13:53,806
Continuum to a coding agent. And what the coding agent can do is it can use Continuum to visualize

211
00:13:53,806 --> 00:13:58,926
through a problem. So imagine you ask your coding agent to create a 3D model for your bike,

212
00:13:59,066 --> 00:14:03,766
and instead of the model trying to just think through that via code, the model can spin a

213
00:14:03,766 --> 00:14:08,446
version of continuum to think through the problem visually. So if you ever think about someone,

214
00:14:08,646 --> 00:14:13,726
I'm sure you have friends or maybe a lot of people here may be visual thinkers, what a visual

215
00:14:13,726 --> 00:14:19,966
thinker is, is it thinks through problems and through ideas visually first. And that helps you

216
00:14:19,966 --> 00:14:23,166
ground and project some of the best things you want to do.

217
00:14:23,466 --> 00:14:25,046
This is somehow very similar.

218
00:14:25,886 --> 00:14:28,346
Agents have the ability to do visual thinking.

219
00:14:28,806 --> 00:14:33,826
Agents can spin up a sort of imagination that you can tap and interact with.

220
00:14:34,346 --> 00:14:37,846
So not only the agent can interact with it, I can also see it myself.

221
00:14:38,206 --> 00:14:43,606
And I can guide the agent in much better ways than I might have been able to guide it just

222
00:14:43,606 --> 00:14:44,646
via text alone.

223
00:14:44,906 --> 00:14:46,806
Which is for us a very powerful idea.

224
00:14:46,946 --> 00:14:48,046
It's a completely new paradigm.

225
00:14:50,506 --> 00:15:03,006
visual thinking. The last example I have is how you think about taking Continuum to become

226
00:15:03,006 --> 00:15:09,066
much more than just using or navigating interactive structural content in the way I've showed

227
00:15:09,066 --> 00:15:16,586
you before, but actually making them fully live experiences that are reactive. So in

228
00:15:16,586 --> 00:15:22,046
And as Continuum understands the intention of what I'm trying to search from, it also

229
00:15:22,046 --> 00:15:26,666
suggests interfaces that are specifically designed in real time just for that.

230
00:15:27,126 --> 00:15:32,706
So here it created for me this Mars outpost that actually should be accurate with the

231
00:15:32,706 --> 00:15:37,746
data about the actual robots that are in Mars right now or the ones that are projected maybe

232
00:15:37,746 --> 00:15:38,186
in 2030.

233
00:15:38,806 --> 00:15:40,886
And since this is fully interactive, I can interact with it.

234
00:15:40,986 --> 00:15:41,766
I can go to places.

235
00:15:41,766 --> 00:15:47,066
I can move around in the same way that I might move around in a 3D system or a 3D continuum.

236
00:15:49,206 --> 00:15:49,966
And it works.

237
00:15:51,426 --> 00:15:52,226
Just fascinating.

238
00:15:56,926 --> 00:16:03,526
So when I say everything will be simulated, I'm really trying to understand or help not

239
00:16:03,526 --> 00:16:08,206
really our team but the world get to the point where a lot of what we'll see today will found

240
00:16:08,206 --> 00:16:10,306
obvious hopefully in a few more years or months.

241
00:16:10,726 --> 00:16:12,126
We'll look back at Continuum

242
00:16:12,126 --> 00:16:13,946
and it might feel very close

243
00:16:13,946 --> 00:16:15,686
to the video of the Orance that we just showed you.

244
00:16:16,406 --> 00:16:17,826
Continuum and a lot of the things we're doing

245
00:16:17,826 --> 00:16:20,026
in video and in runway when it comes

246
00:16:20,026 --> 00:16:22,266
to simulation are still in very early days,

247
00:16:22,406 --> 00:16:24,046
but we're getting to a point where it actually

248
00:16:24,046 --> 00:16:25,466
works and it really scales.

249
00:16:25,846 --> 00:16:27,746
So what we're really excited about is that for the next

250
00:16:27,746 --> 00:16:30,006
8 to 12 months, we're releasing most of what

251
00:16:30,006 --> 00:16:31,766
I showed here for everyone to use.

252
00:16:31,986 --> 00:16:32,646
Thank you, guys.

253
00:16:33,326 --> 00:16:33,466
Thank