Video Has No Memory. Here's How We Built One.

[on-hold music] Thanks so much for having me and, uh, inviting me to, to be a speaker at, uh, the WorldFair.

You know, I, I attended last year and was so impressed about the quality of presenters, so, so glad to, to have a chance to be here and present. Uh, so the title of my talk is, you know, Video Has No Memory, right?

Uh, and, you know, uh, this might sound strange because video is already like a preservation of the past, right? If you think about, like, you have footage, you preserve, uh, recording, training data, incident, creative work, uh, history, et cetera.

But actually, most of the video AI systems these days do not have memory in the system sense. So actually, for this talk, I will try to answer the question, like, what could it take to build a memory layer for video intelligence?

To start, I wanna be clear about what makes video different from other data type, right? So this is the first mental model that I wanna highlight, which is that video is not a bag of frames.

Um, so in, you know, many of my conversation with, like, developers, uh, you know, who are using our product, a lot of them still treat video as, like, a stack of images, uh, maybe a transcript being attached or, you know, um, uh, like, you know, but, uh, essentially like a, like a frame level, right?

And that is useful approximation for some tasks, but it throw away the thing that makes video very unique, which is continuity, right? So meaning in video derives from space, time, modalities, and sequence.

So a better mental model for video is a spatiotemporal volume. So what I mean that inside that volume you have visual information, speech, sound, motion, OCR, camera changes, scene transition, metadata, and time, right?

So the hard part here is really about how can you preserving relationship across this volume so that later an application can traverse it. And then the, you know, uh, especially at the enterprise scale, you know, across, like, industry, like entertainment, sport, you know, um, short-form content, then you're sitting on petabytes of, of footage, right?

So finding moment is already hard, so how can you present meaning across millions of moments in the deeper platform? So I work at Twelve Lab, which is a, a Series B, uh, startup.

Um, we build foundation models that understand, you know, video the way that human do. Um, and the way we talk about our positioning is like the existing, uh, stack of, of dealing with video is not equipped to do that, right?

Obviously, language model are very powerful. They are good reasoning interfaces. They are increasingly multimodal as well. But the supporting stack, right, around that is very, um, I'd say limited, and that create three problem.

Number one is wrong context, right? So video is not naturally a sequence of text token. If we force it into that sequence by sampling frames, by extracting a transcript, uh, by dumping everything into a prompt, you lose the spatiotemporal relationships, right, that actually define the event.

Second, uh, wrong memory. So if you think about text system memory here is often mean retrieval augmented generation, vector search, or probably like larger context window. Uh, those are very useful, but video memory has a different requirement.

It needs to link today's scene for something that happened in another file, another episode, another camera angle, another season, another year. So we actually need durable continuity, right? And the last part here is wrong reasoning.

Like I said, you know, text-first system cannot reason over, you know, um, natively over motion causality of that. So, uh, you know, uh, they do not aut- automatically build like a persistent structure, uh, on, you know, who appear, what happen, what changes, et cetera.

And so my argument is that video intelligence need a memory layer that decide what to traverse, how to connect it, and how to retrieve later. So, um, I wanna kind of ground it into the properties of video, right?

To, to make it even clearer. There's five challenges of dealing with video. Number one is temporal, right? So meaning depends on before and after. So a frame by itself can be misleading, right?

The same expression, product shot, physical action can mean different things depending on the sequence around, right? Second is that video is obviously multimodal. I, I explained already, um, you know, uh, a transcript alone may miss, you know, the logo.

A frame alone may miss the spoken claim. Video's-- video is also very dense, right? So a few minutes can contain dozens of shots, people, objects, action, location claims. The useful signal is uneven across the distribution on, on the frame.

Some second are decisive, others are noisy. Fourth is that, uh, video is also ambiguous, right? Um, people reappear under different lighting and angles. Brands are partially visible. Location are implied.

Concepts emerge over time rather than being named in a single moment. And lastly, it is, uh, expensive, uh, because in a lot of, uh, big enterprise and, and complex workflow, you need to, you know, point back to the source moment, like where it come from, right?

So these are the five properties explaining why video memory, um, is, is very complex. You need to preserve temporal span, multimodal evidence, continuity also. Uh, this is a very simple, uh, stack of how we build things at Twelve Labs.

Um, at, at the bottom, we have these semantic chunks that capture, you know, meaningful temporal units. Above, above that is our, uh, multimodal embedding coder called Marengo, which essentially turn those span into spatiotemporal relations.

Um, basically vector embeddings that represent video content. And then we have a spatial con- spatiotemporal context store, which is where it preserve pre- reasonable structure like moment, entities, metadata, all of that.

Uh, we also build our own, uh, VLM, video-contextual aware language model called Pegasus, that essentially serve as the, the reasoning layer, right? That can preserve-- prepare over video content.

So think about summaries like, um, metadata, synthesis, comparison. And we expose our, our models as API because, you know, we wanna get developers to use them as infrastructure.

Now, moving beyond, like, kind of the stack right here, I wanna talk about the difference between search and memory, right? Very quickly speaking, uh, search is obviously super important.

It's how you recover r- relevant moments from large video library. But then it give you a candidate. It actually not give you, like, any continuity. So memory, on the other hand, is, is all the, um, you know, uh, the things that enable the system to answer a different class of question, as y- you see here on the

right side o- of my screen. So these are not the single retrieval call, right? They require the system to preserve entities, timeline evidence across an entire corpus. Um, and so, like, you can actually build product moving beyond from, like, "Show me something like this," to, you know, "Tell me what this collection knows," right?

And so that, that might sound, you know, simple and subtle, but, uh, the, the, the, the output is completely different. Like, with search, you, you get, like, an output, like a time-bounded moment, but with memory, you actually return, like, structure knowledge, timeline, uh, explanation, a composable output.

And that, like, you know, is very important because we can now move, um, the, the unit output from clip retrieval to corpus memory, right? Um, there are two scaling dimensions shown here in the slide.

The first is time scaling. So a real video system should be able to reason over years of footage without reprocessing the whole archive every time, right? That means memory first retrieval, uh, be reusable re- representation once, and then support multi-hop timeline, episodic recall, follow-up question at lower latency and cost.

And then the second, uh, dimension is in space, right? So many real workflow actually, um, involve multiple perspective, like different camera angles, uh, you know, live stream, creator, broadcasting content, body cams, store cameras, uh, event feed, right?

So how can you build a system that can fuse evidence across all the sources and then maintain current understanding, right? And so that is the challenges here. How can you build a representation that let application traverse video across time and across sources?

Um, since this is, um, you know, a, a track on, on graph, right? So, uh, the, the best mental model that I can come up with is to represent, you know, video collection as a context graph.

So a context graph is a durable, queryable representation that connects video moment, entities, appearances, relationship, timestamp, metadata, and corpus-level context, right? So if you take a look here on, on the screen, all the way in the bottom, you got time-bounded moment.

These are, like, the s- the scene, the shot, right? Uh, these are evidence unit. One, one level up are the appearances, where and when each then entity show up.

And then you got the actual entity itself. So think about the people on the video, the brand, the places, the concept. Next, you have relationship, uh, co-occurrences to the same brand, sequences between different places, the causality and timeline.

And finally, at the top, you have corpus-level context. What are the main themes, the patterns, the gap, the coverage that this video collection cover, right? Uh, this matter because, uh, different question traverse different part of the graph.

If you ask a simple search question, then m- might-- that might go directly into the moment. But, like, an entity workflow might start with a person, and then it expand into appearances, right?

And if you ask question like a storyline, like narrative storytelling of certain, uh, you know, uh, you know, person, then it may follow a relationship across time, right? Uh, so the key idea here is that memory, in the context of video understanding, is a navigable structure over the entire video volume.

From that concept, I come up with these five principles of building, um, you know, a memory layer for video intelligence. Number one is to ingest once and reason many times.

So, um, you don't want to, like, do sing-- every single query, uh, from scratch. Like, you want to pay the cost up front, do one, uh, interpretation from the video content up, um, up front, pay the cost, and then you move expensive understanding into ingestion.

So this is same mental model, uh, of, of database, right? You do- you do not repeatedly have to parse your entire source of data, um, for, you know, every application request.

Um, second principle is to store primitive, not just answer. So, uh, you know, moments, entities, appearances, I already talked about that. Those are the, the primitives, right? That allows you to, uh, do downstream workflow, like search, editing, um, you know, analytics, all of that.

Third is to ground every claim. Like, basically, if you ask a question, you need to sign back into where that scene happening in the video. So, uh, evidence, like, you know, should be grounded to a specific timestamp within the video, right?

Uh, fourth is to let intent shape memory. Um, this is important because the same footage mean different thing in different workflow. We work across spots, uh, application, brand safety, compliance review, creator analytics.

All of them require different primitives from the same video, so the memory layer should be configurable, right? Developers should, uh, should be able to tell the system what matters.

And lastly, uh, keep the layer composable. Um, so basically being API first, you know, um, it should provide the, the layers that allows those application, uh, on top of that to, to serve it, uh, structure grounded metadata that can be plugged into any sort of application.

Um, so moving beyond these five principles, I want to talk about, like, kind of the, the harnesses around building a memory layer, right? There's a lot of talk these day about, um, you know, building the right harnesses for the context of language model.

So what does it look like for, for video understanding model, right? Um, a model call produce a single answer. It is stateless. It start fresh each time, uh, it start fresh each time.

It doesn't have any constraint. So the output is largely based on what the model decide to produce. A video worker, on the other hand, operate inside a, a very deterministic system, understand what is available.

Uh, it can plan the task, retrieve evidence, inspect, uh, uh, the relevant moments, synthesize, validate, return output, and then the entire workflow can be evaluated, right? Um, so, so for video understanding, this is very important because the worker need to know what memory is, is available, what evidence matters, and also, like, how deep to inspect, because that

will depend, uh, determine how much cost to spend, what output contract to satisfy, right? Um, talking about harness engineering for, for video understanding, um, I come up with these, like, different capabilities for, for, like, a video worker, right?

Um, number one is memory. I talked about that already. Number two is task planning. So given, given a query from, from an end user, uh, you have to decide, like, what task to execute.

Is it, like, search, or is it, like, summarization or, like, you know, multi- multi-step reasoning? Uh, third is retrieval. Like, every single system should be able to, like, select the right evidence from, from your video corpus to read for a specific task.

Uh- Expert tools, right? So we work with customer where they require like, you know, zoom in, zoom out, uh, comparing different, uh, uh, you know, frames, enriching, uh, co- content with like additional metadata.

So building expert tools inside, uh, like a, uh, like a video worker, uh, is very important. Operating envelope, so these are like explicit limit on time, cost, depth, scope, autonomy.

Uh, an output contract, so sometimes natural language is enough, sometimes the patient needs structured data with references and timestamps. And of course, fi- finally, we have evaluation, right? Uh, like, you know, did the retrieval find the right evidence?

Did the synthesis, synthesis preserve it, right? Did the worker stay within the budget? All right, so, so that's a lot of like, you know, uh, slide and, and, and talk.

I wanna quickly jump into some demos, uh, that I actually built using TwelveLabs', uh, you know, video agent product. So the, uh, there'll be three demos. Um, the, the video agent product that we've been building is called Jockey.

So this first example here is for sport understanding. Uh, you know, ob- obviously, everyone is super excited about the World Cup that happening right now. So what I did is I ingest, um, 67 videos from the 2022, uh, World Cup in Qatar, and you can see here I asked it to find the near misses, uh, the shot

that almost become goal but did not. For each, explain why it was not a goal, uh, but do not include the actual goals, right? So these are the, the, the top output that it return.

So that, um, that's hitting the woodwork. This is saved from the goalkeeper. I don't know if, if the sound is up, but like I'm playing the, the video, by the way.

Um... Right, this is another save from the goalkeeper.

It even catch like, you know, the offside, uh, from one of the goals.

And then ask question, "Okay, when is the goal? Uh, find the most dramatic actual goals. Show the build-up play and the finish. For each goal, describe the sequence," right?

So if you know this one, this is the,

um, the first goal of the World Cup final like four years ago. And it, it actually like returned like, you know... It, uh, understand who, who are the passer, like it was Alvarez passing to Mac Allister and passing to Di Maria to score the goal.

Um, take a look at this one from Richarlison. This is goal of the tournament, uh, from Brazil against, uh, South Korea, I believe, right?

And it returned like, you know, um, an outrageous skill in a build-up, right? Name the player who did the return pass.

You can even do player tracking, so I asked it to track Lionel Messi across this entire corpus, including the shot, where he's one of the manage figure on the screen.

Describe the camera framing, right? Uh, so this, uh, uh, highlight reel of all the important moment in the game, and this is a scene where Messi dribble past a sliding defender, you can see here.

It pick up the scene where he scored the first goal against Australia in the round of 16, I believe.

Right. This is another scene where he scored the third goal in, in the final.

Yeah, so that one example on spot, uh, spot understanding.

But then you can obviously build more interesting and more, uh, like real practical application of which, uh, security is one that we encounter a lot. So on a c- on this example, I ingest eight, um, you know, publicly available camera footage.

For context, these are the clip. You have, uh, traffic jam, suburban, uh, you know, urban a- aerial. Uh, and given this footage, right, I asked Jockey to count and classify every vehicle in the intersection, break it down by type plus pedestrian, and it return the number of vehicles and the peak foot traffic as well.

Um, it can detect safety events, right? So you see there, uh, a red SUV turn and almost get struck.

Another scene here, turn left into an upcoming car.

Yeah, so that a clear red light entry.

Uh, it works well in, you know, different scenario. This scene is a very crowded, uh, aerial in Bangkok.

Uh, it asked, I also asked it to work on the, uh, you know, the rain, right?

So this is another scene where it understanding the, um, rainy condition.

It identify the busiest, uh, intersection vehicle window.

So yeah, those are some example for, for camera security surveillance footage. Um, finally, advertising. So, um, you've probably seen this, uh, Adidas clip in all, all the commercial leading up to World Cup recently.

It's a five minutes Adidas footage, and I asked it to classify all the point where you can put an ad on. So it find the reveal, the hard cut, the impact, energy peak, right?

It find a scene where a certain player appear on the screen.

It identify like, you know, high impact action like this. Condition, um, the hard cut to Nike football underlines. And of course it, it point into the logo, um,

of Adidas. So, you know, uh, from, from perspective of an advertiser, these are very important moments because they can, you know, find a scene with slow motion hero.

Hard cut on a beat. Or peak action.

In which they can advertise their brand content against this footage, right? Um, yeah, so those are three sample demo application, um, that I wanna highlight, uh, of using TwelveLabs.

Um, and again, um, now what can you build with, with this sort of video memory layer based on the just example? These are the categories of, of application that I believe developers can build.

You can discover things. You can build reasoning experience. You can organize your content across different video library, and it can be action workflow, assemble, uh, different scene together, do compliance review, data operation, et cetera.

The same framework apply for different verticals, in media and entertainment and sport, segmentation, highlight generation, in commercial security, evidence review, contextual analysis, in advertising, uh, brand safety, uh, creative intelligence, right?

And, uh, yeah, so this is our product, uh, that are up, coming up. Um, one quick highlight is that we, we try to code as a video con-cognition infrastructure.

So we have a knowledge store that basically become the video memory layer, web configurable ingestion that let you to shape what the system can, can extract, corpus digest, so that you can understanding what is in the library, and a resolution agentic search responses API.

So the thing I wanna highlight here is it's not an application layer. Right? It's not an editing platform, not a compliance product. It's the cognition infrastructure with the layer and the harnesses that enable, like, those product being, to become available.

Um, and if you found the content of this talk interesting, um, definitely recommend you to, to scan this QR code. Uh, the product is currently in private beta right now.

Um, if you bring any sort of workflow that touch video content, especially around content assembly, content organization, you know, think about media archive, content creator, YouTube, TikTok, uh, uh, spot analysis, uh, media workflow, uh, definitely, uh, either scan this QR code and register for the interest or come talk to me after the talk.

So that should be my time. Thanks a lot. [audience cheering] [upbeat music]