State Space Models for Realtime Multimodal Intelligence

[on-hold music] Maybe to set the stage a little bit, um, the last four or five years of AI have basically been really focused on this idea of batch intelligence, which is sort of, um, pretty core to this, uh,

idea of, uh, building, like, an AI system that can reason, uh, for long periods of time on a problem and then, uh, solve it. So you can think about, like, math problems or, you know, physics problems that are hard.

Um, there's a lot of applications where actually what you need are systems that are streaming. So they're real-time, they, uh, work instantly. So imagine, um, generating video, audio, or, um, doing, like, understanding applications on, um, sensor streams, et cetera.

So, um, so it sort of bifurcates where there's these two different types of, uh, applications, similar to how there's, you know, generally this idea of having, uh, batch workloads and streaming workloads.

And so, um, a lot of what we've seen over the last few years has really been focused on, uh, batch APIs where you call a model in the cloud, it takes a few seconds, and then you get a, a pretty good response back.

Um, and now we're seeing some shift towards more real-time applications where you constantly will be querying a model and, um, h- asking it to return responses at low latency and then, uh, using that to sort of interpret, uh, or generate information.

Um, and I think this, you know, this area is really exciting because it's going to be transformative to a lot of interesting applications that, um, have so far actually not necessarily been, uh, the main focus for, uh, a lot of what we've seen, uh, over the last few years.

So conversational voice, uh, is an example of this where you should be able to interact with a system and then talk to it, um, a-and it should be able to understand you and, and do all kinds of, uh, tasks on your behalf.

Uh, this is similar to having assistants, um, that are on device and run kind of really efficiently at low power, um, at, you know, all times, um, regardless of whether you're on a phone or, or a laptop.

And then, um, things like world generation where, like, you can imagine actually playing a game that is generated in real time similar to, um, how the graphics are rendered, um, on GPUs.

Um, and, um, all of this, you know, should be able to happen in real time, on low power, on your phone, uh, on your MacBook, et cetera. Um, robotics is another great example where it sort of culminates with all of these coming together on a, on a, a single, um, uh, device that is, uh, trying to kind

of interpret everything in the world. And so I think this is sort of the ex-exciting intersection, which is like, how do we make intelligence faster and cheaper so that we can put it, um, everywhere, basically.

Um, and a couple examples that are really powerful. Real-time intelligence for conversational interfaces, uh, is gonna be really interesting because you would be able to have a agent that can provide customer support for a problem, answer questions about health insurance, uh, you know, call your vendor to pick up a shipment.

A-all these coordination tasks that, uh, generally are, um, annoying to do should be really automated and, uh, real-time intelligent agents should be doing them. Um, and then humans can spend their time solving sort of harder problems that are, uh, more interesting and, uh, in customer support, that could be dealing with, uh, you know, the tail customers that

are, uh, much more important because they're pissed off or they're, uh, uh, uh, more important because they have, uh, you know, more customer value, et cetera. And similarly in robotics, there's this idea of, like, ingesting similar to humans, like audio, video, sensor data, and then responding instantly to a lot of these, um, uh, pieces of information.

So I think this is sort of the, the world we, we should be living in where all of these intelligent models run super fast, they solve all these different problems and, uh, they're able to really kind of, um, power these new experiences that are interactive at their core.

So this is where we come in. We're building, uh, these real-time foundation models. So, um, some of what I'll talk about is, uh, the work we've done in really building kind of new ideas for how you can create, uh, deep learning models.

So, um, I did my PhD before this. I was working with a lot of these folks for my PhD. Chris was our PhD advisor. Um, and, um, we were really focused on this idea that you should be able to have a model that can compress information as it comes into the model and, um, use that to really

kind of build powerful systems that are streaming at their core. Um, and I'll talk a little bit about this, but that's really the technology that we've been working with for the last four or five years.

We've been developing in academia, and some of you might have heard of things like Mamba, which is sort of a, a more recent iteration of this technology. You know, we-- I did my PhD working on some of the early iterations that nobody uses anymore, but, um, are sort of the precursors to a lot of the modern stuff

that is now, uh, more widely used. And, uh, and now what we're doing at Cartesia is basically taking this and trying to understand how we can improve it. Mmm, how do we push the boundaries on, uh, what architectures can do?

And, um, and I think it's an interesting question because, you know, we should not settle for having a, uh, one way of doing things. I think that's sort of a, a poor way to kind of, um, think about the future.

So our approach is sort of like, let's think about new ways of actually designing models that aren't necessarily built on, let's say, the transformer architecture and the standard recipe for, uh, deep learning that's, uh, you know, prevalent today.

And I think it boils down to this question of, like, efficiently modeling long context is a huge problem because, you know, a lot of practical data is really long sequence data.

I think text is maybe the least interesting, uh, long sequence data because text is actually fairly, uh, compressed already, right? Like, you have a lot of information that is, uh, embedded in two minutes of, uh, or, uh, two sentences of text.

But there's all these other domains where, you know, audio, video, et cetera, where there's so much information. Um, you know, imagine looking at a security camera for a day.

Like, you would probably have just so much information coming into the system and just, um, very little of that would be useful. So compression is kind of really fundamental to intelligence because we're able to do this where we can look at all this information and then sort of compress it down to whatever's necessary to remember or understand.

Um, and I think so far what we've seen is that the AI systems that we built-

Have not necessarily exhibited that same behavior. So they're really kind of built not on the principles of compression, but more on this idea of retrieval, like keeping all the context around and then using it to reason over all the information that you've seen.

So I think, um, our kind of point of view is that multimodal AI will remain challenging as, as long as you're sort of, uh, working in this paradigm. Because, uh, if you try to think about what humans do in a year, um, you're basically processing understanding about a billion text tokens, ten billion audio tokens.

These are, you know, back of the envelope calculations that I did. And about a trillion video tokens, probably underestimates how much video we process, and not including all the other sensory information that you're processing, and you're doing it simultaneously, and you're doing it on a computer that fits in your brain.

Um, and you, you know, sometimes don't eat and drink and, you know, you're still functioning fine, so you know, you can have variable amounts of power in the system.

Um, so I think the idea that like intelligence is solved is sort of very far from the truth because humans just are an extremely amazing machine that, uh, does something very extraordinary, um, in a very compressed way that, um, our AI models can't do.

So I think that's sort of our, uh, you know, sort of, uh, the reason we get up in the morning is we think about this and we're like, yeah, we're very far away from where we should be.

Um, and the best models today are in the, you know, ten million, a hundred million sort of token range. So, uh, that's really good. A lot of progress has been made, but really this is sort of what we aspire to is how do you kind of build these machines that are long-lived that can actually understand information over

very long periods of time? And I think the cool thing is like, as a human, you can remember things that happened thirty years ago with very little effort. You don't need to do RAG or retrieval or anything.

You just, you know, you remember it. It's just it in your brain, and then you figure it out, basically. So I think that's kind of an extraordinary capability that we should be able to put into our, uh, AI models as well.

And so, uh, some of the big problems with models today are, uh, you know, they're built on transformers really optimized for data center. I think, um, we see this with like a lot of the work we did, which was on subquadratic models.

So quadratic scaling and context length really just means that, um, you know, the amount of, uh, computation you have to do to process long amounts of, uh, context is very large.

And so right now, the sort of predominant approach is to throw compute at that problem and then hope that that would scale. Um, obviously, compute is a very important piece of the puzzle because you do need more computation to be able to do more difficult things.

But, um, this type of approach, because the quadratic scaling actually has poor scaling with, you know, very large multimodal context. And text context tend to be shorter. Multimodal context will get larger because you have just way more tokens and information that's going into the system.

So that's gonna be a big challenge for these models, especially how do you do this inference efficiently so you're not, you know, burning down the data centers to, you know, do, uh, a fairly limited amount of inference.

Like, you have to imagine that we're doing a thousand times or, um, you know, a hundred thousand times more inference. And then, um, if the models are scaling the same way, it's gonna be really, really, really expensive.

So you're not gonna be able to permeate all these applications that I talked about very easily. Um, and so, you know, that's sort of a big challenge, I would say.

A-and so, you know, again, our hypothesis is you need new architectures, and that's kind of where we spend our time, and we wanna make these models more efficient, faster, more capable, while being able to handle all these long context problems.

Um, this is a slide about, you know, transformers being somewhat inefficient at handling this, um, but obviously, um, a very good recipe for scaling, uh, uh, these models out.

And so, you know, some of the work that we've been doing is new, fundamentally efficient architectures. So they have compression at their core. So they sort of-- The way they operate, uh, I'll, I have a slide on this just to give you kind of a, a, a quick illustration, but, um, they really scale more linearly in context

length. So you should be able to have, uh, because of this, like, more low power implementations of these models, um, you can compress information as it comes into the system, you have low memory usage, um, and you can actually scale to much more massive context because of that.

Um, and this is all the work around SSMs. I just threw this, uh, nice slide, which I thought was cool. Um, Jensen had an interesting quote about SSMs in one of his Wired articles that I like to keep talking about.

But, uh, uh, but I think it's a cool technology that has a lot of potential and sort of that's where we're, um, spending a lot of our time. And if you folks are interested in reading more, there's lots of videos on YouTube and lots of sort of resources that try to make this more accessible to understand and,

and kind of get into some of the details. Um, but, you know, the working intuition is basically, uh, transformers are generating quadratically by attending to every past token of information.

So as tokens come into the system, you're kind-- sort of keeping them around and then looking at all the past tokens. So if you wanna generate the word jumped, uh, from the quick brown fox, you would actually look at the entire context, try to understand what the next word should be, and then, um, generate it, push it

into the context, do it again. Uh, with SSMs, you just have a streaming system. So you, you have a to-- uh, token stream in, uh, they update an internal memory for the model, and then the token gets thrown away.

So that actually really simplifies the system, and that's why it's such a core sort of streaming interface, because you're just not keeping all this memory around about what happened in the past.

You're compressing it into some sort of zipped file, uh, state inside the model that's gonna be used to do a future generation.

And so this is sort of taking this idea of, uh, taking advantage of this idea of recurrence, which is sort of core to how even humans, uh, do a lot of their reasoning.

And, you know, last few months, a lot of these models have been getting adopted. So it's great that, you know, a lot of folks are now excited about the, uh, this, you know, alternate way of doing things that is much more, uh, sort of are oriented around this idea of recurrence rather than, um, retrieval.

Um, and so I think, like we'll see a lot more activity in this, especially with multimodal data becoming more important. And, uh, you know, a lot of the challenges of multimodal data around efficiency will mean that I think that these models will have more of a role to play in the next three to five years, um, as

we also do our work in, uh, scaling them up and making them more interesting. A lot of people ask me about quality. Uh, I only have a few minutes, so I'll go through this, uh, the rest of the slide super fast.

But Um, you know, SSMs generally have the right, uh, quality. Obviously, there's a trade-off between compression and keeping all of the information around. Uh, but actually, like, compression can be helpful.

So if you imagine the security camera example, if you're watching twenty-four hours of footage, actually compressing all of that information on the fly would help you solve tasks and a- answer questions better, uh, rather than looking at all twenty-four hours every time.

So I think that's sort of the rule of thumb to think about, which is compression's super helpful for a large context, uh, not as helpful for, um, short context.

And so we see that quality actually is very good for long context problems and, and multimodal problems. Let me talk quickly about some of the work we've been doing.

So we've been starting to work on sort of multimodal data, and we did a release a few weeks ago for a, uh, voice generation model. So this is sort of text-to-speech and sort of in line with some of the work we're doing to bring more multimodal, uh, data into a single model, um, and use SSMs to power

the inference and the training and, and so on. So this is a model you can actually play with. I'll try to show you a demo. But, um, one of the things we're proudest of with this model is that we really shrunk the latency down, so when you play with it on the playground, you get instant voice back

generated from the data center. And there's some cool work we're doing to actually run these models on Mac, um, and other devices so that you can basically have the same experience as you have in the data center, but on any device and, and do that efficiently and at low power.

How much time do I have? You have four seconds. Okay. We're, we're out of time, but I was also almost done. So, um, go to the website play.cartesia.ai. I unfortunately couldn't walk through the demo, but, um, play with it and, um, send us feedback.

This is my email in case you wanna, uh, send me a note. Uh, would love to hear feedback and anything that you folks, uh, find interesting. Thank you. [clapping] [upbeat music]