Neil Zeghidour - Voice AI: when is the "Her" moment?

[upbeat music] Hi, everyone.

Uh, thanks for being here today. Uh, really happy to, to talk. I think it's the, it's the right time for this talk because we had a lot of great presentations around voice, uh, this morning.

And I want to take a bit, uh, you know, uh, time to reflect where we are in the, uh, modestly where we are right now in the, in the voice, uh, uh, AI, uh, domain, and what is left to do as main challenges.

Uh, quick intro, uh, about Gradient. So we are-- Our mission is to unlock the unrealized potential of voice AI. So basically, we train voice models, uh, speech-to-text, uh, text-to-speech, uh, speech-to-speech, whether it's transformation, translation, uh, speech-to-speech dialogue, and so on and so forth.

We want to be, um, main, uh, model provider, uh, for voice, uh, for everyone building voice agents and voice, uh, solutions. So we are not working on orchestration, we are not working on specific verticals.

We just make, uh, building blocks for people who want to build, uh, voice AI. Um, here, I mean, I can let it be said, uh, uh, by a famous podcaster that I cloned, uh, this morning.

Uh. There should be audio that is output.

Man, have you seen what Gradium is doing with voice cloning? It's kinda crazy. Like, seriously.

Can we put a bit, uh-

You record like ten seconds of your voice. That's it.

Can we have much more volume, please?

Man, have you seen what Gradium is doing with voice cloning? It's kinda crazy. Like, seriously. You record like ten seconds of your voice. That's it. And the system analyzes the tone, the pitch, the accent, all those little quirks that make your voice yours.

Then boom, you type text and it talks back-

Okay, I guess you recognize maybe Joe again. I hope you did. Uh, so basically this is a spinoff from a nonprofit lab we created two years ago, uh, with a funding, uh, uh, from philanthropists, including Eric Schmidt, uh, Rodolphe Se- Rodolphe Seid and Xavier Niel.

The main idea was to create a lab that does, uh, open research. Uh, and, uh, we focused mostly on speech. So we developed Moshi, uh, which was the first, uh, s- uh, speech-to-speech model for conversation, uh, speech-to-speech translation.

Pocket TTS, most recently a CPU model. Uh, and we decided to also create this for-profit structure to, uh, make, uh, products that can be used in production beyond open source.

So the, the goal of the talk is, uh, uh, basically based on the Her movie. So, uh, this has been the most overused, most annoying, uh, analogy, I, I think, in the field.

At the same time, it's extremely relevant, uh, because it's certainly old, and if you look at one of the introduction scenes, so that's when the main character, uh, meets, uh, his AI voice, Samantha, for the first time.

It still sounds like, uh, it was, you know, anticipating, I think, well, what interaction could look like. Oh, what, what do I call you? Do you have a name or...?

Um, yes. Samantha.

Where, where'd you get that name from?

I gave it to myself, actually.

How come?

'Cause I like the sound of it. Samantha.

And, you know, then came the, like, the, the trend of the Her moment. Uh, and we got so many Her moments on, uh, Twitter and in real life. Again, I don't want to be mean to anyone, so I will also make fun of myself.

It's more like, you know, trying to be pragmatic about what was the promise and where we are right now. So this is a very, very recent, uh, demo from, for me, what is the best voice AI company in the world, which is ElevenLabs.

Hello, you're speaking to the government's AI helper. How can I help you today?

I would like to start a new business.

To start a new business, you'll need to choose a business structure and name, complete the registration, and upload required documents.

And this one is a demo that I did, uh, uh, a few months ago with a Richie Mini. So our presenter this morning. Uh, you know, but it, it was very risky, I recognize.

I was very scared when I did it. But, uh, basically putting our, uh, streaming voice models into a, into a Richie Mini.

Action, just say the word.

Okay. Maybe you can do something a bit fun. Um, I want to get, um, you know, to improve my health overall, and so I, I'm looking for a, for a bro to go to the gym.

Can you take the personality and the voice of a gym addict?

Hey, I'm Logan, your gym bro. Let's crush those gains together. You ready to lift, sweat, and feel awesome? I've got your back. No excuses, just results. [laughing]

So, okay, this it's, it's fine. It, it, it sounds more natural than it used to. But in both cases, you know, I mean, we are still not there, right?

Uh, the latency is still quite high. Uh, the ability to handle simultaneous speaking between the user and the system is not there. Uh, intelligence starts to become much better, and I think that's also why we see all this traction around voice agents, because there are agents actually to whom we can give a voice, and they can be

useful. And, um, also this is mostly just a glorified text model with a voice around it. And, uh, so, you know, anything that is not in the text will not be able to be leveraged.

And maybe I could ask one last time to increase a bit the volume. I think that could be even better. Um, and so what does it take to get there?

So basically, we have very nice presentation this morning by Samuel about cascaded systems, so I will go quickly. Speech-to-text, LLM text-to-speech, that's the classic cascade. Uh, in our case, we do, um, uh, streaming speech-to-text, uh, streaming text-to-speech with voice cloning, uh, semantic VAD, the classic stuff.

Um- So latency, you know, getting to a fast, uh, conversation. So we have a very fast TTS, so that's the latency for our TTS compared to a few, uh, other models.

But, you know, that means that just the TTS is still more than two hundred milliseconds, while in a human conversation you need the entire stack of understanding, producing an answer and pronouncing it to be around two hundred milliseconds.

So that will none, none of this will allow to have a conversation that sounds, uh, that sound human. And this is just latency for text conversation. There is no tool call, no actual task that is performed.

Now that's another scene from the- from the movie, uh-

Okay, let's start with your emails. You have several thousand emails regarding LA Weekly, but it looks like you haven't worked there in many years.

Oh, yeah. Okay. So here it just went instantly into through all the emails and gathered information and so on and so forth. Obviously, we are not there at all.

Today, if you have a, a voice agent that is supposed to use a tool, uh, you're going to wait. Uh, the tool call, you know, we are fighting for latency of the TTS, trying to grab ten milliseconds, twenty milliseconds, and then you have a tool call or open router that is going to have a latency between five

hundred milliseconds and four seconds. So in a way, sometimes you feel like we're fighting for something that is not the most relevant anymore. I think now the main bottleneck is becoming the tool call, which is very unpredictable, and we should have models that are resilient to very complex tool calls.

So what we are, uh, you know, one solution to do that is to have fillers. So basically, you know, your LLM, it splits, uh, into two things. So it sends, uh, a tool call and while it waits for getting the result back, it can keep the conversation going in a natural way, and then it retrieves the result

and it tries to insert it back, uh, naturally in the conversation. So just because I always do live demos, I will do a very short, uh, improvised live demo with something that was vibe coded very quickly, so it may be go very wrong.

But the idea here is to have like a just a vibe coded, uh, travel agent and, uh, that can, when I ask it to retrieve, uh, places, it's going to try to find a nice thing to say about the location I'm going to.

Hello. This is Colin from Wanderlust Travel. I see you're looking to book a trip for two people from April tenth to the thirteenth, twenty twenty-six. Where would you like to travel to for your getaway?

Um, I want to go to Tokyo.

Tokyo is such an incredible choice. It's a fascinating mix of ultra-modern skyscrapers and beautiful, peaceful shrines.

I found some fantastic options for you. You could stay at the Fairmont for a serene-

So this needs polishing. The main idea, you know, is that while it was trying to gather things, it doesn't really know how long it will take to get the data.

It tries to say a nice thing about, uh- about the place you're going to. So that's one way to get latency that is going to be more controlled, uh, uh, and more reliable even despite the complexity of the thing.

But then we know that latency w-- of cascaded system is inherently high, right? And so then what I hear a lot from customers, from investors and so on is what about speech-to-speech?

So I think a big confusion is what is speech-to-speech. So speech-to-speech is the idea that now instead of having the three blocks, you only have one that does everything together, right?

So instead of having audios that go in speech-to-text and then LLM text-to-speech, you have a model that takes speech as input and is going to output speech. And that reduces latency a lot, but that's still not a human conversation.

In particular, because every single speech-to-speech model except Moshi is half-duplex. What that means is even the best, uh, speech-to-speech model we could argue, I don't know, maybe that's the advanced voice model for OpenAI or Sesame.

I'm a huge fan of the Sesame, uh, uh, voice, uh, uh, voice model. It's still half-duplex. What it means is that the model is either listening or it's speaking, and it cannot handle the ambiguity of human conversation where you can have overlap between, uh, people speaking on one another.

A lot of things happening. You cough, uh, you know, you just do, "Mm-hmm," which is called backchanneling, and then it breaks completely. Full duplex, that's a human conversation. It varies between cultures and languages.

But for example, in Japanese, it's a sign of politeness and that you are actively listening to do a lot of backchanneling. So you say, "Mm-hmm, uh, uh, uh," constantly when the other person is speaking, and you get up to twenty percent of the time that is overlapped between the people, right?

So that's what makes a conversation human. And so I showed this video in my hotel room this morning just to sh-- you know. That's a speech-to-speech model, but that's not a full duplex model.

And you see how it can, it can become annoying. Hey, how's it going?

Hey, I'm doing really well. Thanks for asking. How about you? How's your day going so far?

Uh, I'm great. Um, you know, I'm preparing a talk about voice AI and how far we are from, um, uh, you know, um, from the Her movie. And, uh, yeah, I'd just like to brainstorm a bit about it with you.

Oh, that's a great topic. Yeah-

Mm-hmm

... I'd love to help you brainstorm. Are you thinking more-

Yeah.

Exactly. Yeah. I was thinking maybe-

No, no, I didn't mean to interrupt. You know, I was just saying, yeah, like that. You can just keep going. Uh, you know, don't mind me. That's something I typically do.

Ah, no worries at all. Yeah. I was just gonna say we could break it down into a few aspects.

Sure.

Like... Yeah, exactly.

No, no, please stop, stop interrupting. You know, it's called backchanneling. Uh, humans do it all the time. It shows that you're just following the conversation. That shouldn't, you know, like interrupt you in your flow.

Just, just keep going.

Ah, got it. Thanks for letting me know.

No problem.

Yeah. No problem.

Oh, come on. [laughs] Okay. So yeah, basic-- I was a bit mean, right? But, you know, that's my point. It's not, it's not an actual conversation, right? And it can become very annoying in particular.

That's also why, you know, most of the voice AI demos, they are shot in empty, like a quiet room next to the phone and so on. A lot of things can break.

So what we did instead, uh, in our case was, uh, sorry, where am I in my presentation? I'm right here with Moshi. Was, uh, the first full duplex system.

So here you'll see my, uh, co-founder Alex talking to it. Uh, it's almost two-year-old now. I think it's still kind of, uh, aged well because what you'll see is, um, you know, they are going to talk on one another constantly and- It's just fine.

So the planet is Sirius 22. Can you plot a trajectory course to it, please?

Yes, sir.

Okay. How long is it gonna take us to get there?

I've mapped it out.

Okay.

It's approximately five months to get there.

Okay, that's, that's not too bad. Uh, do you think we have all we need on board the ship to start the mission?

Yes, sir. We have everything we need.

Okay.

So, you know, even when the model has guessed what you're going to say, it starts answering before you're done. At the same time, you can talk over it, and it's not ignoring what you're saying.

It's really, you know, s- consider it afterwards and so on. So you, you have what is the most robust, uh, to this day, uh, conversational experience, robust to noise, to a lot of people speaking, and so on and so forth.

But now if we compare it to, um, uh, to the Her movie-

So do you know what I'm thinking right now? Well, I take it from your tone that you're challenging me. Maybe because you're curious how I work? Do you wanna know how I work?

Yeah, actually.

So maybe this was not very clear, but this snippet here, it's the AI understanding that the character is a bit uncomfortable. So that's paralinguistic understanding. You know, it's understanding all the cues that are come from the way people speak.

Technically, that is in Moshi, that is in any speech-to-speech model because this information is not lost. However, if you don't exploit this information, uh, to make your model say relevant things, it's never going to exploit it.

If you train it on the audio version of a instruct dataset, uh, and it's just factual question answering, wh- why would it even try to capture this information? So basically, Moshi, I think we, we saw still the f- only full duplex model.

Uh, recently NVIDIA published a Personal Plex model based on it. Um, what was great is still the, the flow of it is just honestly impossible to match, I guess.

Um, it's conversational and, uh, and very robust. At the same time, the model was very stupid. So basically it was, you know, just useless. You could talk for, for a few minutes, and then it was a bit pointless, you know.

Because it was not an agent, it had no tool call, no ability to do anything. Uh,

it's impossible to use in production something that is, has no observability. You don't know if the... It's very hard to detect if someone said something that should be not accepted and so on and so forth.

And there was no real, uh, paralinguistic, uh, understanding. So

the main takeaway for me is, uh, we know that this nature of interactive model that are going to be full duplex, that are going to be, um, really the, that will be the way to get to

an interaction that is as natural as you would have with a human. But as long as we don't, are ab- we are not able to give to this kind of very natural sounding, uh, models the same level of reliability, intelligence, and personalization as cascaded systems, I don't see a path towards, uh, uh, them, you know, replacing cascaded

systems. So I used to be really at war against cascaded systems. I think they are so practical and so convenient, that the main challenge, honestly, I think this, we solved that with, uh, Moshi.

Uh, anybody who implements it and train it on better data will have something that is, sounds just indistinguishable from humans. But all of this is going, all the challenge is going to happen here.

Uh, the last point is the scalability. So now let's say you have the best, uh, speech-to-speech model, okay? Uh, it solves ev- all the stuff that I've talked to, talked about.

If we take again the analogy from the movie, um, you will talk to it maybe several hours a day, or it will always be on because, you know, it's on your computer when you work and you asynchronously ask things to it and so on and so forth.

Um, I'm not going to mention the cost of the API of our competitors, but voice is very expensive. Everybody in this room probably knows it. Uh, the voice mode of most hyperscalers is run at a loss.

It's a gigantic multimodal model, and they lose money every time they, they, you use it. But it's, you know, it's kind of a marketing thing, and so on. It's fine.

But now if we want to make it an actual profitable product that people are using at massive scale, just not going to work. And, uh, in particular, anyone who tried developing a consumer app with voice, uh, realize that LLM now is almost nothing in terms of cost.

All the bill is TTS. Speech-to-text is very cheap as well. Uh, diarization is affordable. Uh, TTS is really what is going to consume most of the... And we, I saw people burning their fundraising in TTS bills, and they don't even get the opportunity to get their user base to grow.

So another aspect is privacy. The more you're going to open to your AIs, the more, um, you know, you'll want it to be more controlled and private and feel more comfortable that things are not shared publicly.

In particular, uh, we see now people with Mythos being afraid that any, any single database was going to be hacked in a few months. And so, you know, you will be more comfortable if all your private data is, uh, is local.

And so to solve that, our first step is, uh, Gradium Phonon. Uh, it's a on-device TTS. So on-device means a lot of things. For some people, on-device means it runs on a gamer GPU.

For us, on-device means it runs on the smartphone CPU. And so it's a very small model, uh, less than one million parameter. And for its size, it works quite well.

It's better than all the existing, uh, uh, on-device models. CoCo is, is a good one, but it doesn't have voice cloning. And I'm out of time, so I would just play, uh, a short, uh, demo if I can.

Yes.

Uh, geez, Morty, stop looking for a signal. Gradium Phonon runs locally on the CPU, which means high fidelity neural speech right here without those intergalactic cloud government hacks.

It is about simplicity with no servers and no waiting, just a smooth, quiet performance that stays on the device with local processing and total privacy.

What else?

Hmm, local processing. That sounds like the machine is making its own donut. Wait, if the CPU is doing all the work, can it sprinkle some-

Yeah, so this runs on a smartphone CPU, which means that you can use that to power, you know, any kind of voice application without paying a single, uh, cent of, uh, API, uh, fee.

So we opened the private beta for this, uh, model. Uh, the goal for us is to allow people to create consumer apps with voice, basically. And that they are able to scale the usage, uh, without having to lose money on the, on API.

So the conclusion, uh, the path forward is, uh, for us, I have a str- strict opposition to some of our competitors say that voice is a commodity now. I think it's completely false.

Uh, voice is very challenging. The last mile is going to be the most difficult, uh, to solve. And, uh, for us, it's really about science, uh, and engineering. Uh, and it will get Her to us, to the Her movie, sorry.

So you can use us on gradium.ai, and if you want to join us, uh, to bring Her to life, yes. No, so I, I [laughs] now I'm using this analogy.

No, that's if you want to, to, to work on very exciting, uh, voice models, you can apply and, and, uh, and join us. Thanks a lot for your attention. [clapping] [upbeat music]