Your Voice Agent is Just a Walkie Talkie — Neil Zeghidour, Gradium
Read the talk
Why a Fast Voice Agent Can Still Feel Like a Walkie-Talkie
Neil Zeghidour explains how audio tokens and two simultaneous conversation streams change voice interaction—and why he favors separating a natural speech interface from the model that reasons and calls tools.
From a talk by Neil Zeghidour
At a glance
Ideas worth remembering
Open-ended dialogue, external action and natural conversational timing are distinct capabilities. The historical examples improve them at different stages rather than advancing all three together.
Speech-to-speech removes the explicit text cascade, but full duplex additionally requires representing simultaneous contributions. Low latency alone cannot make backchanneling work naturally.
Audio tokenization and multi-stream modeling solve different representation problems: codecs compress waveform sequences, while two streams allow speaking, silence and overlap on both sides.
Full duplex enables continuous interaction but does not guarantee polite timing or strong reasoning. Zeghidour acknowledges frequent interruptions and weaker intelligence in his early models.
The proposed hybrid architecture keeps a small full-duplex interface responsible for conversation and delegates reasoning and tools to a text backend. Its appeal is lower conversational cost and backend flexibility; reliable recognition of when to delegate remains an essential requirement.
Building models around voice
Neil Zeghidour introduces Gradium as a Paris company with a research background that trains audio foundation models for voice agents and applications. The approach is to build a foundation that supports several tasks, including text-to-speech, on-device speech-to-text and speech-to-speech translation. He describes earlier research work on full-duplex conversation, real-time translation and speech synthesis that can run locally on a smartphone.
The applications range from customer agents and game characters to language learning and coaching. To explain where these interfaces might go, Zeghidour starts with their history. His first example is the 2011 announcement of Siri: an early voice assistant whose constrained interactions already demonstrated useful action through speech.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a transcript to an application action
The Siri clip shows requests for today's weather and the NASDAQ's performance. The assistant returns a forecast and a market reading, with the presenter emphasizing access from the lock screen. These examples establish an important capability: a voice interface can connect a spoken request to a specific application and return information from it.
Zeghidour calls this a voice agent, although a constrained one. Its architecture used speech recognition to obtain a transcript, then natural-language understanding to classify what the user wanted. That classification identified the application to control and the action to trigger. There was no LLM handling the dialogue. The system depended on a complex pipeline built around specific supported requests, which he describes as closed-ended dialogue.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Opening the dialogue without adding agency
The next generation is illustrated by OpenAI's original voice mode. An LLM replaces the earlier dialogue logic, allowing conversation about a much wider range of subjects. In the embedded example, a user requests a bedtime story about Larry, a hedgehog with sunflower petals instead of spines. The system develops that premise into a story, demonstrating how an LLM can respond to an imaginative request outside a predefined application command.
Conversational breadth and agency are separate capabilities in this comparison. Zeghidour says the original voice mode lacked the ability to retrieve weather or market information as Siri did. It made dialogue more open and speech more natural, but did not supply the actions needed to accomplish those tasks. It also retained several seconds of latency. Replacing dialogue rules with an LLM therefore improved what the system could discuss without resolving either external action or response delay.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Restoring action through a tool-using agent
A drive-through demonstration introduces the next step. The customer asks for a chicken sandwich, chooses the classic option, asks what else is available and adds mac and cheese. The agent retains the selected items and reports a total. Zeghidour explains that this system takes actions and tracks an order, with payment intended to follow. The exchange demonstrates order tracking; the shown conversation does not itself complete a payment.
Here the LLM participates in an agent with tool calling, reasoning and planning. Zeghidour argues that this restores agency in a more powerful and general form than the earlier assistant offered, while preserving natural dialogue. The system can maintain a task across several utterances rather than merely generate an isolated conversational answer.
The remaining weaknesses come from the cascade through speech recognition, text reasoning and speech synthesis. Zeghidour says latency has improved but remains above human conversational timing. Text also discards information carried by the voice, including tone and emotion. Speech-to-speech becomes the next architectural step because it can address the delay and the loss of expressive information together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A single speech model still has limitations
The advanced voice demonstration features a person asking for help with nerves during a live presentation. Zeghidour uses it to illustrate an architecture in which speech-to-text, the LLM and text-to-speech have been absorbed into a single model. Processing speech directly allows the model to understand nonlinguistic information as well as words. He regards the resulting latency as already good enough that further reductions would offer little value.
That improvement does not establish equivalent reasoning ability. Zeghidour says speech-to-speech models remain substantially less intelligent than their cascaded counterparts and the text models behind them. This is his qualitative assessment rather than a benchmark presented in the talk, but it sets up the central design problem: improving the speech experience while retaining the capabilities needed for useful work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why acknowledgments break turn-taking
Speech-to-speech and full duplex describe different properties. A speech-to-speech model can still divide a conversation into periods when it speaks and periods when it listens. Zeghidour demonstrates the consequence through backchanneling: the brief acknowledgments people make while someone else continues talking. Such sounds signal attention without necessarily asking for the conversational floor.
In the demonstration, the model begins a brainstorming response, then stops when the user acknowledges it. The user explains that the acknowledgment was not an interruption and asks the model to continue. The pattern repeats. Even an explanation of backchanneling does not produce a smooth exchange: each small contribution becomes another conversational event that breaks the response.
This is the walkie-talkie limitation: a half-duplex interaction alternates listening and speaking. Faster responses do not remove that structural restriction. Human conversation allows acknowledgments, interruptions and overlapping speech within a continuous exchange. Zeghidour says that calls with a relative can involve both people speaking for up to 20% of the time. The talk supplies no measurement behind that quantity, but uses it to emphasize that overlap can be a normal part of comfortable conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compressing audio into a sequence a model can learn
Zeghidour explains speech modeling by starting with the prediction task of a text LLM. Instead of predicting the next word from earlier words, the model should predict subsequent audio from earlier audio. The immediate obstacle is representation. A waveform describes air-pressure variations at a much finer temporal resolution than a sequence of words.
His example is an eight-word sentence that takes around three seconds to pronounce. At 24 kHz, that produces 72,000 waveform timesteps: 3 × 24,000. Under the quadratic sequence-cost assumption he describes, increasing sequence length by a factor of 10,000 increases the corresponding cost by 100 million. That factor is a rounded illustration rather than the exact ratio of 72,000 to eight. The calculation explains why feeding individual waveform samples into this kind of language model would be prohibitively expensive.
Neural codecs, also called audio tokenizers here, provide the compression step. An encoder converts audio into a dense, compact representation, and a decoder reconstructs high-quality audio from it. The language model then learns over that abstract representation in a manner analogous to text training. The talk does not specify a token rate or compression ratio, so the mechanism establishes how the sequence becomes manageable without quantifying the resulting savings.
In the sequential speech-to-speech architecture he describes, audio tokens represent contributions from the user and the model. Those contributions are placed one after another, and the model predicts its next audio tokens using context from both sides. Audio tokenization changes the content being modeled, but this arrangement still represents the conversation as successive contributions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Modeling two simultaneous streams
Full duplex requires another change: a multi-stream language model. Instead of modeling a single token sequence that alternates between participants, the transformer models two streams. Both participants can be active, both can be inactive, or one can speak while the other remains silent. This representation accommodates overlapping speech directly. Zeghidour also discusses other providers' use of related approaches, but explicitly treats the architecture of an OpenAI model as speculation.
The demonstration uses a fictional spacecraft conversation: a person asks about plotting a course, travel time and supplies for the mission. Zeghidour draws attention to how the model anticipates question endings and answers over the speaker without breaking the exchange. These responses illustrate conversational timing; they do not establish that the system actually computes trajectories or verifies mission readiness.
The ability to overlap introduces its own behavioral tradeoff. Zeghidour acknowledges that the early model interrupted so often that it could irritate people. He nevertheless reports that conversation continued through noise, coughing and other disturbances without the repeated flow breaks of turn-taking. Supporting simultaneous speech makes the interaction resilient to overlap, but does not by itself teach the model when speaking over someone is appropriate.
Zeghidour presents full duplex as the strongest route to human-like conversational dynamics, while candidly describing his earlier models as less intelligent even than other speech-to-speech systems. Casual conversation may tolerate that limitation. An agent expected to perform real tasks cannot simply exchange reasoning ability for a more natural voice interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The capacity tradeoff and the path of scaling
Zeghidour explains the tension between naturalness and intelligence in terms of model capacity: a model has a finite number of weights. A text model adapted to speech must also learn to understand and produce audio. In his account, those additional responsibilities consume capacity that would otherwise support intelligence. Cascaded agents can draw on a capable text model without asking that same model to perform the entire speech task. The explanation offers a rationale for his observed tradeoff, rather than a quantified law of how much reasoning each modality costs.
He proposes two paths that can develop concurrently. The first is to make speech-to-speech models larger and improve both pretraining and post-training until their intelligence is sufficient for more use cases. The second is to separate natural conversation from intelligence. Scaling preserves a unified model, while separation assigns those responsibilities to different models.
Zeghidour guesses that OpenAI follows the first path, but says he does not know its internal model design. He describes the general approach as adapting a frontier text LLM to audio through post-training and instruction tuning. Its operational advantage is a single model to orchestrate and deploy. Its development disadvantage is the complex, costly process of turning a text model into a capable speech-to-speech model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separating conversation from thinking
The second path uses a small, full-duplex speech-to-speech model as the conversational interface, potentially running on-device. It maintains the natural exchange while delegating thinking, tool calling, reasoning and other agent capabilities to a background text model. Zeghidour describes a shared background LLM receiving asynchronous queries from hundreds to thousands of small voice interfaces and returning text to them. This is an architectural description, not a measured throughput result presented in the talk.
A crucial responsibility remains with the small interface: recognizing when it does not know enough and should delegate. The division of labor depends on that decision. Zeghidour states the requirement but does not explain how uncertainty is detected, how delegation is trained or how asynchronous results are coordinated with a changing conversation. The proposal therefore identifies the essential handoff without supplying a complete implementation of it.
Although his research culture has favored end-to-end systems and scaling, Zeghidour now favors the hybrid approach for two practical reasons. The first is cost. A gigantic speech-to-speech model capable of solving differential equations is expensive machinery for ordinary small talk. Keeping the conversational interface small creates a way to reserve the more capable backend for work that needs it, rather than routing the entire interaction through a large multimodal model. He offers this as an economic argument without reporting comparative prices or savings.
The second reason is control over the backend. Developers want to replace the reasoning model as better options become available. A unified speech-to-speech model ties speech behavior and intelligence to the same provider and adaptation cycle. Zeghidour cites a voice offering that continued using an older underlying model through several generations of text-model progress, attributing the lag to the expense and duration of audio adaptation. His example concerns the product state he describes during the talk; it does not establish current availability.
Gradium's bet is that a hybrid system can combine natural interaction with economics and agent capabilities competitive with the best cascaded systems. Zeghidour frames this as the path to a viable product beyond an impressive demonstration, rather than presenting that outcome as already proven. He closes by inviting people to use Gradium's models and apply for research scientist and engineering roles.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:01
[music]
- 0:12
>> Okay, hi everyone.
- 0:14
I'm Nel
- 0:15
co-founder and CEO of of Gradio.
- 0:18
So Gradio is a startup based in Paris.
- 0:22
Most of our background is from research.
- 0:24
In particular, we have invented
- 0:26
algorithms such as audio LLMs, speech
- 0:28
speech-to-speech models, neural codex,
- 0:31
and so on and so forth.
- 0:32
And
- 0:33
basically
- 0:35
we started from a research project
- 0:37
called QTI, a non-profit research lab
- 0:39
that has been focusing on voice since
- 0:41
day one. So in particular, we released
- 0:43
in 2024 the first
- 0:45
full duplex speech-to-speech model
- 0:46
called Moshi, the first real-time
- 0:48
speech-to-speech translation system
- 0:50
called Hibiki, and the first
- 0:52
TTS model that can run locally on a on a
- 0:54
smartphone.
- 0:56
And basically I will just say a few
- 0:58
words about what we do, but we are
- 1:00
a model company that trains models for
- 1:02
building voice agents and voice
- 1:04
applications. So we do TTS, API, and
- 1:06
on-device speech-to-text,
- 1:08
speech-to-speech translation, and much
- 1:09
more to come. What we do is that we
- 1:12
train foundation models for audio, and
- 1:13
then we can apply them for a lot of
- 1:15
different tasks.
- 1:17
So I go quickly on voice agents because
- 1:19
it's the fourth talk
- 1:21
about the topic, but basically now we
- 1:24
have these
- 1:26
voice interfaces that we can use to do a
- 1:28
lot of things across a variety of
- 1:30
products and types of interactions with
- 1:32
NPCs, with customer agents, language
- 1:35
learners, coach, and so on and so forth.
- 1:37
And in this talk I tried to go through
- 1:40
the history of this technology and
- 1:44
where I see it going in the next years.
- 1:46
And maybe to start, I think we can take
- 1:48
a look at the announcement of Siri back
- 1:50
in 2011.
- 1:52
And you'll see that it's actually, you
- 1:55
know, I think it it aged pretty well.
- 2:00
>> What is the weather like today?
- 2:06
>> Here's the forecast for today.
- 2:09
>> It is THAT EASY.
- 2:11
>> [cheering]
- 2:12
[applause]
- 2:13
>> LOTS OF THINGS. We've integrated with
- 2:14
the stocks. So, you can ask it about the
- 2:16
stock market. Something like
- 2:18
How is the NASDAQ doing today?
- 2:23
>> NASDAQ composite is down right now at
- 2:26
2,321.70.
- 2:29
>> Again, you can ask this from the lock
- 2:30
screen anywhere. Just press the button
- 2:32
and ask. You can ask about, you know,
- 2:33
the NASDAQ, the Dow.
- 2:35
>> So,
- 2:36
what you just saw is what kind of a
- 2:38
voice agent. It was a bit constrained,
- 2:39
but it was technically a voice agent.
- 2:41
And the architecture behind it, so you
- 2:43
have seen a thousand times today the STT
- 2:45
LLM TTS. Back then, it was even worse,
- 2:47
right? So, there was no LLM, obviously.
- 2:50
So, there was what was called natural
- 2:52
language understanding. So, you would go
- 2:53
from the transcript and uh try to do
- 2:57
basic classification of what is
- 2:58
expected, uh what is the app that is
- 3:00
supposed to be uh controlled, what is
- 3:03
the action to trigger, and so on. So, it
- 3:05
was a very complex pipeline and very
- 3:09
constrained to very specific use cases.
- 3:11
So, it was what was called close
- 3:14
uh ended um
- 3:16
uh dialogue back then.
- 3:18
Then, the next generation of models, uh
- 3:21
I could say, I think was
- 3:24
is illustrated pretty well by the OpenAI
- 3:26
voice mode, not the advanced voice mode,
- 3:28
the original one. In which case, you
- 3:29
just have um a conversational experience
- 3:32
that is open-ended. So, it's much less
- 3:34
constrained than Siri. You can talk
- 3:35
about pretty much anything. At the same
- 3:37
time, you don't have um
- 3:39
as the same agentic capabilities, let's
- 3:42
say, in the sense that it cannot really
- 3:43
give you the NASDAQ or uh give you the
- 3:46
weather.
- 3:51
>> We want to hear a bedtime story.
- 3:53
Tell us a story about the super-duper
- 3:54
sunflower hedgehog named Larry.
- 3:57
Start with telling us a little bit about
- 3:58
him.
- 4:03
Larry was a unique hedgehog unlike any
- 4:05
other.
- 4:06
He had bright sunflower petals instead
- 4:08
of spines. Residing in Meadowville,
- 4:11
Larry spread joy and color wherever he
- 4:13
rolled.
- 4:15
>> So, in that context, basically that was,
- 4:17
you know, kind of second generation of
- 4:18
models where all the logic, the very
- 4:20
complex logic was replaced
- 4:22
uh by an LLM.
- 4:24
And
- 4:25
if we compare Siri and the voice mode
- 4:27
from OpenAI, uh
- 4:29
not the Ad Ons one, Ad Ons one, the
- 4:31
original one,
- 4:32
uh
- 4:33
the big strength is opening the
- 4:35
dialogue, but the agentic capabilities
- 4:37
were inexistent. The naturalness was,
- 4:39
obviously, much better than with Siri.
- 4:41
The latency was still pretty bad. So, we
- 4:43
are talking about several seconds, and
- 4:44
that was documented in the in the
- 4:46
previous presentation.
- 4:48
And so, then, you know, we're in this
- 4:51
third phase, which is a voice, what we
- 4:54
call voice agents. Uh and here I'm going
- 4:56
to show a quick demo of uh of a
- 4:58
drive-thru bot uh live-coded with our
- 5:01
with our voice models.
- 5:03
>> Hi there. What can I get started for you
- 5:05
today?
- 5:06
>> Hi, I'm actually in a hurry and I'd like
- 5:08
to order a chicken sandwich quickly.
- 5:11
>> We have the classic, spicy, deluxe, or
- 5:14
grilled chicken sandwich. Which one
- 5:16
would you like?
- 5:17
>> Uh let's go with the classic chicken
- 5:18
sandwich.
- 5:19
>> Anything else for you?
- 5:21
>> Uh what else do you have on the menu?
- 5:24
>> We have sandwiches, sides, drinks, and
- 5:27
desserts. What can I get for you?
- 5:30
>> Can I get some mac and cheese, maybe?
- 5:32
>> You have a classic chicken sandwich and
- 5:34
mac and cheese.
- 5:36
Your total comes to eight.
- 5:38
>> So, now you know, unlike what we just
- 5:40
saw, it's an actual agent. It's taking
- 5:42
actions. It's keeping track of the
- 5:44
order. It's then going to make you pay.
- 5:46
So, it's it's an actual voice agent that
- 5:48
can do uh real tasks. So, here instead
- 5:51
of having an LLM that is just a
- 5:53
conversational interface,
- 5:54
we have a real agent that is empowered
- 5:56
with tool call, reasoning, planning, and
- 5:58
and all this stuff.
- 6:00
So,
- 6:02
what we see now is we have gained back
- 6:05
agentic capabilities, and actually they
- 6:07
are much more
- 6:09
uh powerful and generic than before,
- 6:12
while keeping a very good level of uh of
- 6:15
naturalness.
- 6:16
And
- 6:17
that's where speech-to-speech LLM came.
- 6:19
In particular, what we could see here is
- 6:22
the latency, it's better with cascaded
- 6:24
system, but it's still higher than you
- 6:27
will have with human conversation. And
- 6:29
as also was explained before, the
- 6:31
naturalness is fundamentally limited by
- 6:32
the fact that you go through text, so
- 6:34
you lose a lot of information about what
- 6:37
uh is said, the tone, the emotion of the
- 6:39
user, and so on and so forth.
- 6:41
So, now that we have tackled
- 6:42
intelligence and agentic capabilities,
- 6:44
speech-to-speech seems like a natural
- 6:46
next step for naturalness and latency.
- 6:48
And so here it's the announcement from
- 6:49
the uh OpenAI advanced voice mode.
- 6:53
>> [clears throat]
- 6:53
>> Hey, ChatGPT. I'm Mark. How are you?
- 6:56
>> Oh, Mark.
- 6:58
I'm doing great. Thanks for asking. How
- 7:00
about you?
- 7:02
>> Hey, so I'm on stage right now. I'm
- 7:03
doing a live demo, and frankly I'm
- 7:05
feeling a little bit nervous. Can you
- 7:07
help me calm my nerves a little bit?
- 7:09
>> Oh, you're doing a live demo right now?
- 7:12
That's awesome.
- 7:13
Just
- 7:14
>> I think we all remember it was very
- 7:16
impressive very impressive release.
- 7:18
And in that context now, all the steps
- 7:21
of STT, LLM, and TTS have been absorbed
- 7:24
into a a single one.
- 7:27
And so now,
- 7:28
intelligence, you know, like naturalness
- 7:30
is
- 7:32
still very good. Uh
- 7:34
actually it can be better because it can
- 7:36
understand non-linguistic information.
- 7:37
Latency is really, really nice.
- 7:40
Honestly, it doesn't make sense to go uh
- 7:42
better than that.
- 7:44
Interestingly and everyone was used any
- 7:46
uh speech-to-speech model can uh attest
- 7:48
that
- 7:50
the intelligence
- 7:51
is still much more limited in that
- 7:54
context than uh the cascaded
- 7:56
counterpart. So, the speech-to-speech
- 7:58
models are fundamentally still limited
- 8:00
compared to the textual models.
- 8:03
Another limitation is turn-taking. So,
- 8:07
people tend to
- 8:08
mix speech-to-speech and full duplex.
- 8:11
But basically,
- 8:13
when you do have a speech-to-speech
- 8:14
model like GPT-3 time, it's still based
- 8:16
on fundamental turn-taking. In the sense
- 8:18
that it's going to segment the
- 8:19
conversation into as long as the model
- 8:21
is speaking or the model is listening.
- 8:23
And to give to show you how this can
- 8:26
make an interaction unnatural, I'm going
- 8:28
to show a a small demo with what is
- 8:30
called backchanneling, which is this
- 8:32
very human thing that you do when
- 8:33
someone talks to you is that you say,
- 8:34
"Mhm, yeah." and so on.
- 8:37
>> Hey, how's it going? Just like to
- 8:39
brainstorm a bit about it with you.
- 8:42
>> Oh, that's a great topic. Yeah, I'd love
- 8:44
to help you brainstorm. Are you thinking
- 8:45
what
- 8:47
Exactly. Yeah, I was thinking
- 8:48
>> No, no, I didn't mean to interrupt, you
- 8:50
know, I was just saying, "Yeah." like
- 8:51
that. You can just keep going. I know,
- 8:53
don't mind me. It's just something I
- 8:54
typically do.
- 8:56
>> Uh no worries at all. Yeah, I was just
- 8:58
going to say we could break it down into
- 8:59
a few aspects. Like
- 9:01
Yeah, exactly.
- 9:02
>> please stop stop interrupting. You know,
- 9:03
it's called backchanneling. Humans do it
- 9:05
all the time. It shows that you're just
- 9:06
following the conversation. That you
- 9:08
don't,
- 9:09
you know, like interrupt you in your
- 9:10
flow. Just just going.
- 9:12
>> Ah, got it. Thanks for letting me know.
- 9:14
>> No problem.
- 9:16
Oh, come on.
- 9:17
>> Yeah, so you see, you know, it's it's
- 9:18
still very annoying. Uh you can have
- 9:20
lightning speed latency. Fundamentally,
- 9:23
this is
- 9:24
uh an issue that can not be resolved
- 9:27
when you're using turn-taking. So, here
- 9:29
that's the walkie-talkie.
- 9:30
Um any real-time model today, I mean,
- 9:33
now there is a bidirectional one that
- 9:35
will come from OpenAI, but it's called
- 9:37
half duplex. So, the model is listening
- 9:39
or speaking. A human conversation
- 9:42
has a constant flow between two people.
- 9:45
People do back channeling. People
- 9:47
interrupt one another, talk on one
- 9:48
another, and so on.
- 9:50
If you have If you're having a relative
- 9:52
on the phone, there is up to 20% of the
- 9:54
time where you are both speaking at the
- 9:56
same time.
- 9:57
And that makes, you know, this very
- 10:00
flexible dynamics in the conversation
- 10:02
makes it much more comfortable for
- 10:03
humans.
- 10:04
And so, to understand how
- 10:07
we can make a model full duplex, I'll
- 10:09
give a very short
- 10:12
presentation of how we train such
- 10:13
models. So, the way you create a
- 10:15
speech-to-speech model half duplex or
- 10:16
full duplex is the following one. So,
- 10:18
you you start from a text LLM, which is
- 10:20
a probabilistic models over over words.
- 10:23
And instead of predicting the next word
- 10:24
based on the past,
- 10:26
what you want to do is rather predict
- 10:28
the next audio based based on the past
- 10:30
audio.
- 10:31
The issue now is that if you pass a raw
- 10:33
audio to your model, which is, you know,
- 10:35
a waveform, it's
- 10:37
air pressure variations.
- 10:39
Uh
- 10:40
basically, you take this sentence, it's
- 10:42
eight words.
- 10:44
It takes around 3 seconds to pronounce
- 10:46
it. And so, at 24 kHz audio, instead of
- 10:49
having eight words, the audio form is
- 10:51
72,000 time steps that you would need to
- 10:53
feed to your LLM. Given that LLMs have
- 10:56
quadratic complexity with sequence
- 10:58
length, so the complexity is the square
- 11:00
of the sequence length. A 10,000 times
- 11:02
longer sequence is 100 million times
- 11:04
more expensive to to process. So, there
- 11:06
is no way you can train an LLM on raw
- 11:08
audio. So, the way you address it is by
- 11:10
creating neural codecs, or you can also
- 11:12
call them audio tokenizers. And
- 11:14
basically, it's an encoder that takes an
- 11:16
audio and compresses it in a very dense
- 11:19
compressed representation, a bit similar
- 11:21
to text. And then you have a decoder
- 11:23
that can reconstruct high-quality audio
- 11:25
from it. So, now
- 11:27
you have gone from the audio domain into
- 11:29
a abstract representation domain, where
- 11:31
you can train an LLM exactly like you
- 11:33
would train it on text.
- 11:35
And the speech-to-speech model from
- 11:37
ElevenLabs, as I was showing before,
- 11:38
works in this fashion. So, instead of
- 11:40
having text tokens into your model, you
- 11:43
have audio tokens that represent either
- 11:45
the LLM or the user, and you put them
- 11:47
one after the other, and the model
- 11:49
predicts the audio tokens uh that should
- 11:52
be said by the model, being given the
- 11:53
context from both sides of the
- 11:55
conversation.
- 11:56
However, you can see that it's still a
- 11:58
sequence between user and system, which
- 12:01
is still half duplex. So, how did we
- 12:04
make the first full duplex model ever?
- 12:06
Very simple. We call it multi-stream
- 12:08
language models. That's the technology
- 12:10
now used also by Thinking Machines for
- 12:12
their interaction model, and most likely
- 12:14
by the for the by the directional model
- 12:16
of OpenAI. Is that instead of having a
- 12:18
transformer that models one sequence of
- 12:20
tokens, it models two of them, so that
- 12:23
both parties can be active at the same
- 12:25
time, inactive at the same time, one
- 12:27
active and one inactive. And
- 12:30
I just show a very quick demo of uh of
- 12:32
how it sounds [snorts] like, but that's
- 12:34
the release of machine August 2024,
- 12:37
uh where we did an announcement live on
- 12:39
stage talking to it for the first time.
- 12:41
And you'll see that the model often
- 12:43
guesses the end of the question, answers
- 12:46
over the speaker,
- 12:48
and both speaking at the same time is
- 12:50
not breaking the flow like we saw with
- 12:52
GPT. The whole thing is just extremely
- 12:54
resilient to the most chaotic uh
- 12:56
situations.
- 12:57
>> So, the planet is serious 22. Can you
- 13:00
plot a trajectory course to it, please?
- 13:02
>> Yes, sir.
- 13:03
>> Okay. How long is it going to take us to
- 13:05
get there?
- 13:06
>> it out. It's approximately 5 months to
- 13:08
get there.
- 13:09
>> Okay, that's that's not too bad. Uh do
- 13:11
you think we have all we need on board
- 13:13
the ship to start the mission?
- 13:14
>> We have everything we need.
- 13:16
>> So, back then it was even a bit
- 13:18
irritating to people because it was
- 13:19
interrupting you all the time. But the
- 13:21
thing is that you can use you could
- 13:23
still use it in extremely noisy
- 13:24
environments with a lot of noise, people
- 13:26
coughing, and so on. And you know, the
- 13:28
flow is just constant. You don't get
- 13:30
this very irritating break of the
- 13:32
conversational flow. So,
- 13:35
these full duplex models, they are the
- 13:37
highest level of naturalness you can
- 13:38
expect. That's the same conversation
- 13:40
with a human.
- 13:42
The thing is, in with our models, it was
- 13:44
even more stupid than
- 13:46
speech-to-speech models that were
- 13:47
already less intelligent than cascaded
- 13:49
systems.
- 13:50
It's probably fine for some use cases if
- 13:52
you just want to have a chit-chat. You
- 13:53
know, the model doesn't need to be very
- 13:55
intelligent. But make an actual full
- 13:57
duplex voice agent,
- 13:58
there is no way we can give up on on
- 14:00
intelligence just to gain uh
- 14:01
speech-to-speech abilities.
- 14:04
So, how do we finally make models that
- 14:06
tackle all these aspects jointly?
- 14:08
And I think interestingly, if you if you
- 14:11
look at the history I showed, there is a
- 14:12
tension between naturalness and
- 14:14
intelligence. So, every time we improve
- 14:16
naturalness or humanness of the of the
- 14:18
models, they were less intelligent than
- 14:20
the cascaded system. The cascaded
- 14:22
agents, they are basically as smart as
- 14:24
the best text models. So, if you have a
- 14:26
voice agent that is powered by the
- 14:28
latest model from Anthropic or OpenAI,
- 14:29
it's going to be extremely smart, have
- 14:31
all the same reliability for tool call,
- 14:33
and so on. Speech-to-speech has this
- 14:35
naturalness
- 14:36
aspect. However, you give up
- 14:38
intelligence to get that. And the reason
- 14:40
why you give up intelligence is remember
- 14:43
that the LLM is a model that has a
- 14:45
certain number of weights that we call
- 14:47
the capacity.
- 14:48
And if you take a text model and now it
- 14:51
not only has to handle text, but it also
- 14:53
needs to understand speech and produce
- 14:54
speech,
- 14:55
it's taking some of its capacity, and
- 14:58
this capacity now
- 15:00
is taken from the intelligence. So,
- 15:01
fundamentally, there is a cost of adding
- 15:04
a new modality to a text model that is
- 15:06
going to be paid in in intelligence.
- 15:08
So, where do we go from here? There are
- 15:10
two paths that are in front of us, and
- 15:12
both are going to be explored at the
- 15:14
same time. The first one is scaling the
- 15:16
model. So,
- 15:18
making your speech-to-speech model
- 15:19
bigger, better pre-trained, better
- 15:22
post-trained, and so on. We likely
- 15:24
progressively increase its intelligence
- 15:26
until it it's good enough for a lot of
- 15:28
use cases.
- 15:29
The second one is splitting the model
- 15:32
between naturalness and intelligence.
- 15:35
The first one,
- 15:36
I'm not at OpenAI, so I don't know
- 15:37
because they don't release their model.
- 15:39
I guess OpenAI is the path one, so it's
- 15:42
a frontier text LLM with a lot of
- 15:44
science around post-training, in-
- 15:46
instruct tuning to fine-tune it on
- 15:48
audio, and teaching it to be quite smart
- 15:51
while using audio.
- 15:54
The nice thing about that is you have a
- 15:55
single model to orchestrate, so it's
- 15:57
quite easy to deploy.
- 15:59
Um
- 16:00
and
- 16:02
one
- 16:03
big aspect, however, is that it's a
- 16:06
extremely complex and costly process to
- 16:08
go from the text model to the
- 16:09
speech-to-speech model.
- 16:11
The second path is to split it. It's an
- 16:13
approach that we introduced in one of
- 16:14
our recent papers called Moushiraq, and
- 16:16
that has been reused by, in particular,
- 16:18
the Thinking Machine Interaction models.
- 16:21
Where, basically, the idea is that now
- 16:23
you have two models. The first one is a
- 16:25
small, maybe even on-device,
- 16:27
full-duplex, extremely natural
- 16:29
speech-to-speech interface. And its only
- 16:31
role
- 16:32
is to keep a very natural
- 16:34
conversation and be able to delegate
- 16:37
all the thinking, tool calling,
- 16:40
reasoning, agentic capabilities to a
- 16:42
background text model. And so, the way
- 16:44
to see it is you have a background text
- 16:46
LLM
- 16:47
that receives asynchronously queries
- 16:49
from hundreds to thousands of small
- 16:51
voice interfaces and just give them
- 16:53
their text, you know? And, basically,
- 16:55
what we did is um
- 16:57
very small full-duplex model that just
- 16:59
needs to know when it doesn't know, so
- 17:01
that it can delegate to the background
- 17:03
model.
- 17:04
And the reason why we believe mostly in
- 17:06
this approach,
- 17:07
um
- 17:08
and I go back to to it later, it's a
- 17:12
our
- 17:13
uh
- 17:14
let's say our culture is more of first
- 17:16
one, the bitter lesson. So, every time
- 17:18
we've been pushing for end-to-end
- 17:19
systems and so on. But now I think the
- 17:22
hybrid approach has two main
- 17:24
advantages. The The first one is cost.
- 17:26
So, speech-to-speech models are
- 17:28
notoriously quite expensive.
- 17:30
And when you think about it,
- 17:32
it's a loss of money to do chit-chat
- 17:35
with gigantic speech-to-speech models
- 17:37
that can resolve differential equations
- 17:39
and so on. So, it doesn't really
- 17:42
make sense economically to get all your
- 17:44
workflow through this gigantic
- 17:46
multimodal mixture of experts.
- 17:48
At the same time, we see that people are
- 17:50
very attached to their ability to
- 17:51
control the backend, to be able to
- 17:53
switch So they So they 5 was released a
- 17:56
few minutes ago. People want to switch
- 17:58
the backend and the intelligence and get
- 17:59
a lot of optionality on that, right?
- 18:02
When you're using a speech-to-speech
- 18:03
model, your your hands are a bit tied
- 18:06
with this model provider. And to give
- 18:08
you an idea of that,
- 18:09
until recently the AdSense voice mode
- 18:11
from OpenAI was powered by GPT-4o,
- 18:13
despite the fact that there have been
- 18:15
several generations of the text model
- 18:17
since then, because this process is so
- 18:19
expensive and so long.
- 18:21
For this reason, we rather bet on the
- 18:24
hybrid approach because that will give
- 18:26
something that is not only very natural
- 18:28
and very nice for demos and impressive,
- 18:31
but also will be a viable alternative
- 18:33
from a economic point of view and
- 18:35
agentic capabilities point of view
- 18:38
to the best cascaded systems that are
- 18:40
still most of the market today in voice.
- 18:43
So,
- 18:44
what now?
- 18:45
Uh you can use our models on gradium.ai.
- 18:48
You can apply to gradium. We are
- 18:50
recruiting research scientists and
- 18:51
engineers. And thanks for your
- 18:53
attention.
- 18:56
>> [applause]
- 19:10
[music]