AI Engineer Europe 2026
Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral
Read the talk
Why Text-to-Speech Models Now Look Like LLMs
Responsive voice agents need speech before the whole waveform is ready. Audio codecs and autoregressive models make that possible, but streaming text into speech remains a separate architectural problem.
From a talk by Samuel Humeau
Before you start: Basic familiarity with language-model tokens and autoregressive generation is helpful; the audio codec concepts are introduced as they arise.
How soon can an agent start speaking?
How do you make a capable text agent feel responsive when the interface is a conversation? Generating convincing speech is only part of the problem: the listener also needs it to start promptly. Samuel Humeau, an AI scientist at Mistral who previously worked at Facebook FAIR, explores that problem through the recently released Voxtral TTS. Mistral combines frontier-model research with tools, products and dedicated support for businesses; a usable speech interface connects those models to another set of applications. The architecture described here is an emerging pattern, not a settled endpoint.
Reading a blog or article aloud is a straightforward offline use case. A voice agent has a tighter timing constraint. Its basic pipeline is speech-to-text → text agent → text-to-speech: transcription lets a strong text model hear the user, and synthesis lets it answer aloud.
On the input side, real-time transcription can leave the transcript ready when the system detects the end of the user's turn. On the output side, playback should begin as soon as the first audio packets arrive. The eventual goal goes further: accept the text agent's first tokens and start speaking while it is still writing. Streaming audio output and accepting streamed text input are two different capabilities. First, consider what output streaming alone buys.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Paul speaks before the waveform is complete
Humeau's demonstration app starts with Paul, a real person's recorded voice. The reference recording describes anxiety being calmed by the taste of good food. The synthesized comparison reads a different passage, beginning “And so with the sunshine,” in Paul's voice. When Humeau replays the generation, the important event is the arrival of the first audio packet: the app can play it while the remaining audio is still being computed. The full waveform need not exist before the listener hears the opening words.
The next demonstration wraps that synthesis model around speech recognition and a Mistral text model. Humeau asks Paul about the conference schedule:
| Question | Agent's answer |
|---|---|
| Session at 12:20 | Reachy Mini: Giving a Body to AI, by Andres Marafioti |
| Session at 11:15 | Beyond Transcription: Building Voice AI That Actually Understands Conversations, by Hervé Bredin |
Asked whether it enjoyed the latter session as much as Humeau did, Paul replies that it has no personal experiences or emotions. The exchange illustrates the practical benefit of early playback: a conversation can proceed while the rest of an answer's audio is still being generated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A voice carries more than the words
Humeau says the model needs only a few seconds of reference audio to clone a voice. After replaying Paul's reference and generated speech, he switches to a French recording that mentions Ryan Coogler and Black Panther. The model then reads the English sunshine passage using that voice. Humeau hears both the speaker's identity and a strong French accent in the output. The interface shows Marie selected, with the English text and completed audio waveform.
He then plays his own short introduction and the generated passage in his voice, joking that he can now discuss complicated problems with himself. These examples make voice configuration tangible: the same text can acquire different recognizable vocal characteristics from a reference recording.
That flexibility also changes branding. Large companies already manage how they sound in advertising. Humeau expects vocal identity to become more widespread, much as companies now define a website's visual identity. The voice of an agent could become another deliberately chosen part of the product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model audio patches instead of individual samples
A waveform records pressure measurements sampled thousands of times per second. Historically, speech systems could concatenate prerecorded words—the familiar SNCF train announcements are Humeau's example. Neural systems subsequently generated waveform samples one after another, and other approaches generated the whole audio at once. For a conversational interface, producing the beginning first is especially useful because it permits immediate playback.
The emerging approach borrows the autoregressive decoder from language modeling. Rather than predict every waveform sample, the model predicts a sequence of audio patches. This preserves sequential generation without asking the large model to take a separate step for every microphone sample.
An audio codec supplies the bridge between waveforms and sequence modeling:
- An encoder converts a short audio frame—roughly 80 milliseconds in this example—into a compact representation.
- A generative model learns to predict those representations over time.
- A decoder converts generated representations back into playable audio.
The challenge is making the representation compact enough to model efficiently while retaining enough information to reconstruct speech.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The information budget between captions and audio
Text has an obvious starting point for tokenization: words already provide discrete units, even though practical tokenizers can do better. Audio does not offer the same shortcut. A vocabulary of roughly 1,000 choices gives only about 10 bits per token; the slide uses 1,024 choices, for which the value is exactly 10. Humeau uses a 200 kbps MP3 as an illustrative compressed-audio comparison. Representing that much information with small discrete tokens would require an unwieldy sequence, so the codec must discard information as well as encode it.
Captions show how far that reduction can go if the system keeps only the text. In a live speech-to-text demonstration, Humeau reports barely 15 bits per second for his text representation. That estimate concerns transcribed text, with no detailed calculation protocol supplied; it is not the information rate of his full voice signal. Captions remove the speaker's voice and other acoustic features that synthesis needs to recover. Speech codecs occupy the space between these extremes, retaining useful acoustic information at a few thousand bits per second.
For Voxtral, Humeau describes 80 ms frames with 37 tokens per frame, rounding the resulting rates to about 12 frames and 500 tokens per second. The arithmetic gives 12.5 frames and 462.5 tokens per second:
python
frame_ms = 80
tokens_per_frame = 37
frames_per_second = 1000 / frame_ms
tokens_per_second = frames_per_second * tokens_per_frame
print(f"{frames_per_second:g} frames/s")
print(f"{tokens_per_second:g} tokens/s")
The codec is trained before the speech generator. It reconstructs a large audio dataset through a bottleneck that forces each frame into the token representation. Reconstruction and adversarial losses encourage it to preserve useful speech detail, while additional guidance encourages some tokens to retain enough linguistic information to reconstruct the text. Compression is therefore shaped by what downstream speech generation needs, rather than by waveform fidelity alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend large-model computation once per frame
Even after compression, hundreds of tokens per second are expensive if every token requires another pass through the large transformer. Humeau describes a roughly four-billion-parameter model; the release's more precise breakdown distinguishes the approximately 4B overall system from its 3.4B backbone. The common architectural response is to separate generation across time from generation within a frame.
The large backbone advances once per audio frame. A smaller depth transformer then generates the tokens belonging to that frame. The system still carries the frame's full token representation, but it avoids paying for a large-backbone step at every token position. The depth model handles the extra work within each time step.
Voxtral differs in that second stage. Humeau describes the frame's tokens as generated together using diffusion, then identifies flow matching as the related technique used here. The Voxtral TTS technical report makes the division more precise: the backbone predicts one semantic token autoregressively, and flow matching generates the 36 acoustic components. The useful distinction is between the autoregressive progression through audio frames and the smaller generator that fills in acoustic detail within each frame.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Conditioning determines when speech can begin
A model that predicts audio is not yet a text-to-speech model. It needs conditioning that specifies what to say. Here, implementations vary more than they do in their shared autoregressive backbone. Some receive all the context before synthesis begins; others receive additional context while producing audio. Voxtral's released model belongs to the first group: it receives a few seconds of the target voice followed by the complete text to pronounce.
Humeau reports 17 ms from text input to first playable audio on a single GPU, excluding network latency. The talk does not specify the GPU, input lengths or caching conditions, so that figure should not be read as end-to-end API response time. It measures the interval after text is available, not the time a text agent spends composing its answer.
Accepting a live stream of text was the next capability Humeau wanted, with no clear architectural winner. The alternatives address the same problem in different ways:
| Approach | How incoming text reaches synthesis | Main distinction |
|---|---|---|
| Independent chunks | Synthesize pieces separately, then stitch | Creates continuity problems |
| Interleaving | Insert new text alongside audio in one sequence | Shared sequence |
| Dual streams | Blend separate text and audio streams during inference | Separate streams |
Chunking can make text available sooner, but independently generated pieces do not automatically form continuous-sounding speech. The other patterns bring the arriving text into the ongoing generation process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the demo and the release actually provide
An audience member notices an apparent contradiction: Paul seemed to generate text and audio simultaneously, even though the model requires complete text. Humeau returns to the app and asks for a poem; Paul declines to recite one. The clarification is about timing, not that particular answer: a small, fast LLM produces the complete response first, and speech generation follows. Short answers make those sequential stages look almost simultaneous. The demo streams audio output, not text input into TTS.
Another question establishes the release boundary. The official model weights are available under CC BY-NC 4.0, but Humeau says the voice-cloning encoder was not released. Users can run the TTS model with the supplied open voices; the released components alone do not let them clone their own voice. At the time of the talk, custom cloning was served through a proprietary offering. Humeau gives a specific reason for withholding the encoder: the team did not want to give everyone the ability to clone any voice.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the agent, improve the speech interface
The next question contrasts native voice-to-voice models with a cascade of transcription, a text agent and synthesis. From the user's perspective, either architecture can appear to be one system that listens and answers. Humeau's case for the cascade is practical: existing text LLMs are highly capable and already perform many tasks. A shared speech interface can wrap different agents without replacing their central intelligence. Streaming the agent's output tokens into speech would make that interface more responsive.
The final question interprets interleaving as Mistral's chosen next step. Humeau corrects that inference: streamed text input is the goal, but the architecture has not been chosen. Interleaving is one possibility; delayed sequence modeling is another. He does not identify an established winner for the team.
The reason to pursue it becomes clear with a longer answer. Paul's short schedule responses concealed the wait for complete text. If the agent instead generates a full page, waiting for the last word before speaking the first creates a noticeable pause. Accepting text as it arrives would let synthesis begin from the first available portion of the answer. Output streaming removes the wait for the complete waveform; input streaming can also remove the wait for the complete written response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original report explaining the speech codec, semantic-token backbone, acoustic flow matching, training and streaming inference.
Official model weights, reference voices and vLLM-Omni deployment instructions, under CC BY-NC 4.0.
Further reading
- Speaking of VoxtralArticle
Mistral's release announcement with voice examples, architecture details and evaluation context.
Updates since the talk
Current guidance for voice customization and speech generation, including the distinction between model processing and API delivery latency.
Read the complete timestamped transcript
- 0:00
[on hold music] So I'm, I'm from Mistral AI.
- 0:17
Uh, and we are going to talk about, uh, speech generation and text-to-speech. Uh, there is an occasion, uh, we released last week our first text-to-speech model and it's open source, so I really encourage you to, uh, check it out.
- 0:29
It's an extremely strong text-to-speech model. We are very proud of it. Um, and for this occasion, uh, I thought we could review some of the recent trend in text-to-speech architecture since there is a, a dominant, uh, trend, uh, emerging these days, although this can change like, uh, ve- very quickly.
- 0:52
Um, and, uh, so this talk is slightly academic and addressed to people who wants to know a bit more about how you, uh, do text-to-speech. Uh, this being said, we have a few years before the machines do all the science for us, so we might enjoy it, uh, today.
- 1:09
Um, I'm Sam. Uh, yeah, I work at Mistral as AI scientist. Before I was at Facebook FAIR when it was called, uh, Facebook.
- 1:19
Uh, and Mistral, a few words about the company. It-- We are a frontier lab. We have-- we've been founded, uh, a couple of years ago. Uh, we produce frontier model, uh, but we're also a B2B business.
- 1:32
We, we help organization, uh, in their AI transformation, which is kind of a buzzword, but literally every company is transforming with AI. We help them by providing them tools, product, and, uh, dedicated people to help them in their custom needs.
- 1:48
Um, back to the text-to-speech. So there, there are a few offline use case of, uh, speech generation, like the, the famous listen to the blog or listen to the article.
- 1:59
But nowadays, the, the king use case for text-to-speech is, uh, its usage within agents. And in particular, it's used to interface, uh, with a chat agent, typically in a pipe like this, uh, where you have a central, uh, chat agent that does text-to-text but does it extremely well and you want to talk to it, so you add
- 2:20
a, a speech-to-text, and you want it to speak to you, so you add a text-to-speech.
- 2:26
Um, as everybody in this conf will tell you, the latency is key here. Uh, so you can reduce the latency on the left by having the speech-to-text done in real time so that when you detect the end of turn, uh, you already have the transcript, it's already done.
- 2:41
Uh, and we are gonna focus a bit on the right side today. Uh, it's also very important that as soon as you have the first audio packets, you, you, you start to, um, to voice them out.
- 2:54
This way the perceived latency is lower. In fact, since your LLM can stream, uh, some text to you, uh, actually what you ultimately want is something like this if, if you're gonna interface a chat assistant, uh, which is a, a real time text input, text-to-speech where, uh, as soon as you have the first token of the LLM,
- 3:16
uh, the, the machine starts to speak. We're going to talk, uh, about it in the, in the end of the talk. I want to focus a bit, uh, at the beginning at the output side, uh, and what it is-- what it means to stream audio.
- 3:30
So to illustrate this, I have, uh, this app that I vibe coded for the occasion. Um, and so we are, we are going to, uh, to use this text-to-speech model that we released, that I mentioned, and we are gonna hear Paul.
- 3:46
So Paul is an actual human being that sounds like this.
- 3:50
The persistent anxiety that fills the rest of my life is calmed for as long as I have the flavor of something good in my mouth.
- 3:58
Um, so this is, uh, uh, like some actual recording on some, uh, actual person named Paul, and we are copying his voice. Uh-
- 4:08
And so with the sunshine and the great bursts of leaves growing on the trees, just as things grow in fast movies, I had that familiar conviction that life was beginning over again with the summer.
- 4:18
Um, so let's focus first on what's happening here. Uh, as you can see-
- 4:23
And so with the sunshine-
- 4:24
We are copying the voice and the, the first audio packet happens first, and we can start to emit audio, which greatly reduce the perceived latency even though the full computation of the, the audio happens like a few seconds later.
- 4:39
And so with the sunshine and the great bursts of-
- 4:40
So in... If you use it in an agent, so here I, I crafted, uh, a small agent using a, a speech-to-text, uh, one of our, uh, LLM and, uh, this very text-to-speech.
- 4:52
So we, we can speak to Paul, um, and... Hey, Paul, uh, can you tell me what's the title of the session at twelve twenty, please?
- 5:04
The session at twelve twenty PM is titled Ricci Mini: Giving a Body to AI by Andres Marafioti.
- 5:11
And what was the session at eleven fifteen, please?
- 5:19
The session at eleven fifteen AM is Beyond Transcription: Building Voice AI That Actually Understands Conversations by Hervé Bredin.
- 5:29
Did you enjoy it as much as I did? [laughs]
- 5:33
I don't have personal experiences or emotions, but I'm glad you enjoyed it. [laughs]
- 5:38
That's all I can do. Um, so the important thing here is that since the audio packet arrived first, you still have a, a decent latency, and you can enjoy the conversation with the agent, uh, despite the fact that the audio is still, uh, not generated fully.
- 5:56
And so we're gonna dig... Oh, yeah, sorry. Uh, I want to make, uh, one digression. So I, I mentioned the, the voice cloning here. Uh, this model can Like, only need a few second to clone the voice of, uh, someone.
- 6:09
Uh, so again, this is how Paul sounded and this is how we generate text
- 6:15
All day long I could And so with the sunshine and the great bursts of leaves growing on the trees
- 6:17
It, it really sounds like ... It's also very good at inferring how a person would speak in an, in, uh, another language. So for example, this is a voice, uh, a [REDACTED:origin] voice.
- 6:30
Maybe.
- 6:31
Ryan Coogler, réalisateur et scénariste des deux Black Panth-
- 6:35
And, like, if we generate
- 6:36
And so is the sunshine and the great burst of leaves growing on the trees, just as things grow in fast movie.
- 6:43
Yeah, so we can clearly recognize her, and we can clearly recognize the very strong [REDACTED:origin] accent, which as a [REDACTED:origin] myself, I do enjoy. Um, I can, I can even clone my own voice.
- 6:54
So this is how I sounded during the recording.
- 6:58
Hi, this is Sam. And, uh, this is how I sound. And so with the sunshine and the great bursts of leaves growing on the trees. It works pretty well.
- 7:08
So this way in my time of dis- delusion and at the peak of my ego, I can discuss with myself, uh, on, uh, on a complicated problem, which is nice.
- 7:19
Um, and so it, it's becoming so easy to, uh, impersonate a voice that it's, uh, it's becoming very easy to, to configure. Um, so it's a small digression, but currently actually a lot of large company, they do have a concept of vocal identity, and they do care in their branding about how they sound, uh, in their advertisement
- 7:39
in particular. But I think this, uh, concept will be becoming more mainstream and just as like a lot of company, uh, define how their website appear as their brand identity, uh, it would be the same for the voice, uh, identity.
- 7:55
Um, oops, sorry. Uh, back to how we do it, uh, in general. So, uh, right, this is, this is, uh, an audio. Uh, physically it's the pressure of the microphone that we measure, uh, from time to time, like, uh, several thousand of times per second.
- 8:12
So it looks like this. And historically, to generate the audio, uh, there have been a lot of, uh, attempts, a lot of systems like i- in the prehistoric time you have like stitching of, uh, words that were spoken like, uh, in the [REDACTED:origin] train, uh, system SNCF for, for those who knows.
- 8:30
Um, and then a neural generation arrived, uh, at some point the trend was to generate each sample one after the other. Uh, then, uh, another era was generating the whole audio at once.
- 8:41
But as we can ... Uh, as we saw, it, it's very interesting to have the beginning of the audio generated first so that we can start to play it out.
- 8:49
So it seems that most labs have converged to some common patterns, and obviously the, the first one is inspired by large language model. We are trying to, uh, transform the problem as a language modeling problem because humanity is extremely good at modeling sequences of token.
- 9:07
So pretty much, uh, everybody is using an autoregressive decoder backbone and, uh, generate audio one piece after the other. Now, um, as I said, we really don't want to generate one sample after the other.
- 9:24
So what we want to do is generate like patches of audio one after the other.
- 9:30
Um, so the expected system looks like this where you, you have an encoder that transform like a, a frame of audio, something like 80 millisecond into something that ideally is a token because humanity is very good at, uh, modeling sequences of token, and then you have a decoder that does the opposite.
- 9:49
Uh, now for text it's pretty easy because transforming the text into tokens, well, it, it's easy, right? You can take like words, uh, as token and it, it works pretty well even though we, we do much better.
- 10:00
Um, for audio it's much harder because one token doesn't have a lot of in- information. Like one token of a vocabulary of a thousand is 10 bits of information and the audio requires much, much more, uh, like a much larger bitrates.
- 10:15
For example, a standard quality MP3, that's 200 kilobits per second.
- 10:20
And so in order to transform this into a sequence of token, like, and not have thousands and thousands of tokens, uh, we need to somehow compress it and reduce the size of it, maybe drop what's not needed.
- 10:35
Uh, an interesting point of comparison is tec- text captioning, uh, because if you drop all the acoustic information and you, you just focus on the text like, uh, with a subtitle track, uh, you actually drop most of the information.
- 10:48
It's a massive reduction and, and you only have, uh, a few bits per second, uh, remaining. So here in this, in, in this demo I use, uh, our real time speech-to-text to measure my bitrate in terms of, uh, tokens per second of text.
- 11:04
And, um, I, I'm a very competitive person and a very ... I'm very good at speaking. Uh, yet, uh, I'm barely 15 bits per second of, uh, actual information.
- 11:14
You, you can try to, to, to, to beat that but like in the grand scheme of things, compared to 200,000, uh, bits of informations per second, that's not a lot.
- 11:25
Uh, obviously, uh, we want to use something that allow us to recover like acoustic, uh, features like the voice and, uh, and other aspects and not just the, the semantic information as in the text.
- 11:39
So the codec that are used, uh, typically reduce the audio to about a few thousand, uh, bits, uh, per second.
- 11:49
Um, in our case for instance, uh, we treat the problem with ... Uh, sorry, we cut the audio as, uh, with pieces of 80 milliseconds, so 12 frame per second, and we transform each frame into several tokens, like 37 in our case.
- 12:05
So we reduce the problem to about 500 tokens per second. I'm not, uh, gonna dig too much, uh, on how we do train these codecs, but obviously first we train them.
- 12:17
Uh, we train them by reconstructing a very large set of audio and using a bottleneck here. Uh, the, the training procedure constrained, uh, the reconstruction to go through a step where each frame is, uh, decomposed into several tokens.
- 12:38
Uh, to do this, typically it's guided so that the model drops the information that is useless and only retain the one that is, uh, useful. And so we guide it via some losses, uh, reconstruction losses, uh, adversarial losses and, uh,
- 12:58
particularly like for, for the, for some of the tokens, we try to make sure that they contains the text information so that you can reconstruct the text from it.
- 13:10
Still 500 tokens per second is a lot of tokens. Uh, and you, you could put them one by one aligned as a sequence like this, uh, but it would make a lot of step of the main transformer that is, uh, at the core of the system, which is usu-
- 13:28
It's huge. Um, in our case, it's 4 billion parameter, [clears throat] sorry, which is, which is still a lot, even though it's not like extremely big now. Um, what most people do then, uh, is,
- 13:45
uh, have, having one step of the backbone per frame and a smaller model here, uh, typically a dev transformer that, um, recomputes all the, the tokens of one frame, uh,
- 14:00
at, at each step. So this way you still have a lot of tokens, you still carry a lot of information, but the computation is much faster.
- 14:09
So this is the, the main, uh, pattern that we see. Uh, even though for that last bit, like the model we released does not follow this pattern, we actually, uh, defer on that part.
- 14:22
I'm not gonna dig too much on it, but just so you know, each frame, which is represented by 37 tokens in our case, we do generate these 37 tokens at once using a diffusion model.
- 14:33
So it's slightly different from the, the vanilla, uh, text-to-speech nowadays. I encourage you, by the way, to read our technical report, which contains all this information.
- 14:44
Uh, also it's a pretty cool, uh, use case of, uh, flow matching, uh, models which is similar to diffusion model.
- 14:53
Uh, now the, the main path is conditioning actually, uh, because so far we are just generating audio, but we are not conditioning it on text, so it's not really a text-to-speech, it's just a speech.
- 15:04
Um, and to conditioning the... For conditioning, there is way more variance across labs and papers and implementations. Um, you have typically two categories. Uh, there are the, the people who focus on, uh, producing the audio once you have the, the text, and some of them who focus on having a stream of text.
- 15:30
Typically, the first category will tend to provide all the context at the beginning and then produce the audio as we saw. Um, and typically, the second category will also add some context as the audio is, is produced.
- 15:46
Uh, the model we release is in the first category. So what we do is we provide the audio of the voice we want to clone, so a few seconds, then the text to pronounce, uh, and that's our context in our case.
- 16:03
Um, yeah, regarding the latency, so it, it, it's, it's pretty fast, uh, if you remove the network and with a single GPU you have 17 milliseconds between the moment where you input your text and the moment where you have the first audio you can play.
- 16:22
Uh, regarding real-time text input stream, uh, which, uh, is our next step, uh, for us, uh, there is no, uh, real... There, there is no clear, there is not a clear winner.
- 16:35
First, uh, it's still possible, you know, like to, to generate independently the text and, and stitch them out, but obviously you will have, uh, a lot of continuity problem.
- 16:45
Um, and there are several patterns. The, the two main ones are people who interleave audio and text, so as soon as there is a new text, they, they put the text in the same, uh, layer.
- 16:58
Uh, and some other who have a dual stream architecture where you have a stream of audio and a stream of text, uh, and you kind of blend them together during the, um, the inference.
- 17:10
Um, how am I doing on time? That's... There's two minutes remaining. Thank you. Uh,
- 17:17
the takeaway is, uh, check our, uh, open source model please. Uh, read the, the technical paper and I hope you learned a few things today. [audience applauding]
- 17:31
We do have two minutes. Uh, how you spend them is up to you. Yeah.
- 17:35
Um, you said that your model first takes all of the text and then produce the audio. But on the example that you showed of the voice agent, it seemed like it was generating the text and the audio at the same time.
- 17:47
No. Um, so the question was like, A, on the demo it looks like we are generating the text and the audio at the same time. Um,
- 17:56
no, for the, for the voice agent... Hello, Paul. It's me again. Can you say anything like, I don't know, a poem?
- 18:06
I'm afraid I can't recite poems.
- 18:08
It, it's just... So the text is produced in one go. It's just that I'm using a small LLM that is very fast, so it, it's nearly immediate and then the audio is, uh, is produced later.
- 18:21
Yeah. Yeah.
- 18:23
Uh, so thanks for a great model, by the way. Uh, I know that the weights are open. Is the voice cloning encoder also open?
- 18:31
Yeah, there is a small asterisk here. Um, we, we didn't release
- 18:40
this part, like the, the encoder part. Uh, which means it, it's the only thing that is missing for you to clone your own voice. Uh, it's a feature that we
- 18:53
only serve like, uh, in a proprietary fashion for now. So what you can do is use the text-to-speech model, use one of the open voices that we provide. Um, we may provide more, uh, in, in the future.
- 19:09
Uh, yeah, so far, we just didn't want to give everybody the ability to clone any voice.
- 19:20
Yeah.
- 19:22
Um, so, so big labs like, uh, Google and Mistral AI are working more on like a native voice-to-voice, let's say, models.
- 19:30
Yeah.
- 19:31
Your lab, ElevenLabs are doing more like this, like, cascading architecture. How ... Like what's your take on, uh, on both like guarding and how do you see things evolving?
- 19:42
What's my take? Um, so on the consumer side, you will always have the impression to speak to a single system that hear what you say and output something, right?
- 19:56
So it, it's purely an architectural, uh, model here. Um,
- 20:02
my take on this is that y- we can go very, very far by just using speech as an interface, uh, especially because these central LLM, they are extremely capable, but they also do a lot of things.
- 20:17
Uh, so just for the sake of being able to use any agent that has been released, uh, with the same interface, it has an advantage, uh, you know, to, uh, to interface.
- 20:27
So you, you can, you can go very far with just interface, especially if you are, uh, doing this kind of thing where you stream the, uh, the text token that's are output by the LLM.
- 20:41
Yeah. Last question because I'm out of time. Sorry.
- 20:44
Yeah. So the, the next steps on the interleaved, uh, function, which, uh, accepts both audio and text sounds really interesting. So if you do that real time, what do you see as the possibilities with that, uh, feature?
- 20:58
Um, no. So, so I, I, I didn't say that our next step would be this, right? I, I just said that there are several patterns to handle a stream of text as input as opposed to like a finite amount of text.
- 21:13
Uh, we actually don't know which one we'll, we'll choose. So whether it's interleaved or another solution like, uh, delayed s- uh, sequence modeling, for instance. Uh, so it's unclear which architecture is best, at least to, uh, to us, at least to me.
- 21:28
Uh, what it l- what it allows is lower latency because as soon as you have the first bit of text that are produced by the LLM, you can start voicing them out.
- 21:39
So you, you ... It-- In this agent, that was not clear because the, the utterance were very short. But imagine I ask Paul to generate a full page of text, um, it, it will be nice if I don't have to wait the end of the text generation to voice it out.
- 21:58
Yeah. Yeah. I mean, so, so it also increases kind of the UX side of things using these types of systems.
- 22:03
Yeah, absolutely.
- 22:04
Yeah.
- 22:05
Yeah.
- 22:06
Very nice. Thank you.
- 22:07
Thank you. [audience applauding] [outro music]