AI Engineer World's Fair 2025
Why ChatGPT Keeps Interrupting You
Read the talk
Why ChatGPT Keeps Interrupting You
A pause is not always the end of a turn. Better voice agents need conversational context, acoustic cues, and a deliberate choice about when to start speaking.
From a talk by Tom Shapland, PhD
Before you start: Basic familiarity with LLMs is helpful; speech recognition, synthesis, and turn detection are introduced as they appear.
A pause can cost a patient
A patient calls a dental practice and starts explaining what they need. The voice assistant interrupts; the patient hangs up; the dentist stops paying the developer. That is Tom Shapland’s opening example of why interruptions matter. In ChatGPT Advanced Voice Mode, an interruption is annoying. In a service workflow, it can end the customer relationship.
Turn-taking is the largely unspoken system that determines who controls the conversational floor. It operates quickly, but its timing is not uniform. In the cross-cultural study Shapland describes, Danish listeners take relatively longer to respond, while Japanese listeners respond almost immediately. Individuals differ too: Shapland describes himself as a slow responder, sometimes prompting people to ask whether he intends to answer. Even his own timing changes with the situation—anger can make him respond faster. A single silence threshold has to contend with all of that variation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a silence-based cascade actually detects
The simplified voice-agent cascade starts with incoming audio and ends with generated audio:
- Speech-to-text transcribes incoming audio chunks.
- An endpoint decision determines whether the user has finished speaking.
- The completed transcript goes to an LLM.
- The LLM streams its response into text-to-speech.
- The synthesized audio plays back to the user.
The important gate is the second step. If it opens during a thinking pause, the rest of the pipeline can produce a perfectly coherent response at exactly the wrong time.
Shapland separates the conventional endpointing arrangement into two parts. A voice activity detection, or VAD, neural network classifies audio as speech or nonspeech. A silence algorithm then interprets the absence of speech: in his example, more than half a second of silence means the user is done. This is the distinction that matters: detecting silence is not the same as detecting a completed thought.
The basic decision can be expressed as a small function:
python
def should_end_turn(
speech_detected: bool,
silence_seconds: float,
silence_limit: float = 0.5,
) -> bool:
return not speech_detected and silence_seconds > silence_limit
The function knows nothing about sentence structure, conversational intent, or whether the user stopped after an unfinished clause. Shapland describes this combination of speech detection and silence timing as common in production at the time of the talk, with newer approaches beginning to replace or augment it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Humans prepare an answer before the turn ends
Human conversation presents a timing puzzle: conversational gaps are often around 200 milliseconds, while planning and encoding even a new word can take roughly 600 milliseconds or longer. Shapland uses this mismatch to motivate prediction. The related research account in Timing in turn-taking and its implications for processing models of language explains why a listener cannot simply wait for silence and then begin all the work of responding. Some preparation must happen while the other person is still speaking.
The main predictive input, in Shapland’s account, is semantics: what the speaker is trying to communicate. Syntax supplies sentence structure; prosody supplies tone, expressiveness, and other acoustic information; visual cues provide additional evidence. He presents the following as a useful proposed processing model, not a complete explanation of human cognition:
- Predict the intended message. Infer what the speaker means and use that inference to anticipate an utterance ending. Keep updating both predictions as more speech arrives.
- Refine the endpoint. As the expected ending approaches, combine meaning with sentence structure to narrow the prediction.
- Finalize with acoustic cues. Near the end, use prosody and other acoustic features to decide when to begin speaking.
This account is full duplex: comprehension and response production overlap. In the figure Shapland describes, zero marks the end of the current speaker’s turn. Well before zero, the listener is both interpreting the incoming message and beginning to prepare an answer. The short audible gap between turns therefore does not represent the full time spent generating the response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the endpoint detector conversational context
Compared with that overlapping human process, the conventional cascade is backward-looking: it measures how long speech has been absent, then starts the response stages in sequence. A practical improvement keeps the cascade but augments its VAD with models that consider semantics, syntax, or prosody.
The LiveKit end-of-utterance model is Shapland’s text-based example. In the version he describes, a transformer receives the last four conversational turns in order: agent, user, agent, current user. It predicts an end-of-utterance token using that content and context. If the prediction indicates that the current turn is unfinished, the system extends the VAD’s silence allowance instead of triggering the agent. The semantic model and VAD work together: silence supplies a candidate stopping point, while conversational context can tell the system to wait longer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An interview that keeps interrupting its answer
Shane’s paired demonstration makes the endpoint decision audible. In the first version, the agent uses traditional VAD. Shane tries to explain that he needed to build a demo but did not know what to build. Before he can finish, the agent says, “Please continue. What happened next?” He tries again, explaining that the project is a LiveKit turn-detection demo, and more questions arrive during his pauses. Eventually he identifies the worst part of making it: the agent kept interrupting him. The agent responds by asking how he overcame that challenge.
The second version enables semantic turn detection. Shane asks to be interviewed about building a demo, then delivers another hesitant answer. He pauses after an unfinished negative clause, pauses again while introducing his next thought, and pauses once more before completing his explanation that the comparison should be side by side. This time, the agent leaves those pauses alone. The audience applauds; Shapland describes the difference as dramatic. What the excerpt establishes is the qualitative listening behavior: unfinished thoughts no longer provoke the same stream of premature questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Listen to the audio as well as the words
A text-based detector gains conversational meaning but does not directly receive the acoustic signal. Audio-aware endpoint models add that evidence: their input is audio tokens, and their output is a probability that the user has finished speaking. Shapland points to Kwin and the Daily team’s open-weight Smart Turn as an example combining transformer modeling with acoustic characteristics.
Endpoint prediction can also be integrated into transcription. Shapland describes AssemblyAI’s streaming speech-to-text release from earlier that conference week as emitting both a transcript and a likelihood that the speaker has finished, using acoustic and semantic features. He also mentions a recently released Kyutai alternative, without giving implementation details. The products here are the versions discussed in the 2025 recording; current repositories and documentation may describe later implementations.
| Approach | Evidence available to the endpoint decision |
|---|---|
| LiveKit’s text model | Recent agent and user turns |
| Audio-aware detector | Speech content and acoustic characteristics |
| User-stream STT endpointing | User speech, without the agent’s turns |
The last distinction is Shapland’s caution about built-in speech-to-text endpointing: a model receiving only the user’s stream lacks the agent’s side of the conversation. Even so, he regards these approaches as a substantial improvement over silence-only endpointing and recommends adopting them through the available voice-agent platforms.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Audio in and audio out still need a turn boundary
Speech-to-speech does not automatically solve turn-taking. A model can accept audio and generate audio while still relying on a separate decision about when to respond. Shapland uses the OpenAI Realtime API as an example: it uses VAD and offers semantic VAD as an option. Despite the name, the useful distinction is the addition of semantic completion judgments to speech activity detection.
Shapland attributes Advanced Voice Mode’s interruptions to premature judgments that the user has finished, based on elapsed silence or preceding content. That is his explanation of the consumer product; the public Realtime API documentation does not establish ChatGPT’s internal configuration. He emphasizes that none of the approaches has perfected interruption handling. He also distinguishes transport from endpointing: LiveKit supplies Advanced Voice Mode’s audio transport, he says, but OpenAI does not use LiveKit’s end-of-utterance model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Continuous listening, continuous output
Full-duplex models move closer to the overlapping process described earlier: they process incoming speech while generating output. At the time of the talk, Shapland says he knows of no commercial applications of these models. Their appeal is that they learn conversational timing from audio rather than depending entirely on handwritten rules.
His analogy is early computer vision. Trying to recognize a stop sign by explicitly checking its color and number of sides proved less effective than letting a neural network learn from images. He links that shift to the field’s emergence from the AI winter. For conversation, the corresponding ambition is to let models learn when to speak from recorded interaction. That describes the timing-learning principle, not a claim that every full-duplex model is trained exclusively on raw recordings. Shapland’s concern is capability: he characterizes the models under discussion as small, trained on limited data, and weak at instruction following.
Moshi makes the continuous-output idea concrete. It is always listening and always emitting output. When it should not speak, its output is natural silence. Silence therefore becomes part of the generated audio stream, rather than merely the absence of a response job.
Shapland next discusses SyncLLM, which he associates with Meta AI’s experimental in-app full-duplex experience; that specific model identity remains his attribution. He describes its anticipation of user speech as about five tokens, or 200 milliseconds. These are audio units, not ordinary text tokens: more precisely, the paper’s mechanism estimates an unavailable user-audio chunk and later replaces that estimate with actual input. Its raw audio units run at 25 Hz before deduplication, so five raw units correspond to 200 milliseconds, rather than a fixed five-text-token forecast. The important mechanism is acting with an estimate of incoming audio while continuing to listen for the real input.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why a better cascade may still win in production
Shapland’s forecast favors improved cascades over full-duplex models for commercial voice agents. The deciding requirement is control, including something as concrete as how an agent pronounces a brand name. Smarter VAD augmentations can improve endpoint decisions, while faster components elsewhere in the cascade leave more time to make those decisions well.
This is a prediction about the engineering path, not an established winner. His reason for resisting mandatory imitation of human cognition is practical: computers already perform mathematics differently from people, and LLMs need not reason through the same mechanisms humans use. Voice AI may likewise achieve useful conversational timing through a different arrangement of components.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the demonstration leaves unanswered
The first Q&A question identifies a missing part of the improved demonstration: the audience never hears the agent respond at the end. Shapland explains that he shortened a two-minute demo to fit his speaking slot. The moderator jokes that turning speech off altogether would also eliminate interruptions. Shapland says the detector eventually identifies the end of the turn from context, but that eventual response is not shown in the excerpt. He directs viewers to LiveKit’s Twitter account for the full demonstration.
The next question asks whether avatars and other real-time systems should use visual cues for turn detection. Shapland again puts semantics first: despite the importance of vision to humans, he ranks visual cues below conversational content for predicting an ending. He expects someone to be exploring multimodal turn detection, but has not seen a concrete implementation he can point to.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cost, integration, and evidence
Asked about call costs and repeated response generation, Shapland says text-to-speech tends to be the most expensive part of a cascade. In that setting, invoking the LLM multiple times within a turn can be comparatively inexpensive. He does not provide an average call price or a provider-specific breakdown: he is not personally building agents and does not have those unit economics at hand. Conversation length matters, and he suggests using cost calculators to evaluate a particular setup.
Another audience member asks why OpenAI does not use LiveKit’s detector, whether developers can use it themselves, and whether the team has measured its improvement. Shapland does not know OpenAI’s reason; he had joined LiveKit only weeks earlier. For developers, he describes following LiveKit’s agent quick start and adding one line to enable the detector. He says the model is open-weight and free to use, while expressing uncertainty about whether the quick start already includes it by default.
On evaluation, he reports that LiveKit has internal test-set benchmarks, but supplies no scores or user-study results. The audience’s suggested percentage improvement is a question, not a result. His substantive concern is the difficulty of constructing a good benchmark dataset: even the team’s recent discussions centered on what suitable evaluation data should look like. In his view, the industry still lacks a good shared benchmark for turn-taking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An acknowledgment is not always an interruption
The final question reverses the problem. What happens when the human speaks while the agent is talking? A brief acknowledgment such as yeah, mm, or yes can mean that the listener is following along. The same words can also begin a substantive response. These backchannels make it unsafe to equate all user speech with a request to take the floor.
Shapland distinguishes this from the talk’s main focus: preventing the AI from interrupting the human. For user interruptions, he describes LiveKit’s then-current approach as a duration rule using Silero VAD or ordinary speech activity detection. Speech lasting more than a configured number of milliseconds is treated as an intentional interruption; he does not specify the threshold. A model that distinguishes backchannels from attempts to interrupt is something the team wants to build, not a capability he claims is already implemented.
Full-duplex models offer an intriguing counterpart on the generation side. Shapland says Meta’s experimental experience can naturally produce acknowledgments while listening, having learned that behavior from conversational audio. Generating a well-timed acknowledgment and recognizing whether a user’s acknowledgment should stop the agent are different tasks. The remaining challenge is not simply to hear speech sooner, but to understand what that speech means for who should hold the floor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A full-duplex dialogue model that predicts interleaved speech streams and estimates missing user audio to handle latency.
Kyutai's speech-text foundation model and full-duplex spoken dialogue framework.
Further reading
Research on conversational timing and how listeners prepare responses while comprehension continues.
LiveKit's companion explanation of transformer-based end-of-turn detection.
- Kyutai's Unmute announcementArticle
The May 2025 announcement of a voice interface combining streaming transcription, synthesis, and semantic turn detection.
Updates since the talk
Current Smart Turn model, training materials, and examples for running audio-based endpoint detection locally.
Official documentation for voice activity detection in the Realtime API.
Read the complete timestamped transcript
- 0:15
I'm gonna jump right into it. I'm gonna be talking about voice AI's interruption problem. Then after that, I'm gonna talk about how we currently handle interruptions and turn-taking in voice AI, what we can learn from the study of human conversation on how to handle-- how humans handle turn-taking, and then about some of the really neat and interesting
- 0:33
new approaches out there for handling turn-taking and interruptions in voice AI agents. Um, voice-- Excuse me. Interruptions are the biggest problem in voice AI agents right now. When you're talking to ChatGPT Advanced Voice Mode and it interrupts you, it's annoying.
- 0:51
But when you're talking... When a patient is talking to a voice AI dental asst-assistant and it interrupts the patient, the patient hangs up and the dentist stops paying the voice AI developer.
- 1:02
This is our collective problem. This is all of our problem that we're all, we all have to solve.
- 1:07
Um, and the, the problem is, like, turn-taking is just hard. Um, let me first, like, define what turn-taking is. Turn-taking is this unspoken system we have for who controls the floor, uh, on, um, uh, between speakers, uh, during a conversation.
- 1:24
And fundamentally, it's hard because turn-taking happens really fast in human conversation, um, and there's no one size that fits all. So this is this really cool, um, study that I pulled this data from where they're looking at how long it took a listener to start responding after the speaker finished speaking, um, across different cultures.
- 1:44
And we can see that the Danes take a relatively long amount of time to start speaking after, uh, the other speaker finishes speaking. But the Japanese do it almost instantaneously.
- 1:56
So there's differences across cultures. That's part of what makes it hard. There's also differences across individuals. I'm one of those people that takes a long time to respond. Even before I got into voice AI, people would sometimes comment being like, "Are you gonna respond?"
- 2:08
I'd be like, "Yeah, yeah. Thinking about it." Um, and, uh, also, like, even though I'm one individual, there's a lot of variability in how quickly I respond. Like, if you make me angry, I'm probably gonna respond kinda quicker.
- 2:21
Um, so it's just a, a hard problem. And in the, the next slide I wanna talk about for people who are not very familiar with how voice AI agent pipelines work, I'm gonna provide like a simplified overview of how we handl-handle turn-taking and interruptions in voice AI agents currently.
- 2:38
So the user starts speaking. That's the speech input. And that audio, those au-audio chunks are passed to a speech-to-text model. Um, and that speech-to-text model, uh, transcribes the, uh, audio into, uh, a transcription.
- 2:51
The next step is something called a VAD that determines whether or not the user has finished speaking. Um, I'm gonna go more into that in a moment. The next step, if the user has finished speaking, the transcript is passed to an LLM and the LLM outputs its chat completion.
- 3:07
That chat completion is streamed out and that stream is passed to a text-to-speech model where it's converted into audio. And that audio, which is the audio of now of the voice AI s-agent, is passed back to the user.
- 3:19
Um, let's dig more into the voice activity detection system. It's a system with primarily two parts. So it's a machine learning model, a neural network that is detecting whether or not somebody is speaking.
- 3:33
So it's like speech or not speech. It's pretty simple, um, machine-- It's not-- I shouldn't call it simple, but it's, it's a really neat model, but it's, it's ultimately just looking at speech or not speech.
- 3:44
And then the next thing, next part of it is a silence algorithm. And the silence algorithm is saying, "Okay, if the person hasn't s-spoken for more than half a second, they're done speaking and it's time for the agent to start speaking."
- 3:59
So that's the-- In most production voice AI systems, we're using something like that sort of VAD. Um, and that's changing. We're building all sorts of new interesting things that I'll cover later in the presentation.
- 4:09
Um, the next part of my presentation, I want to dig into what we can learn from linguistics and academic research about how human-- how turn-taking works in human conversations.
- 4:19
And one of the lines I read in a, a paper I read was, that I really liked was that turn-taking in human conversation is a psycholinguistic puzzle, and that we respond in 200 milliseconds, but the process of like finding the words and generating speech and articulating speech takes 600 milliseconds.
- 4:36
So how can we possibly be speaking so quickly? How can the listener start returning, uh, an answer so speak-- so quickly when it takes much longer to actually generate the speech?
- 4:47
And the answer is that there has to be prediction going on. The listener is predicting when the end of turn is going to occur, and then they're go- they start to generate speech before that end of turn.
- 4:59
And what are the primary inputs in creating that prediction? The primary inputs on creating that pred-prediction are the semantic, and this one's the most important one, like what the content is of the, of what the person is saying.
- 5:10
Other inputs into this, you know, prediction algorithm in our head of when we're trying to predict when the speaker is gonna finish speaking is the syntax, the structure of the sentence, the prosody, like the, the expressiveness, um, the tone, and then also visual cues.
- 5:27
Um, like most things in, uh, how the human mind works, we, we actually don't really know. Like, you know, that's, that's complicated. Um, and-- But I, I would say one of the generally accepted models is what I'm gonna walk through now of how turn-taking works in the human minds.
- 5:44
It's broken up into three stages. The first stage is semantic prediction. So what the listener is doing, and you'll notice this as you speak to other people at the conference i- if you like are paying attention to your, your thought process, is you're constantly inferring the intended message of the person that's speaking to you.
- 6:01
So before they finish speaking, you're kind of figuring out, "Wait, what are they trying to say?" And then they're, you're using what the ... what your, your prediction of what they're trying to say to, um, uh, to then use that information to predict when the end of utterance will occur.
- 6:17
And you're not doing this just once. You're doing this multiple times, again and again, right? You're constantly updating this prediction as the speaker keeps going. So that's the first stage.
- 6:25
Um, then the, the next stage, once it seems like you have a general idea, your, your prediction is coming true of like when you think the end of utterance will occur, as you start getting closer to that, you start refining that, that endpoint prediction, um, based on both the semantics and the syntax.
- 6:43
And then as you start getting really-- As the speaker starts getting really close to the end of turn, the listener finalizes the prediction by using prosody, by using information around like the tone and other acoustic features.
- 6:55
Um, so it's three steps: a semantic prediction, a refinement, and a finalizi- uh, a finalization. And one of the things I wanna point out is that the human mind is full duplex.
- 7:06
We're both processing input and we're starting to generate output at the same time. And I think that's really nicely described in this figure from this paper. Um, and so on the X-axis, what we have here is time, where the zero millisecond is the end of the speaker's turn.
- 7:23
And what these different blocks represent is what, of mental processes that are happening inside the mind of the listener. And you can see well before the end of the turn, there's this whole comprehension, uh, track, um, that's going on where the listener is inferring the intended message of the speaker and making predictions about when they're gonna stop,
- 7:43
finish speaking. And at the same time, there's also this production or generation track where the, um, the listener s- is starting to produce what they're going to say.
- 7:55
And we're gonna talk more about full duplex models in computer, in silico rather than human minds in a little bit in my presentation. Um, oh, Jordan Dearslay just texted me. [laughs]
- 8:09
He, he texted "boo." Like how does he know to boo me? He's not even here. Um,
- 8:16
okay. It's, uh, it's all good. We'll keep rolling. Uh, so, um, so let's go back to, uh, um, like let's contrast this really interesting complex process that's going on in the human mind compared to mod- current voice AI systems.
- 8:31
You'll see it's just so much more simple, right? It's just speech or not speech. It's looking backwards. It's not making a prediction. Um, it's done in serial. Nothing's happening in parallel.
- 8:41
Um, so it's, it's much more simple. And that's part, part of the problem, right, of why these interruptions are happening. Um, [lip smack]
- 8:50
so there's, uh, I'm gonna talk through three types of models and the approaches people are using in these three types of models. Um, so the, the prevailing model for building voice AI agents is the cascading model, system of models where you have, um, what we talked about earlier, speech-to-text, VAD, LLM, TTS.
- 9:09
And, uh, what we're doing in those is we're augmenting these new... The new approaches to better handling interruptions i- is we're augmenting the VAD with models that look at the semantics, syntax, or prosody.
- 9:23
Um, and, uh, I wanna jump into an example of it. Um, I really have too much content for, uh, for my allotted time that I'm looking at down here.
- 9:35
Um, but maybe I can just take some time from Jordan. Um, so, uh, [chuckles] let me, let me, um, give an example for one of these semantic-type models that is used to augment VAD.
- 9:46
I'm gonna talk about our model at LiveKit. It's a text-based semantic model. So what we're doing is we're taking the last four turns of the conversation as input. So that means that it's the, uh, the voice AI agent's turn, then the user's turn, then the voice AI agent's turn, and then the user's current turn.
- 10:05
Those are the inputs into a transformer model. And what we're-- The token that we're predicting, um, because this is an LLM, we're predicting the end of utterance token. And if that end of utterance token, based on the s- the content, right, based on the, the, the context and the semantics of that, that input, um, if the end
- 10:24
of utterance token is saying that the end of turn, it hasn't happened yet, then we don't... We extend the silence algorithm part of the VAD and say, "Don't trigger the end of turn.
- 10:36
Wait longer." Um, so they work in concert. But that's generally the idea of how it works. I'm gonna walk through a quick demo of h- of how this works in action.
- 10:45
I needed to build a demo, but I wasn't-
- 10:48
Oh. So in this first part of this demo, what Shane is going to do is, uh, talk to a voice agent that is just using the, the traditional VAD that I was discussing earlier that just looks at speech or not speech.
- 11:04
And in the second half of the demo, he's using our semantic end-of-utterance model.
- 11:12
100% sure what it was that I should-
- 11:15
Please continue. What happened next?
- 11:16
I had to, I had to build a demo for-
- 11:20
What was the project or what-
- 11:21
... a LiveKit, [chuckles] a LiveKit, uh, turn detection, a turn detection demo where one of the agents, like-
- 11:33
Got it. What challenges did you face while building that demo?
- 11:35
It just, it just kept on interrupting me. Uh, being interrupted by the agent constantly. That was the worst part, for sure.
- 11:44
I understand. How did you overcome that challenge during the demo? [laughs]
- 11:50
Well, hopefully, we're gonna overcome the challenge by using the new turn detection model that LiveKit's offering. So let's try that one now instead.
- 12:03
Hey, can you interview me about a time when I had to build a demo?
- 12:10
Absolutely. Can you share your experience with building a demo?
- 12:14
Yeah, definitely. So, I needed to build a demo, but I wasn't,
- 12:23
I wasn't 100% sure what it was that I should build. And then I,
- 12:32
I was thinking, like, probably the best way to show that would be, uh,
- 12:38
would be side by side. [audience applauding]
- 12:42
Yeah. Thank you. I'll, I'll, I'll let kn- I'll let Shane know that you all applauded his demo. Uh, he'll appreciate that. Yeah. You can really see it's a night and day difference when you augment the VAD with models that look at the semantics and syntax and prosody.
- 12:55
Um, which is a good segue to my next slide. So, there's another type of-- another approach people are taking to augmenting the VAD. W- and what they're doing is not just taking the semantic input, the text-based input, but they're also looking at the audio signal as well and trying to infer things from the acoustic features of the,
- 13:13
the dialogue. So they're taking-- Uh, the basic idea is the input is audio tokens, and the output is the probability that the user is finished speaking. Um, Quinn and the Daily team have built their also open-weight smart turn model that, that, uh, is, is this neat combination of a model that is both transformer and looking at acoustic,
- 13:31
um, characteristics. Um, and then one of the new things that has just emerged has been, um, Assembly AI dropped their, their speech-to-text, new s- streaming speech-to-text service earlier this week.
- 13:44
And their model is really neat in that it emits-- it takes audio in and it emits out both the transcript and a likelihood that, uh, the speaker is finished speaking.
- 13:55
Um, so it's one model that's kind of doing both these two things at the same time. Um, and it's also looking at the acoustic features and the semantic features.
- 14:03
One of the things I wa-- Qtai ha- also has one that they recently released that's pretty neat. Um, one of the things I wanna note about this, though, is if you're using your, uh, speech-to-text's built-in end-of-utterance model, it's only seeing half the context.
- 14:15
It's only seeing what the user is saying. It's not also seeing what the agent is saying, so it doesn't quite have the full picture on the context. Um, but it works remarkably well.
- 14:24
These-- All these approaches work remarkably well and are a major step forward from these more traditional VADs, and like, definitely something that if you're building a voice AI agent after th-this conference, you should go, like, implement it.
- 14:36
They're pretty easy to implement on the different platforms too. Um, okay. So we often talk about in, uh, uh, speech models or in voice AI that, um, speech models are, uh, are going to save us.
- 14:52
Um, speech-to-speech models, audio in and audio out, are going to save us. Um, but actually, if you look at how these models work, the, like, OpenAI's real-time API, they're still using a VAD on the, on the internals.
- 15:05
Um, so they're still just looking s- at speech or not speech, or you can al- you can opt to turn on their semantic... They call it semantic VAD, which is kind of a, uh, paradox.
- 15:15
It's not the best term. But turning on a semantic model that augments the VAD. Um, so to answer the title of my talk of why ChatGPT Advanced Voice Mode keeps interrupting you, it's because it thinks you're done speaking based on how long it's been since you last said a word, um, or based on what you've, uh, what
- 15:30
you've said previously. Um, and it's just not quite cutting it, um, when those interruptions happen. Um,
- 15:38
uh, and it's a problem that is not totally solved. I wanna also bring that up too, that like, this is an ongoing problem, um, with all the different approaches.
- 15:46
Nothing has perfected it yet. Um, our end of-- LiveKit doesn't-- Although we power the transport, the audio layer transport for, uh, Advanced Voice Mode, um, OpenAI is not using our end-of-utterance model.
- 16:00
So the next topic I wanna cover is the, um... [lip smack]
- 16:04
I'm running out of time here, but, uh, is full duplex models. These are really neat. So a full du- duplex model is more like a human mind in that it's processing input and generating speech at the same time.
- 16:14
Um, and as far as I know, there's not really any commercial applications of these. Um, but they're, they're fundamentally, they're intuitive talkers. They're trained on the raw audio data, and the analogy I like to use is that it's like computer vision.
- 16:27
In the early days of computer vision, we were handwriting algorithms to try to recognize a stop sign based on the color and the number of sides on it, et cetera.
- 16:35
Um, and it just didn't work very well. But when we started giving the raw image data to the neural network and let the neural network figure it out, all of a sudden it just started working.
- 16:44
And, uh, I think it's, uh, and actually that, that, uh, what we learned from computer vision, that really helped us emerge from the AI winter. That was a major kind of, uh, seeding process for where we are now with AI.
- 16:54
Um, and the-- it's a similar analogy with full duplex models in that we're handing them the raw audio data, and we're just letting them figure out how turn-taking works rather than trying to handwrite all the rules.
- 17:04
Um, but the downside of these models is they're really optimized for, like, being really good at turn-taking, and they're kind of dumb LLMs. They're small models. They're not trained on a lot of data.
- 17:13
They can't do instruction following very well. Um, and just to give you a sense of like more specifics of how these models work, let's talk about the Moshi model.
- 17:22
Um, what really made it more concrete for me of how this model works is this idea that it is always listening to input, and it's always generating output. And even when it's not its turn to speak, it's emitting natural silence.
- 17:34
So it's just basically emitting silence that you can't hear, but it's still always emitting silence. Um, so it's always kind of doing both just like a human is. Um, SyncLLM, which is Meta AI's, uh, full duplex like experimental mode that you can access inside the app, um, is a similar d- full duplex model, or it's also a
- 17:53
du- full duplex model. Something neat that I wanna bring up about SyncLLM is they're actually, in the internals of that model, they're forecasting what the user's say- saying about five tokens ahead or 200 milliseconds ahead, which is more closely like what humans are doing, except we're, uh, forecasting a much longer timeframe.
- 18:12
And then lastly, my predictions for the future of how we'll sol- solve this problem, uh, is
- 18:17
I think full duplex models are neat, but I don't think they're gonna solve the problem. Like, I think we just, for, for real production commercial use cases of voice AI, we need more control.
- 18:27
Um, and we need more control over how it says things like brand names. Um, and instead, what I think is gonna happen is we're gonna get smarter and smarter VAD augmentations and faster and faster models in the cascade pipeline, and we're just gonna have more budget to work with to do a good job with this sort of
- 18:42
thing. Um, and the reason I think that's true is, like, computers don't do math the same way humans do. They don't have the same conceptual way of thinking about it, and, uh, LLMs think differently than us.
- 18:53
And similarly, I wouldn't expect voice AI to use the same mechanisms as the human mind to generate speech and, and to talk. Um, thank you all for your attention.
- 19:03
This was fun. Really appreciate it. [audience applauding]
- 19:07
We do have some time, so I don't know if you want to take Q&A. Um-
- 19:12
I would love to.
- 19:13
We could do that. Um, and I could start with the first question. So the demo you showed, there, there wasn't any response at the end, right?
- 19:24
Uh, I cut off the demo. It's actually a two-minute demo-
- 19:26
Right
- 19:26
... and I only have 18 minutes to speak, so I truncated on-
- 19:28
Fair
- 19:29
... the sides.
- 19:29
Do-- Because I was like, "Okay, maybe you just turned everything off," and it was an impressive demo. [laughs]
- 19:35
Oh, yeah.
- 19:35
No interruptions. [laughs]
- 19:36
No speaking, yeah. No, it's no interruptions.
- 19:37
Do you ha- do you have the end of the demo? Do you wanna show it or...?
- 19:41
Um-
- 19:42
No, no worries if not
- 19:43
... it's, it's, it's more of the same idea. Like what you could see-
- 19:45
Okay
- 19:45
... is that Shane was like, you know, taking his time talking and really st- pausing and thinking, and it wasn't interrupting him. And then when it eventually would find his end of turn based on the context.
- 19:54
Cool. Can we find it on your Twitter or...?
- 19:57
Yeah. It's, it's on our link, our LiveKit Twitter. Mm-hmm.
- 19:59
Awesome. Yeah. So we can look that up on the LiveKit Twitter. Uh, awesome. Yeah, we can take some questions. I saw you had one.
- 20:08
Hi. How important are visual cues for turn detection in, in the human context? And are there any, um... Is there any development to kind of replicate that in the, uh, voice AI context as well?
- 20:23
Yeah. It's a really neat question of like how important are visual cues and are, are people working on integrating that into the turn-taking, um, intelligence for, uh, avatars and real-time experiences.
- 20:36
So visual cues are actually, despite the fact that we are visual animals, um, very, very much so, like visual is the most visceral, like, you know, input for us.
- 20:47
Visual cues are actually pretty low down the stack of like, uh, predictors for when it will be end of turn. It really is semantics. That's like one of the main messages that I wanna convey to people from this talk is it's the content of what people are saying is the main thing we're using to predict when they're
- 21:02
gonna finish speaking. Um, and then these visual cues are, are... and these other ones are ancillary to it. Um, and I'm sure somebody's working on building something really cool where it's like multimodal and looking at visual cues to look at the e- to infer the end of turn.
- 21:16
Um, I'm just haven't seen it yet and can't keep up with all the AI stuff on the internet. Yes.
- 21:22
What is the average cost for... What is the average cost for usually a voice, uh, generated call? And then how is-- what is the effect when you try to keep regenerating the response?
- 21:40
W- so the question is, what's the average cost for a voice AI call, um, and what is the cost when you keep trying to regenerate the, the response? Um, so I would, I would first say that the-- I think your, your reference to what does it cost to keep trying to regenerate the response is, um,
- 22:02
what I-- The way I wanna answer that is actually the thing that's most expensive in the pipeline tends to be the text-to-speech. So there's all these optimizations you can do in the cascade, and if you end up hitting the LLM multiple times within a turn, it's not, it's not all that costly and those sorts of things.
- 22:17
Um, there's some really neat calculators online. Because I'm personally not a voice AI agent builder and, uh, those unit economics don't, uh, don't directly affect me, I don't have the numbers off the top of my head, but there's some really nice calculators.
- 22:29
It's gonna depend on how long the conversation is and that sort of thing.
- 22:37
You had a question?
- 22:40
Yeah, thanks for the demo. That was great. Uh, the question is about your new model that you just shown and that blew us away. So, uh, one is like why is ChatGPT not using that model to improve their stuff?
- 22:52
And two, is it available for us to use now if I were to build a, uh, voice bot on LiveKit? And, uh, three, maybe during your development of that, one demo is great, uh, do you also do some kind of benchmarking with user to see if, you know, this is like 50% better or something like that?
- 23:15
Yes. So the first question is about why isn't OpenAI using our end-of-utterance model. I don't know why they're not. I think that's maybe above my pay grade of this company I just joined four weeks ago.
- 23:26
Um- [laughs] ... uh, and the, the second question is like, can you-- is our end-of-utterance model available, um, for use? And so it's really easy on our website to, uh...
- 23:39
or it's really easy to follow our quick start on our website and build a voice AI agent that you can talk to. Um, and it's just one more line in our, in the pipeline that you, you build.
- 23:49
You just turn on-- You have one more line in there, and you get to use our end-of-utterance model. It's open weight. It's-- You don't have to pay for it.
- 23:56
It's just baked in. Um, and our docs show you pretty... I think by default it's in there. Um, and the, the third question, uh, remind me of the third question.
- 24:06
I'm sorry.
- 24:06
Uh, did you do a benchmark?
- 24:09
Ah, yes. Benchmarking. Um, so we have benchmarks where we have our test data set and, you know, the numbers of course look great. Um, uh, [laughs] but I think, uh- I was on a long call with our machine learning team this morning, where we spent a lot of time just talking about, like, how do we get a good
- 24:26
dataset for benchmarking it? And it's just, it's just really-- it's a tough problem. Um, and I feel like the industry as a whole doesn't have a good benchmark around turn-taking, um, and that it's something that I'm sure will eventually emerge.
- 24:42
Uh, okay. We'd-- we'll do one last question, I think. So there was... Yes, you in the back. Not great to be sitting in the back if you wanna ask questions, but-
- 24:52
Sorry. [laughs] Well, thank you, Tom, for the time on, and the presentation. Uh, I got a question related to the backchannel. So how did you tackle the backchannel challenge in the turn-taking detection problem?
- 25:04
So first, uh, for a natural conversational AI, the backchannel is the one that cannot be, uh, ignored, and, uh, sometimes it would cause trouble for the voice agent to detect whether it is the w- it is the endpointing.
- 25:18
And second, for a typical backchannel like yeah, mm, uh, yes, uh, those words can occur like a backchannel, or it can be started as the, uh, the agent who sh- uh, should be response in the, in the following period.
- 25:33
So how the backchannel is handled should be kind of important in this field.
- 25:40
So, um, my question was-- or the question was about not when the-- not the case where the AI is interrupting the human, but the case where the human is accidentally interrupting the AI.
- 25:52
Um, and I didn't really cover that in my talk. I was mostly focusing on the AI interrupting the human. Um, we don't have... Our approach is simple, like we're just using like the Solero or the, the normal VAD approach of like, if the person is speaking for more than X milliseconds, assume it's not a backchannel and that
- 26:12
they're actually trying to interrupt the voice AI. But one of the things we wanna build is another machine learning model that can, like, recognize the difference between whether or not it's a m- backchannel or someone trying to, uh, interrupt the, the voice AI.
- 26:27
One quick note on the full duplex models, uh, the Meta AI one can natively backchannel 'cause it's, like, learned from the raw audio data, so when you're talking to it, it'll go, "Mm-hmm.
- 26:37
Uh-huh," which is just so neat. Um, and, uh, yeah, it's just a, it's a tough problem, the backchanneling thing.
- 26:47
Awesome. Um, yeah, if you have more questions, you can find Tom. Uh, please give another warm, uh, applause for Tom. [clapping]
- 26:53
Thank you all. [outro music]