Why ChatGPT Keeps Interrupting You

Talk transcript

1. 0:15

   I'm gonna jump right into it. I'm gonna be talking about voice AI's interruption problem. Then after that, I'm gonna talk about how we currently handle interruptions and turn-taking in voice AI, what we can learn from the study of human conversation on how to handle-- how humans handle turn-taking, and then about some of the really neat and interesting

2. 0:33

   new approaches out there for handling turn-taking and interruptions in voice AI agents. Um, voice-- Excuse me. Interruptions are the biggest problem in voice AI agents right now. When you're talking to ChatGPT Advanced Voice Mode and it interrupts you, it's annoying.

3. 0:51

   But when you're talking... When a patient is talking to a voice AI dental asst-assistant and it interrupts the patient, the patient hangs up and the dentist stops paying the voice AI developer.

4. 1:02

   This is our collective problem. This is all of our problem that we're all, we all have to solve.

5. 1:07

   Um, and the, the problem is, like, turn-taking is just hard. Um, let me first, like, define what turn-taking is. Turn-taking is this unspoken system we have for who controls the floor, uh, on, um, uh, between speakers, uh, during a conversation.

6. 1:24

   And fundamentally, it's hard because turn-taking happens really fast in human conversation, um, and there's no one size that fits all. So this is this really cool, um, study that I pulled this data from where they're looking at how long it took a listener to start responding after the speaker finished speaking, um, across different cultures.

7. 1:44

   And we can see that the Danes take a relatively long amount of time to start speaking after, uh, the other speaker finishes speaking. But the Japanese do it almost instantaneously.

8. 1:56

   So there's differences across cultures. That's part of what makes it hard. There's also differences across individuals. I'm one of those people that takes a long time to respond. Even before I got into voice AI, people would sometimes comment being like, "Are you gonna respond?"

9. 2:08

   I'd be like, "Yeah, yeah. Thinking about it." Um, and, uh, also, like, even though I'm one individual, there's a lot of variability in how quickly I respond. Like, if you make me angry, I'm probably gonna respond kinda quicker.

10. 2:21

    Um, so it's just a, a hard problem. And in the, the next slide I wanna talk about for people who are not very familiar with how voice AI agent pipelines work, I'm gonna provide like a simplified overview of how we handl-handle turn-taking and interruptions in voice AI agents currently.

11. 2:38

    So the user starts speaking. That's the speech input. And that audio, those au-audio chunks are passed to a speech-to-text model. Um, and that speech-to-text model, uh, transcribes the, uh, audio into, uh, a transcription.

12. 2:51

    The next step is something called a VAD that determines whether or not the user has finished speaking. Um, I'm gonna go more into that in a moment. The next step, if the user has finished speaking, the transcript is passed to an LLM and the LLM outputs its chat completion.

13. 3:07

    That chat completion is streamed out and that stream is passed to a text-to-speech model where it's converted into audio. And that audio, which is the audio of now of the voice AI s-agent, is passed back to the user.

14. 3:19

    Um, let's dig more into the voice activity detection system. It's a system with primarily two parts. So it's a machine learning model, a neural network that is detecting whether or not somebody is speaking.

15. 3:33

    So it's like speech or not speech. It's pretty simple, um, machine-- It's not-- I shouldn't call it simple, but it's, it's a really neat model, but it's, it's ultimately just looking at speech or not speech.

16. 3:44

    And then the next thing, next part of it is a silence algorithm. And the silence algorithm is saying, "Okay, if the person hasn't s-spoken for more than half a second, they're done speaking and it's time for the agent to start speaking."

17. 3:59

    So that's the-- In most production voice AI systems, we're using something like that sort of VAD. Um, and that's changing. We're building all sorts of new interesting things that I'll cover later in the presentation.

18. 4:09

    Um, the next part of my presentation, I want to dig into what we can learn from linguistics and academic research about how human-- how turn-taking works in human conversations.

19. 4:19

    And one of the lines I read in a, a paper I read was, that I really liked was that turn-taking in human conversation is a psycholinguistic puzzle, and that we respond in 200 milliseconds, but the process of like finding the words and generating speech and articulating speech takes 600 milliseconds.

20. 4:36

    So how can we possibly be speaking so quickly? How can the listener start returning, uh, an answer so speak-- so quickly when it takes much longer to actually generate the speech?

21. 4:47

    And the answer is that there has to be prediction going on. The listener is predicting when the end of turn is going to occur, and then they're go- they start to generate speech before that end of turn.

22. 4:59

    And what are the primary inputs in creating that prediction? The primary inputs on creating that pred-prediction are the semantic, and this one's the most important one, like what the content is of the, of what the person is saying.

23. 5:10

    Other inputs into this, you know, prediction algorithm in our head of when we're trying to predict when the speaker is gonna finish speaking is the syntax, the structure of the sentence, the prosody, like the, the expressiveness, um, the tone, and then also visual cues.

24. 5:27

    Um, like most things in, uh, how the human mind works, we, we actually don't really know. Like, you know, that's, that's complicated. Um, and-- But I, I would say one of the generally accepted models is what I'm gonna walk through now of how turn-taking works in the human minds.

25. 5:44

    It's broken up into three stages. The first stage is semantic prediction. So what the listener is doing, and you'll notice this as you speak to other people at the conference i- if you like are paying attention to your, your thought process, is you're constantly inferring the intended message of the person that's speaking to you.

26. 6:01

    So before they finish speaking, you're kind of figuring out, "Wait, what are they trying to say?" And then they're, you're using what the ... what your, your prediction of what they're trying to say to, um, uh, to then use that information to predict when the end of utterance will occur.

27. 6:17

    And you're not doing this just once. You're doing this multiple times, again and again, right? You're constantly updating this prediction as the speaker keeps going. So that's the first stage.

28. 6:25

    Um, then the, the next stage, once it seems like you have a general idea, your, your prediction is coming true of like when you think the end of utterance will occur, as you start getting closer to that, you start refining that, that endpoint prediction, um, based on both the semantics and the syntax.

29. 6:43

    And then as you start getting really-- As the speaker starts getting really close to the end of turn, the listener finalizes the prediction by using prosody, by using information around like the tone and other acoustic features.

30. 6:55

    Um, so it's three steps: a semantic prediction, a refinement, and a finalizi- uh, a finalization. And one of the things I wanna point out is that the human mind is full duplex.

31. 7:06

    We're both processing input and we're starting to generate output at the same time. And I think that's really nicely described in this figure from this paper. Um, and so on the X-axis, what we have here is time, where the zero millisecond is the end of the speaker's turn.

32. 7:23

    And what these different blocks represent is what, of mental processes that are happening inside the mind of the listener. And you can see well before the end of the turn, there's this whole comprehension, uh, track, um, that's going on where the listener is inferring the intended message of the speaker and making predictions about when they're gonna stop,

33. 7:43

    finish speaking. And at the same time, there's also this production or generation track where the, um, the listener s- is starting to produce what they're going to say.

34. 7:55

    And we're gonna talk more about full duplex models in computer, in silico rather than human minds in a little bit in my presentation. Um, oh, Jordan Dearslay just texted me. [laughs]

35. 8:09

    He, he texted "boo." Like how does he know to boo me? He's not even here. Um,

36. 8:16

    okay. It's, uh, it's all good. We'll keep rolling. Uh, so, um, so let's go back to, uh, um, like let's contrast this really interesting complex process that's going on in the human mind compared to mod- current voice AI systems.

37. 8:31

    You'll see it's just so much more simple, right? It's just speech or not speech. It's looking backwards. It's not making a prediction. Um, it's done in serial. Nothing's happening in parallel.

38. 8:41

    Um, so it's, it's much more simple. And that's part, part of the problem, right, of why these interruptions are happening. Um, [lip smack]

39. 8:50

    so there's, uh, I'm gonna talk through three types of models and the approaches people are using in these three types of models. Um, so the, the prevailing model for building voice AI agents is the cascading model, system of models where you have, um, what we talked about earlier, speech-to-text, VAD, LLM, TTS.

40. 9:09

    And, uh, what we're doing in those is we're augmenting these new... The new approaches to better handling interruptions i- is we're augmenting the VAD with models that look at the semantics, syntax, or prosody.

41. 9:23

    Um, and, uh, I wanna jump into an example of it. Um, I really have too much content for, uh, for my allotted time that I'm looking at down here.

42. 9:35

    Um, but maybe I can just take some time from Jordan. Um, so, uh, [chuckles] let me, let me, um, give an example for one of these semantic-type models that is used to augment VAD.

43. 9:46

    I'm gonna talk about our model at LiveKit. It's a text-based semantic model. So what we're doing is we're taking the last four turns of the conversation as input. So that means that it's the, uh, the voice AI agent's turn, then the user's turn, then the voice AI agent's turn, and then the user's current turn.

44. 10:05

    Those are the inputs into a transformer model. And what we're-- The token that we're predicting, um, because this is an LLM, we're predicting the end of utterance token. And if that end of utterance token, based on the s- the content, right, based on the, the, the context and the semantics of that, that input, um, if the end

45. 10:24

    of utterance token is saying that the end of turn, it hasn't happened yet, then we don't... We extend the silence algorithm part of the VAD and say, "Don't trigger the end of turn.

46. 10:36

    Wait longer." Um, so they work in concert. But that's generally the idea of how it works. I'm gonna walk through a quick demo of h- of how this works in action.

47. 10:45

    I needed to build a demo, but I wasn't-

48. 10:48

    Oh. So in this first part of this demo, what Shane is going to do is, uh, talk to a voice agent that is just using the, the traditional VAD that I was discussing earlier that just looks at speech or not speech.

49. 11:04

    And in the second half of the demo, he's using our semantic end-of-utterance model.

50. 11:12

    100% sure what it was that I should-

51. 11:15

    Please continue. What happened next?

52. 11:16

    I had to, I had to build a demo for-

53. 11:20

    What was the project or what-

54. 11:21

    ... a LiveKit, [chuckles] a LiveKit, uh, turn detection, a turn detection demo where one of the agents, like-

55. 11:33

    Got it. What challenges did you face while building that demo?

56. 11:35

    It just, it just kept on interrupting me. Uh, being interrupted by the agent constantly. That was the worst part, for sure.

57. 11:44

    I understand. How did you overcome that challenge during the demo? [laughs]

58. 11:50

    Well, hopefully, we're gonna overcome the challenge by using the new turn detection model that LiveKit's offering. So let's try that one now instead.

59. 12:03

    Hey, can you interview me about a time when I had to build a demo?

60. 12:10

    Absolutely. Can you share your experience with building a demo?

61. 12:14

    Yeah, definitely. So, I needed to build a demo, but I wasn't,

62. 12:23

    I wasn't 100% sure what it was that I should build. And then I,

63. 12:32

    I was thinking, like, probably the best way to show that would be, uh,

64. 12:38

    would be side by side. [audience applauding]

65. 12:42

    Yeah. Thank you. I'll, I'll, I'll let kn- I'll let Shane know that you all applauded his demo. Uh, he'll appreciate that. Yeah. You can really see it's a night and day difference when you augment the VAD with models that look at the semantics and syntax and prosody.

66. 12:55

    Um, which is a good segue to my next slide. So, there's another type of-- another approach people are taking to augmenting the VAD. W- and what they're doing is not just taking the semantic input, the text-based input, but they're also looking at the audio signal as well and trying to infer things from the acoustic features of the,

67. 13:13

    the dialogue. So they're taking-- Uh, the basic idea is the input is audio tokens, and the output is the probability that the user is finished speaking. Um, Quinn and the Daily team have built their also open-weight smart turn model that, that, uh, is, is this neat combination of a model that is both transformer and looking at acoustic,

68. 13:31

    um, characteristics. Um, and then one of the new things that has just emerged has been, um, Assembly AI dropped their, their speech-to-text, new s- streaming speech-to-text service earlier this week.

69. 13:44

    And their model is really neat in that it emits-- it takes audio in and it emits out both the transcript and a likelihood that, uh, the speaker is finished speaking.

70. 13:55

    Um, so it's one model that's kind of doing both these two things at the same time. Um, and it's also looking at the acoustic features and the semantic features.

71. 14:03

    One of the things I wa-- Qtai ha- also has one that they recently released that's pretty neat. Um, one of the things I wanna note about this, though, is if you're using your, uh, speech-to-text's built-in end-of-utterance model, it's only seeing half the context.

72. 14:15

    It's only seeing what the user is saying. It's not also seeing what the agent is saying, so it doesn't quite have the full picture on the context. Um, but it works remarkably well.

73. 14:24

    These-- All these approaches work remarkably well and are a major step forward from these more traditional VADs, and like, definitely something that if you're building a voice AI agent after th-this conference, you should go, like, implement it.

74. 14:36

    They're pretty easy to implement on the different platforms too. Um, okay. So we often talk about in, uh, uh, speech models or in voice AI that, um, speech models are, uh, are going to save us.

75. 14:52

    Um, speech-to-speech models, audio in and audio out, are going to save us. Um, but actually, if you look at how these models work, the, like, OpenAI's real-time API, they're still using a VAD on the, on the internals.

76. 15:05

    Um, so they're still just looking s- at speech or not speech, or you can al- you can opt to turn on their semantic... They call it semantic VAD, which is kind of a, uh, paradox.

77. 15:15

    It's not the best term. But turning on a semantic model that augments the VAD. Um, so to answer the title of my talk of why ChatGPT Advanced Voice Mode keeps interrupting you, it's because it thinks you're done speaking based on how long it's been since you last said a word, um, or based on what you've, uh, what

78. 15:30

    you've said previously. Um, and it's just not quite cutting it, um, when those interruptions happen. Um,

79. 15:38

    uh, and it's a problem that is not totally solved. I wanna also bring that up too, that like, this is an ongoing problem, um, with all the different approaches.

80. 15:46

    Nothing has perfected it yet. Um, our end of-- LiveKit doesn't-- Although we power the transport, the audio layer transport for, uh, Advanced Voice Mode, um, OpenAI is not using our end-of-utterance model.

81. 16:00

    So the next topic I wanna cover is the, um... [lip smack]

82. 16:04

    I'm running out of time here, but, uh, is full duplex models. These are really neat. So a full du- duplex model is more like a human mind in that it's processing input and generating speech at the same time.

83. 16:14

    Um, and as far as I know, there's not really any commercial applications of these. Um, but they're, they're fundamentally, they're intuitive talkers. They're trained on the raw audio data, and the analogy I like to use is that it's like computer vision.

84. 16:27

    In the early days of computer vision, we were handwriting algorithms to try to recognize a stop sign based on the color and the number of sides on it, et cetera.

85. 16:35

    Um, and it just didn't work very well. But when we started giving the raw image data to the neural network and let the neural network figure it out, all of a sudden it just started working.

86. 16:44

    And, uh, I think it's, uh, and actually that, that, uh, what we learned from computer vision, that really helped us emerge from the AI winter. That was a major kind of, uh, seeding process for where we are now with AI.

87. 16:54

    Um, and the-- it's a similar analogy with full duplex models in that we're handing them the raw audio data, and we're just letting them figure out how turn-taking works rather than trying to handwrite all the rules.

88. 17:04

    Um, but the downside of these models is they're really optimized for, like, being really good at turn-taking, and they're kind of dumb LLMs. They're small models. They're not trained on a lot of data.

89. 17:13

    They can't do instruction following very well. Um, and just to give you a sense of like more specifics of how these models work, let's talk about the Moshi model.

90. 17:22

    Um, what really made it more concrete for me of how this model works is this idea that it is always listening to input, and it's always generating output. And even when it's not its turn to speak, it's emitting natural silence.

91. 17:34

    So it's just basically emitting silence that you can't hear, but it's still always emitting silence. Um, so it's always kind of doing both just like a human is. Um, SyncLLM, which is Meta AI's, uh, full duplex like experimental mode that you can access inside the app, um, is a similar d- full duplex model, or it's also a

92. 17:53

    du- full duplex model. Something neat that I wanna bring up about SyncLLM is they're actually, in the internals of that model, they're forecasting what the user's say- saying about five tokens ahead or 200 milliseconds ahead, which is more closely like what humans are doing, except we're, uh, forecasting a much longer timeframe.

93. 18:12

    And then lastly, my predictions for the future of how we'll sol- solve this problem, uh, is

94. 18:17

    I think full duplex models are neat, but I don't think they're gonna solve the problem. Like, I think we just, for, for real production commercial use cases of voice AI, we need more control.

95. 18:27

    Um, and we need more control over how it says things like brand names. Um, and instead, what I think is gonna happen is we're gonna get smarter and smarter VAD augmentations and faster and faster models in the cascade pipeline, and we're just gonna have more budget to work with to do a good job with this sort of

96. 18:42

    thing. Um, and the reason I think that's true is, like, computers don't do math the same way humans do. They don't have the same conceptual way of thinking about it, and, uh, LLMs think differently than us.

97. 18:53

    And similarly, I wouldn't expect voice AI to use the same mechanisms as the human mind to generate speech and, and to talk. Um, thank you all for your attention.

98. 19:03

    This was fun. Really appreciate it. [audience applauding]

99. 19:07

    We do have some time, so I don't know if you want to take Q&A. Um-

100. 19:12

     I would love to.

101. 19:13

     We could do that. Um, and I could start with the first question. So the demo you showed, there, there wasn't any response at the end, right?

102. 19:24

     Uh, I cut off the demo. It's actually a two-minute demo-

103. 19:26

     Right

104. 19:26

     ... and I only have 18 minutes to speak, so I truncated on-

105. 19:28

     Fair

106. 19:29

     ... the sides.

107. 19:29

     Do-- Because I was like, "Okay, maybe you just turned everything off," and it was an impressive demo. [laughs]

108. 19:35

     Oh, yeah.

109. 19:35

     No interruptions. [laughs]

110. 19:36

     No speaking, yeah. No, it's no interruptions.

111. 19:37

     Do you ha- do you have the end of the demo? Do you wanna show it or...?

112. 19:41

     Um-

113. 19:42

     No, no worries if not

114. 19:43

     ... it's, it's, it's more of the same idea. Like what you could see-

115. 19:45

     Okay

116. 19:45

     ... is that Shane was like, you know, taking his time talking and really st- pausing and thinking, and it wasn't interrupting him. And then when it eventually would find his end of turn based on the context.

117. 19:54

     Cool. Can we find it on your Twitter or...?

118. 19:57

     Yeah. It's, it's on our link, our LiveKit Twitter. Mm-hmm.

119. 19:59

     Awesome. Yeah. So we can look that up on the LiveKit Twitter. Uh, awesome. Yeah, we can take some questions. I saw you had one.

120. 20:08

     Hi. How important are visual cues for turn detection in, in the human context? And are there any, um... Is there any development to kind of replicate that in the, uh, voice AI context as well?

121. 20:23

     Yeah. It's a really neat question of like how important are visual cues and are, are people working on integrating that into the turn-taking, um, intelligence for, uh, avatars and real-time experiences.

122. 20:36

     So visual cues are actually, despite the fact that we are visual animals, um, very, very much so, like visual is the most visceral, like, you know, input for us.

123. 20:47

     Visual cues are actually pretty low down the stack of like, uh, predictors for when it will be end of turn. It really is semantics. That's like one of the main messages that I wanna convey to people from this talk is it's the content of what people are saying is the main thing we're using to predict when they're

124. 21:02

     gonna finish speaking. Um, and then these visual cues are, are... and these other ones are ancillary to it. Um, and I'm sure somebody's working on building something really cool where it's like multimodal and looking at visual cues to look at the e- to infer the end of turn.

125. 21:16

     Um, I'm just haven't seen it yet and can't keep up with all the AI stuff on the internet. Yes.

126. 21:22

     What is the average cost for... What is the average cost for usually a voice, uh, generated call? And then how is-- what is the effect when you try to keep regenerating the response?

127. 21:40

     W- so the question is, what's the average cost for a voice AI call, um, and what is the cost when you keep trying to regenerate the, the response? Um, so I would, I would first say that the-- I think your, your reference to what does it cost to keep trying to regenerate the response is, um,

128. 22:02

     what I-- The way I wanna answer that is actually the thing that's most expensive in the pipeline tends to be the text-to-speech. So there's all these optimizations you can do in the cascade, and if you end up hitting the LLM multiple times within a turn, it's not, it's not all that costly and those sorts of things.

129. 22:17

     Um, there's some really neat calculators online. Because I'm personally not a voice AI agent builder and, uh, those unit economics don't, uh, don't directly affect me, I don't have the numbers off the top of my head, but there's some really nice calculators.

130. 22:29

     It's gonna depend on how long the conversation is and that sort of thing.

131. 22:37

     You had a question?

132. 22:40

     Yeah, thanks for the demo. That was great. Uh, the question is about your new model that you just shown and that blew us away. So, uh, one is like why is ChatGPT not using that model to improve their stuff?

133. 22:52

     And two, is it available for us to use now if I were to build a, uh, voice bot on LiveKit? And, uh, three, maybe during your development of that, one demo is great, uh, do you also do some kind of benchmarking with user to see if, you know, this is like 50% better or something like that?

134. 23:15

     Yes. So the first question is about why isn't OpenAI using our end-of-utterance model. I don't know why they're not. I think that's maybe above my pay grade of this company I just joined four weeks ago.

135. 23:26

     Um- [laughs] ... uh, and the, the second question is like, can you-- is our end-of-utterance model available, um, for use? And so it's really easy on our website to, uh...

136. 23:39

     or it's really easy to follow our quick start on our website and build a voice AI agent that you can talk to. Um, and it's just one more line in our, in the pipeline that you, you build.

137. 23:49

     You just turn on-- You have one more line in there, and you get to use our end-of-utterance model. It's open weight. It's-- You don't have to pay for it.

138. 23:56

     It's just baked in. Um, and our docs show you pretty... I think by default it's in there. Um, and the, the third question, uh, remind me of the third question.

139. 24:06

     I'm sorry.

140. 24:06

     Uh, did you do a benchmark?

141. 24:09

     Ah, yes. Benchmarking. Um, so we have benchmarks where we have our test data set and, you know, the numbers of course look great. Um, uh, [laughs] but I think, uh- I was on a long call with our machine learning team this morning, where we spent a lot of time just talking about, like, how do we get a good

142. 24:26

     dataset for benchmarking it? And it's just, it's just really-- it's a tough problem. Um, and I feel like the industry as a whole doesn't have a good benchmark around turn-taking, um, and that it's something that I'm sure will eventually emerge.

143. 24:42

     Uh, okay. We'd-- we'll do one last question, I think. So there was... Yes, you in the back. Not great to be sitting in the back if you wanna ask questions, but-

144. 24:52

     Sorry. [laughs] Well, thank you, Tom, for the time on, and the presentation. Uh, I got a question related to the backchannel. So how did you tackle the backchannel challenge in the turn-taking detection problem?

145. 25:04

     So first, uh, for a natural conversational AI, the backchannel is the one that cannot be, uh, ignored, and, uh, sometimes it would cause trouble for the voice agent to detect whether it is the w- it is the endpointing.

146. 25:18

     And second, for a typical backchannel like yeah, mm, uh, yes, uh, those words can occur like a backchannel, or it can be started as the, uh, the agent who sh- uh, should be response in the, in the following period.

147. 25:33

     So how the backchannel is handled should be kind of important in this field.

148. 25:40

     So, um, my question was-- or the question was about not when the-- not the case where the AI is interrupting the human, but the case where the human is accidentally interrupting the AI.

149. 25:52

     Um, and I didn't really cover that in my talk. I was mostly focusing on the AI interrupting the human. Um, we don't have... Our approach is simple, like we're just using like the Solero or the, the normal VAD approach of like, if the person is speaking for more than X milliseconds, assume it's not a backchannel and that

150. 26:12

     they're actually trying to interrupt the voice AI. But one of the things we wanna build is another machine learning model that can, like, recognize the difference between whether or not it's a m- backchannel or someone trying to, uh, interrupt the, the voice AI.

151. 26:27

     One quick note on the full duplex models, uh, the Meta AI one can natively backchannel 'cause it's, like, learned from the raw audio data, so when you're talking to it, it'll go, "Mm-hmm.

152. 26:37

     Uh-huh," which is just so neat. Um, and, uh, yeah, it's just a, it's a tough problem, the backchanneling thing.

153. 26:47

     Awesome. Um, yeah, if you have more questions, you can find Tom. Uh, please give another warm, uh, applause for Tom. [clapping]

154. 26:53

     Thank you all. [outro music]