Designing Voice Agents for Real Conversations

Talk transcript

1. 0:03

   Hey, everyone. I'm Chintan, and I have my colleague Daniel with me. We are solutions architect on the AWS APJ StartUp team. So today we are going to talk about something we've been working on for a while, which is turn-taking in voice agents.

2. 0:18

   What I mean by that is, how does the system actually know that you have finished speaking and it's time for the agent to respond? These are all audio engineering problems.

3. 0:27

   They are not, uh, LLM problems because you can have the perfect model, perfect RAG, but the experience still might feel broken if the turn-taking is off. So we are going to walk you through the concepts and, uh, three different approaches to solving this.

4. 0:44

   And Daniel is going to, you know, make it more concrete by running all the three demo for you.

5. 0:56

   So to kick things off, uh, we'll start with the two hundred millisecond constraint, which is basically the physics of why voice is hard. Then we look into the pipeline architecture, what are the components, where turn-taking actually lives.

6. 1:12

   We'll then look at three level of solving this problem, going from simple silence detection all the way up to running your own turn detection model. And then we'll also talk about the latency, the budget, and, uh, production issues that, uh, we have seen.

7. 1:30

   So, but to-- before any of that, I just want to show you the problem statement more clearly because you have the same user, you have the same sentence on the both side, but, uh, the user outcome is very different.

8. 1:44

   Because on the left, when the user says, "I want to fly," and midway they try to correct themselves, the agent just does not notice because it keeps going on for almost two second, while the person is sitting there [chuckles] trying to get a word in.

9. 2:00

   On the right side, the same exact interaction is happening, but the agent is able to capture the interruption in under two hundred milliseconds, and immediately backs out so the user can now speak.

10. 2:13

    Both the scenarios, the LLM was identical. It was the same model, it was the same prompt, but the difference in the user experience is so different, and the difference is purely because of the audio pipeline.

11. 2:26

    And how fast, uh, does it notices, like, someone else is talking and it knows when to shut up. Like, uh, that's the problem we are trying to solve today.

12. 2:37

    So we'll cover this two hundred millisecond constraint because two hundred millisecond is how fast humans switch turns with each other in a conversation. And the implications are pretty brutal because at eight hundred milliseconds, things start to feel off.

13. 2:53

    While at one point five second, your user would just hang up on you because you have, like, your H-- your chat agents might get five seconds to respond and nobody will care.

14. 3:05

    But, uh, with the voice agents, you don't get that luxury. We've seen recently, like, Salesforce was able to paper the pipeline, and they published their results in March twenty-sixth, but even their best measured response time was seven hundred fifty-five milliseconds, so that's like almost four x more than how humans naturally take turns while talking.

15. 3:26

    So the question here becomes: How do we close this gap? And where we cannot close it, like at least how do we get the turn-taking right so the user experience does not feel as bad as the raw numbers, uh, might suggest here?

16. 3:40

    So if you're working with a budget of seven hundred fifty-five millisecond, and every millisecond count, you also need to understand what's actually in the pipeline. Like, where does the time go, and where does this turn-taking fit?

17. 3:54

    So the pipeline itself, you probably might know the main components, the STT, the LLM, the TTS. You have the audio in, text out, text in, audio out. But the piece that's often missing, like, from people, uh, while people are thinking about this, is this voice activity detection, commonly called as VAD.

18. 4:15

    It's a tiny component that sits right at the front, and it is the one that controls turn-taking. Its job is to basically detect whether the user has stopped talking or not.

19. 4:26

    And the two other thing that you set alongside the main pipeline are also very critical for turn-taking. So you have the smart turn detection, which watches the VAD signal, plus the audio features, and then it has to make a decision that, uh, whether should it respond now or should it wait because the user might still be talking.

20. 4:47

    And the interruption handler is triggered when someone barges in. So what it'll do is it'll propagate a flush of downstream, like the TTS would stop, and the LLM generation need to be canceled so that the pipeline is ne-- ready for the new input within about, you know, fifteen millisecond.

21. 5:05

    So, uh, if you remember, I was talking about, like, how there are different levels to, you know, building, uh, this turn-taking in voice agents. So we'll start with, uh, level one.

22. 5:17

    Level one is your Silero VAD. So this is a simple component which you fully own, and like a lot of production system even today, uh, we see are running agents just by using this.

23. 5:30

    So coming to Silero VAD, it's a small three hundred, uh, thousand parameter model. It takes in like a short-time Fourier transform. It has a short-- And it takes raw audio, and it'll convert it into spectral features.

24. 5:46

    It has four convolutional layers, uh, to pick up the pattern, an LSTM that gives it memory, so across the frame, so it's just not looking at one chunk in isolation.

25. 5:57

    And then it has a sigmoid that gives a probability of speech, and it's like a very small two-megabyte model.

26. 6:05

    And the one parameter, that minimum silence millisecond, is basically the entire user experience of level one. If you keep it very low, the agent is gonna be snappy. Like, it will cut people off while they're still, you know, thinking.

27. 6:19

    But if you keep it very high, the agent might become super patient, and it'll never interrupt, but you might get to, uh, get, like, a dead air where people will wonder if the agent is still connected or not.

28. 6:32

    So there's, like, no universally right answer. It depends on the domain. If, uh, if you're working on it from a perspective of a sales agent, you probably want it to be, like, two hundred millisecond.

29. 6:44

    If you're in a domain where you need some-- to give some time to your user to respond, it could be, like, a one-- a thousand to twelve hundred millisecond.

30. 6:53

    It depends entirely on what you're building. So VAD works well. Like, that's why a lot of people are using it, and it's generally good at what it does. That is to return fast response to the basic question of whether there's someone talking right now.

31. 7:10

    But, uh, there are things, like, it was never designed to handle because there'll be situation where just knowing that is there silence will not be enough. Because you also need to answer these questions of how long to wait, like, uh, the silence duration.

32. 7:26

    If someone is pausing for three hundred millisecond or for two hundred milliseconds, uh, VAD is seeing the same exact thing. It has no idea, like, whether the person is, uh, catching up their breath or whether they've, uh, completely finished their thought.

33. 7:41

    So, like, they, they cannot distinguish that. And the second scenario is, like, what if the, you know, the user talks over the agent? Like, uh, it barges in. It does not know how to handle it.

34. 7:54

    And, uh, the same silence can have a different intent, and it can be, like, completely different things. It could be, like, a completed sentence, or it could be an incompleted thought.

35. 8:05

    It could be, like, a thinking pause, or maybe it could be something as basic as, like, a backchannel acknowledgement. Uh, the situation of the silence is same, but the intent, uh, again, was very different in all the scenarios.

36. 8:19

    But VAD considers all four equally, and it literally cannot tell them apart.

37. 8:27

    So those are the limitation of level one. And with the level two, basically what you get is that you let the STT service tell you, like, when the turn is over.

38. 8:39

    So Cartesia Ink-2 does, uh, turn detection right inside their STT WebSocket. So when they stream your audio in, the server would handle both the transcription and the turn detection together, and, uh, it will emit event when it thinks, like, the turn is done.

39. 8:57

    Similarly, Deepgram does the same thing with their-- what they call is their, I think, at the endpointing. And their P50 latency for Cartesia Ink-2 is about three hundred, and it's about two fifty for Deepgram Nova 3.

40. 9:10

    Both of these are also working really well. Like, we see a lot of customers use that because they are using, like, full audio signals plus some linguistic context to make these decisions, so which is, uh, like, much more information than your VAD will ever have.

41. 9:27

    However, the trade-off is with the transparency because when it is working, it works great. But when it misfires or when it cuts someone off at the wrong moment, you, like, have no way to figure out because there's no log that will say, uh, it was done because of this reason, because of the-- what was-- what it saw.

42. 9:47

    And that decision was made inside someone else's server, and you just gotta live with that, basically.

43. 9:55

    So that brings us to level three. So in level three, you keep Silero VAD running locally for the basic signal that is there a speech, and you add Smart Turn on top of it, which is a small model that runs during silence.

44. 10:12

    So Smart Turn v3.2 is the latest model, like, and, uh, let me share some numbers with you. And it has about fifty-eight point nine percent recall and, uh, sixty-eight point four percent precision.

45. 10:26

    So what this means is that about six out of ten time when someone finishes a sentence, Smart Turn will be able to catch it quickly. But the other four times it might not be confident enough.

46. 10:37

    But, uh, that's also fine because you still have your VAD timer, which is running underneath as a safety net. If the Smart Turn doesn't fire, your stop second will kick in, like, at the configured latency anyway.

47. 10:50

    So it could be, like, three hundred millisecond.

48. 10:54

    So in that scenario, you will never stop waiting. You just, like, get the fast respond when the model is confident, and you get a slightly slower, but a safer response when it's not.

49. 11:05

    The other number on this list is from, like, a Meta's, uh, published paper earlier this year in-- around March, where they had reported a high recall at eighty-seven point seven percent.

50. 11:16

    But, uh, they haven't released the code, so you really cannot deploy it.

51. 11:22

    Smart Turn is BSD 2-licensed, and it's a eight-megabyte, uh, small model. You can pip install it today as well. So yeah, like, uh, that works out better for a lot of our customers deciding when to start talking.

52. 11:35

    But, like, what about the other direction? What happens when the user interrupts and, uh, while you are already-- you know, the agent is talking?

53. 11:44

    So on the left side, you can see what happens mechanically. Like, user opens, uh, you-- mouth, the VAD will pick it up in thirty-two milliseconds, and then, then, like, fifteen millisecond, the entire pipeline is flushed.

54. 11:57

    TTS stops, LLM cancels, everything is clean. Like, the user never hears the agent. Uh, you know, that part is taken care by Pipe Cat.

55. 12:06

    But, uh, the important thing is, like, should it have stopped there on-- Because think about a normal conversation when someone says, "Yeah," like, uh, they send an acknowledgment while you speak, you don't stop, right?

56. 12:20

    You know, they're just agreeing. But if you-- they say, like, "Okay, wait, no, like, that was wrong," you have to stop immediately. Same thing here. Like, the red is the-- it's a correction, s-stop on everything.

57. 12:32

    Um, the amber could be just some sort of a filler background noise. Hope you wanna go ahead and keep going, maybe get a bit quieter.

58. 12:41

    Blue is like a signal for cough or a background noise, uh, so you can just ignore it and finish your sentence.

59. 12:48

    So at level three, because you own this classification, you can start a distinction between, like, a real interruption and someone just, uh, you know, going, "Mm." So today, most of these system stop every time, but, uh, this is a piece that's improving.

60. 13:05

    Uh, like, we are seeing a lot of, uh, incremental improvement in this.

61. 13:14

    So by now that you've seen all three levels, and if you look at these three Python files, uh, pipeline is almost identical in all of them. The only thing that differs is how you answer this question, when is the user done talking?

62. 13:31

    In file one, you pass in Silero VAD analyzer, and that says the user had been silent for three hundred milliseconds, so you can assume they are done.

63. 13:41

    In file two, you swap it with the Cartesia turn's, uh, STT service, uh, class, and now the server is gonna tell you when the turn is over.

64. 13:51

    And in file three, you keep your Silero VAD analyzer, but you also add, like, a local smart turn analyzer version three, which is, uh, sort of a small model that runs during your silence.

65. 14:02

    And it's gonna look at the prosody, and it's gonna look at prosody and intonation to decide whether that pause meant I'm done or I'm still thinking.

66. 14:16

    So you have the code-wise the same pipeline, but the configuration changes, which, you know, leads to completely different behavior.

67. 14:26

    So to summarize your three levels, your level one was Silero VAD, so you basically own the silence detection completely. Level two, you let the STT provider handle it for you.

68. 14:38

    In, in most cases, it, uh, it will be smarter, but then you cannot see what, uh, happened inside it.

69. 14:46

    Level three is your VAD plus smart turn, so where we basically own everything, like, uh, you have full portability.

70. 14:55

    And, uh, now coming to the latency piece, because we've been throwing around these numbers, seven hundred and fifty-five milliseconds, one point three seconds, but, uh, let me show you where, uh, the time actually goes in.

71. 15:08

    So this is a breakdown from Quintle, uh, who is a co-founder of Data... Datahead, and is the person who created Pipe Parent. And these are some of the numbers based on their production measurement.

72. 15:19

    So you have the numbers for making encoding, which is about forty milliseconds. Then you have network engine buffer of fifty-two.

73. 15:27

    So these are physics, like you can't really change them much. Then you have your transcription plus, uh, end pointing. That's about three hundred milliseconds. Again, that's your STT doing its thing.

74. 15:41

    LLM time to first byte, and this is generally the dominant bottleneck, because in a typical API setup, you're looking at five hundred to six hundred fifty milliseconds, depending on what model you're calling in from which cloud provider, which region.

75. 15:57

    And then, uh, you have your network out and playback, uh, TTS. That's-- It's gonna take about one twenty to ninety, eighty-five milliseconds.

76. 16:08

    So you are totally looking at roughly eleven hundred to thirteen hundred milliseconds in a standard setup calling cloud APIs. Now, the Quintess team had demonstrated about five hundred, uh, milliseconds total voice-to-voice by co-locating all models in the same GPU cluster,

77. 16:25

    because they were able to eliminate the network hops between the servers.

78. 16:30

    When-- It's sort of an achievable flow because if you're willing to invest in infrastructure, but for most of the developers today calling APIs, you are somewhere in the range of eight hundred to thirteen hundred, uh, milliseconds.

79. 16:43

    And the key insight to note here is that the STT and LLM together eat about two-third of these, uh, latency budget. So these are the really the only two levers that, uh, we have to move this, uh, latency metric, uh, needle in a meaningful manner.

80. 17:03

    So coming to, like, which, uh, LLMs are actually fast enough for this. We benchmark current models specifically for voice and the target, uh, we kept it as under seven hundred milliseconds time to first token, because anything slower will push our total response past where the user is gonna start noticing.

81. 17:23

    So for now, like recording in June 2026, what we've observed Nemotron-3 Ultra gives us a five hundred and twenty-nine p fifty, uh, latency. GPT-4.1 at about five hundred and thirty-six.

82. 17:39

    But also, like what matters more in voice, uh, than p ninety-five...

83. 17:54

    Hmm. So which LLMs are actually fast enough for this? We benchmark current models, especially, uh, for voice and

84. 18:16

    So coming to the, so coming to the LLMs models, which are actually fast enough for the... We benchmark current models, uh, for voice. Uh, so these benchmarks are from June twenty-six, and the target that we kept was under seven hundred millisecond for time to first token, because anything slower would push our total response time past the limit,

85. 18:39

    like where users is gonna start noticing. So for now, the Nemotron-3 Ultra had, uh, five twenty-nine P50.

86. 18:50

    We observed GPT-4.1 at five thirty-six, again P50.

87. 18:55

    But, uh, what matters more in voice than maybe anywhere else is at the P95 tail, because GPT-4.1 was great at P50, but it spikes to one point seven at P95.

88. 19:11

    And it was even worse for Claude Sonnet that it hit over four seconds. And because in a conversation, you cannot average this out, because one slow response and your entire flow is gone.

89. 19:23

    And there's one more dimension that, uh, people often miss, which is the multi-turn drip. Because after fifteen or twenty turn, sometime model starts ignoring parts of the system prompt.

90. 19:35

    They might get too verbose, they go off-spirit. Because in voice, that's fatal. You can't just, uh, dump a wall of text on, uh, someone. If the instruction falling degrades across turn, then you have to do some sort of a context pruning or session resets.

91. 19:56

    So a few production issues that we've seen are, mm, that, uh, false interruptions significantly increase your escalation rate. Because when agents cut people off incorrectly, then users are more likely to request for a human-in-the-loop, uh, support.

92. 20:17

    And a seven hundred fifty-five millisecond, that's still, I think, the best measured voice-to-voice for a cascaded pipeline like the ones which we propose. We are nowhere near the human speed of interaction, which means that the turn-taking also needs to do a lot of heavy lifting to make the experience feel better than the raw numbers it would suggest.

93. 20:40

    And at fifteen eight point nine percent recall on smart turn, that's actually the best deployable turn detection you can get today. Um, because although it still leaves like four out of ten, uh, turns to be filled by a VAD timer, but, uh, this space is going to improve a lot over the next year, few years, and, uh,

94. 21:03

    we'd have better model. And on infrastructure, running all of this in production is also hard, because five systems that all scale differently and, and also fail differently.

95. 21:16

    You can explore Pipecat Cloud for managed hosting, where your focus is on agent, uh, logic, or the AWS, uh, Guidance Reference Architecture for enterprise deployments with all the enterprise guardrails.

96. 21:30

    So yeah, like, uh, uh, that's it from me. Everything, uh, we've discussed, uh, mentioned is linked on the screen, the FO benchmark. Okay, over to you, Daniel.

97. 21:46

    Thanks, Chintan. Hi, everyone. Uh, my name is Daniel Wirjo. I'm also a solutions architect with the AWS Startups team. So I think, um, Chintan walked you through a great overview of some of the key challenges when building voice agents.

98. 21:59

    So what I might do in this session is just to walk you through a hands-on demo, um, just to make the concepts a bit more concrete, and hopefully, um, you can kind of, um, incorporate some of the learnings into your own voice agent pipelines as well.

99. 22:16

    So what we'll be going through is just this Pipecat turn detection demo, which is just a simple repository that we've created. Um, we'll talk through three examples. So one with, um, silence detection, another with, um, built-in turn detection in the speech-to-text model.

100. 22:36

     And then finally, um, we'll go through the open smart turn model as well. So this is really useful when you either want full control of your, um, voice pipeline, just for from a compliance perspective, or if you wanna also fine-tune and customize your own turn detection model base your own-- based on your own data.

101. 22:58

     So yeah, let's get started. So let's just go through the source code of the first example. As you can see here, um, using Silero is pretty easy in Pipecat.

102. 23:10

     All you have to do is import the relevant libraries, and you can configure the parameters as well. Um, I haven't actually, uh, configured any of the parameters, so these are just the default, but you can change the parameters like the confidence to trigger the silence duration before the turn ends, and the minimum audio volume as well.

103. 23:34

     And in terms of the voice, voice pipeline, it's pretty standard. So we've used, um, an STT model, LLM, and also a TTS model here. We've just defined it as the Cartesia model for STT.

104. 23:47

     Same with the Sonic model for TTS. And as you can see, we've used the voice that sounds like a, a British lady. And for the LLM in the voice agent's brain, we have just used the Anthropic Claude Haiku model.

105. 24:00

     So we'll see, um, what that looks like, and I'll spin up this example right now. So,

106. 24:09

     so if you haven't used Pipecat before, one of the nice things is that it can spin up a local prototyping environment, really useful for playing around with your voice agent pipeline.

107. 24:21

     So let me just see if this now works. Okay. So as you can see, it will come up with a, um, local, um, development environment.

108. 24:32

     So what I might do is just connect to the voice agent. But before I do that, I might just mute just in case, um, I'm interrupting the voice agent.

109. 24:42

     It should come up with, like, an opening greeting, and then I'll test it with some, uh, interactions to see how the turn detection is working.

110. 25:01

     Hey there. I'm your travel assistant, and I'm excited to help you plan an amazing trip. So where are you thinking of going? Anywhere in the world you've got your eye on.

111. 25:10

     As you can see, the assistant's, uh, gone ahead to do the opening greeting. And on the right here in the terminal, I've just got, um, some real-time logs just so we can see what's happening.

112. 25:23

     So what I might do now is just I'm gonna pretend that I'm, uh, thinking about my travel destination and see how the voice agent behaves.

113. 25:33

     So I'm thinking of going to, um-

114. 25:41

     I'm all ears. Where are you thinking of heading?

115. 25:44

     As you can see, I haven't really finished my, uh, sentence yet, but the voice agent is already, uh, responding. So you can see how this works in the debug logs.

116. 25:57

     And as you can see, there's an end of turn complete here. And the reason for that is due to the stopSeconds parameter, which is, um, set to, um, three hundred milliseconds it looks like here.

117. 26:10

     So that's why it's gone ahead and ended the turn. Um, what I might do is I might just exit this local server

118. 26:21

     and try the second example. So let's just clear that, and I'll refresh that. Okay, so I'm gonna try the second example here. And in this, uh, example, we'll use a, um, speech-to-text model which has turn detection.

119. 26:39

     So we'll see how the voice agent behaves. Okay.

120. 26:46

     I'm going to try, uh, the same example, so let me just mute myself here.

121. 26:57

     Hey there. I'm your travel assistant, and I'm excited to help you plan an amazing trip. So where are you thinking of heading? Are you dreaming of a beach getaway, a city adventure, mountains, or maybe somewhere completely different?

122. 27:11

     All right, let's see how the model behaves this time.

123. 27:16

     Hi, I'm thinking of going to, um... As you can see, the model is pretty smart.

124. 27:31

     It now can detect the turn. So let's see if we can find out what's going on in the debug logs. So, um, let's see if it emits, uh, some of those turn-taking events here.

125. 27:53

     So as you can see, um, the Cartesia Inc. model does emit some turn events here.

126. 28:10

     And as you can see, it hasn't really completed the turn. It hasn't sort of, um, responded back with any response. But maybe I can unmute myself and then resume and get to the destination.

127. 28:25

     So, Sydney.

128. 28:31

     Oh, Sydney is fantastic. You're gonna have such a great time there. Are you thinking about when you'd like to go? And how long are you planning to stay? That'll help me figure out the best flights and hotels for you.

129. 28:42

     So as you can see, once the model detects that it's a full, complete sentence, it's able to then respond. Now, in practice, you would also incorporate, uh, some silence detection there because as you can see, I was quite silent.

130. 28:56

     Um, I was muted for a while, and you'd probably want the voice agent to respond in practice. Um, having, like, voice, uh, activity detection or silence detection is also useful for, um, interrupting the voice models, um, um, or what you call barge-in as well.

131. 29:14

     So definitely this is just a, a quick demo, but in practice, you would probably combine both the turn detection model and the silence detection. Um, just for the final demo, I might quickly show the Smart Turn model as well and see what that looks like.

132. 29:30

     Um, it will behave very similar to, to this. It's just that, um, the model will be running on your, uh, either your local machine or, um, on a GPU, uh, environment as well if you wanna deploy that into your-- into the cloud.

133. 29:47

     So let me just maybe clear this just to avoid confusion.

134. 29:54

     And then I'll use, um... I'll refresh this as well, and I'll start now the Smart Turn example.

135. 30:01

     I'll refresh this to see if we can... Okay, there you go. It's starting it, and I'm just gonna do the same example, so I'll mute myself again.

136. 30:18

     Hey there. I'm your travel assistant, and I'm excited to help you plan an amazing trip. So where are you thinking about going? Are you dreaming of a beach getaway, exploring a new city, or maybe something else entirely?

137. 30:33

     I'm thinking of going to, um-

138. 30:38

     I'm all ears. Where are you thinking of going?

139. 30:41

     Now let's take a look at the, um, both the silence detection, which is incorporated into this example, and also the turn detection model. Hopefully, there are some, um, debug logs that we can see.

140. 30:55

     As you can see, um, it's not detecting that I've completed the turn. The end of turn state is incomplete, but it did complete it, um, due to the silence that I've, um,

141. 31:07

     I've, um, incurred there. Um, so there, there was some silence and therefore it sort of kept going with the, with the, um, next turn. What I might do is see if I can do a complete sentence and see if, you know, um, it can be detected through the local smart turn model.

142. 31:30

     I'm actually thinking of, um, going to Sydney.

143. 31:36

     Oh, Sydney's fantastic. I'd love to help you plan that trip. So tell me, what time of year are you thinking about going, and how long are you planning to stay?

144. 31:44

     Also, are you traveling solo, with a partner, or with family?

145. 31:50

     Okay, fantastic. So the voice agent was able to respond as soon as I, uh, completed the sentence. But let's see if that was actually intentional or was that, um, something that, um, you know, can be detected through the smart turn model.

146. 32:06

     So let's have a look here to see if we can find the smart turn, um, debug logs.

147. 32:21

     As you can see, this line here, um, sorry about the legibility of the highlight, but effectively this here says that, um, the end of turn state in, is complete, and the smart turn model is actually classifying that there is a high probability there that, um, the sentence is complete and therefore it's okay to go to the next

148. 32:42

     turn. So that's, um, in a nutshell how, um, turn detection works. Hope that was helpful.