Engineering voice agents: Latency, quality, and scale

Talk transcript

1. 0:00

   [on hold music] All right. Well, folks, thanks for being here.

2. 0:16

   Excited to chat more about how to engineer voice agents, you know, high quality, low latency at scale. First, maybe a little bit about me. My name is Rishabh. Uh, I work at a company called Together AI.

3. 0:26

   I lead the voice AI team there. Uh, prior to Together, I was the co-founder, CEO of a company called Refuel that was acquired by Together last year. But generally, been building AI and machine learning infrastructure for about a decade.

4. 0:38

   Um, for folks who maybe don't know about Together, Together is building the AI native cloud. What that really means is for companies that are looking to train models and need access to reliable compute, or you want to do inference at scale, uh, we're probably a very good fit for you.

5. 0:52

   Uh, we work with... You know, there's a million-plus developers. We closely work with hundreds of companies and, you know, very proud to be working with companies like Cursor and Decagon.

6. 1:01

   Okay, here is the agenda for today. So we're gonna first talk about, you know, why are we talking about voice, although the previous speaker alluded to a lot of interesting things that he's doing with voice.

7. 1:11

   But why does voice really matter? What does it actually take to build voice agents at scale? What are the challenges? We'll talk about this pipeline architecture, which is becoming the dominant way to build agents in production today.

8. 1:22

   Um, we'll go a little-- we'll deep dive a little bit. We'll look at all the components. We'll look at the system and trade-offs. And then finally, we'll chat about maybe what might be the next generation of building voice agents in the coming months and years.

9. 1:34

   Okay, so starting with why voice matters. Well,

10. 1:38

    there's billions of phone calls a year that are still handled by humans. I am pretty sure all of us have the experience of calling customer support, asking about the status of our order, looking to change the reservation.

11. 1:50

    We've also probably had the experience of calling a doctor's office because we've got to g- book an appointment for ourself or a loved one, and pretty much everybody here probably has had the experience of being on hold.

12. 2:00

    Now, it would be amazing for AI agents to be able to handle some of these calls for us. But really, one of the more exciting directions is, frankly, voice is just this brand-new interface to interact with systems and computers.

13. 2:13

    Look, humans, we learn how to talk before we learn how to read, right? So this comes very naturally to us. And obviously, we're seeing this with ChatGPT's advanced voice mode, and we're seeing this with folks who are starting to directly talk to Cursor, talk to Claude Code in order to get their work done.

14. 2:30

    And this is, frankly, just the beginning. And one of the exciting pieces of 2026 is, you know, building these rich, high-quality conversations. This is not the domain of science fiction or research anymore.

15. 2:42

    This is primarily an engineering problem today. Now, why is it hard? Well, there's a few things that you've got to solve first. First, building voice AI and building voice agents, this has to be real-time.

16. 2:55

    You know, when humans are having a conversation, we respond to each other's cues in something like three hundred milliseconds. And so if you're talking to an AI and it's taking more than five hundred milliseconds to respond, you'll start to notice.

17. 3:08

    If it takes a second, if it takes two seconds, people will just hang up. So you've got to get latency down. The next thing that matters is you want it to be a reasonably smart call.

18. 3:19

    You want to get the work, the job done. And so for real-world complex workflows, you know, the instructions are complicated. Uh, there's a lot of ambiguity. Uh, you have to be good at tool calling because that's the way you give agents access to the real world.

19. 3:34

    So you have a baseline level of intelligence that you've got to meet. The third piece that you've got to solve for is the voice has to be natural enough.

20. 3:41

    It has to sound pleasant enough. Um, and this means a lot of different things. It means, you know, can it talk to you in your own language with the right accent, potentially?

21. 3:49

    Uh, can it pronounce your name? Can it deliver the right emotion that is needed for a particular situation? A lot of the things fall into that bucket. And finally, you know, you could stitch together like a nice demo with one person calling, but what happens when you're doing a hundred calls, a thousand calls, ten thousand calls concurrently?

22. 4:08

    Reliability really starts to matter. And this is an and problem. You have to solve every single one of them at the same time, or you're gonna be in a little bit of a trouble.

23. 4:20

    So at least today, the dominant way of building these voice agents is this pipeline architecture or this cascading architecture, which attempts to solve all of the problems that I outlined earlier.

24. 4:32

    Now, there's a few boxes that are going on, but you know, conceptually, it's relatively simple to understand, which is audio chunks from an end user that are being streamed in, potentially to an agent orchestrator, something like a Pipecat or LiveKit or something that is homegrown.

25. 4:46

    And then essentially, this audio is being fed into a speech-to-text system that converts it into text. That's being fed into an LLM that then decides, do I do a tool call?

26. 4:55

    What is the output? Produces text that is then fed into a text-to-speech model, which starts to produce audio chunks that are then streamed back to the end user. So that's a rough architecture.

27. 5:06

    Uh, let's maybe look at like each of the components, uh, one by one, the components that matter here. The first is speech-to-text, you know, very much like the ears of your agent here.

28. 5:17

    Um, the, the performance metrics that matter here, the first is quality, word error rate. Now, you know, depending on use case, the numbers might look different, but state-of-the-art models are typically in the six percent word error rate on open benchmarks.

29. 5:30

    What that really means is the transcript that is produced by your model, comparing it with a reference transcript, that six percent of words have an error in them. Now, you can imagine why this might matter, right?

30. 5:41

    Because if you don't get the transcript right, you don't get somebody's name right, you don't get the name of, let's say, a drug right, essentially, there's no way to fix this.

31. 5:50

    Your LLM will make the-- will carry forward the mistake. The TTS model will carry forward the mistake. So you have to get it right for the important keywords. And then the second metric that matters often, which is latency-driven, is time to complete a transcript.

32. 6:03

    The way to understand this is when somebody completes an utterance, they stop speaking, how many milliseconds does it take for you to complete the transcript and have that be ready for the LLM?

33. 6:13

    And so as an example, for some of the models that we run on Together, where we get sort of consistently P90 of like a hundred milliseconds, which is pretty fast.

34. 6:22

    Aside from just raw performance, there's a few other capabilities that matter. Turn detection, very important, still somewhat unsolved problem. Frankly, that could be a twenty-minute talk in itself. But really the best way to understand this is you've got people who are, you know, they're talking, maybe they pause for a second.

35. 6:38

    But do you actually know, does that pause mean their turn has ended? Are they gonna continue talking? Because really the last thing that you want here is for you to-- for the agent to start, you know, sending audio back and talking at this person even though their turn hasn't ended.

36. 6:53

    We don't enjoy this in human conversations, and we will certainly not enjoy this in AI conversations.

37. 6:59

    Depending on who your customers are, language matters, and so being able to do this for a wide variety of language and getting it right there, it's important. And the final piece that I'll mention, this is somewhat new, is, um, we're sort-- also starting to see architectures, model architectures that are streaming native.

38. 7:16

    Um, a little sidebar, we won't spend too much time on this, but there's an architectural evolution for speech-to-text models that is in progress, which is going from batch models to stream models.

39. 7:26

    Whisper is the sort of canonical model, came out a few years ago. Um, it was trained on thirty-second audio clips. Uh, thirty seconds is way too much. You can't wait thirty seconds to start to do transcription.

40. 7:37

    So people have had to build all sorts of complicated logic around models like Whisper to do chunking and to pad it with silences, and then they make multiple calls, stitch that together to produce the final transcript in st- in streaming mode.

41. 7:50

    But recently, and this is a fairly new model from the NVIDIA team, instead you have the encoder, uh, of the model have two interesting characteristics. The first is it's trained with different amounts of look-ahead time, so it only looks at perhaps eighty millisecond or maybe up to a second of audio instead of thirty seconds.

42. 8:08

    Um, and it's also able to cache these activations so that as you kind of make small steps, uh, in audio frames, you're actually only doing the heavy computation once.

43. 8:17

    So again, uh, just kinda stepping out, but, you know, it's an interesting direction that we're seeing to be able to handle streaming conversations for these voice agent use cases.

44. 8:27

    Okay, so that's speech-to-text. Uh, jumping into the next part of the pipeline, which is LLMs, you know, very much the brains of, of your agent. Um, the performance metric that matters here first and foremost is streaming latency.

45. 8:40

    And so, you know, TTFT is the metric here. And a rough kind of, uh, you know, metric is like it would-- it's usually pretty good if you can get to three hundred milliseconds of TTFT, um, because, you know, you want to start producing tokens, start feeding that into the TTS model as fast as possible.

46. 8:57

    That number, two hundred to three hundred millisecond, has implications for what models you can use. And so a good size model typically ends up being in this eight to thirty billion range.

47. 9:06

    Uh, if you go any bigger, uh, you'll burn through your latency budgets. If you go too small, that has implications for the intelligence of the model and frankly, the tool calling that is needed, which are both pretty critical if you want to build a voice agent that does meaningful stuff in the world.

48. 9:24

    Okay, text-to-speech. This is very much, you know, the voice of your agent. There's a few interesting things on, uh, um, on sort of performance and capabilities. Performance, again, you know, the trend continues.

49. 9:34

    What is the time to first audio, right? As you get a transcript, how long does it take to produce the first audio chunk that can start be, start to be streamed back?

50. 9:43

    And aside from TTFA, um, what does the real-time factor look like? Real-time factor is, and this is generally the case for most TTS models, but what it means is how much, um, how much audio can you produce in certain number of seconds of processing time.

51. 9:58

    So if you can produce ten seconds of audio in five seconds, your, your RDF is point five. And so you typically want that to be less than one so that you're not buffering.

52. 10:07

    Um, quality is one of the hard ones with TTS because, you know, there are some objective measures, but frankly, nothing quite beats listening to audio samples for the models, for the voices that you care about, and getting a feel for is this the right, uh, experience that you want your end customers to have.

53. 10:26

    Some of the other capabilities, you know, it's, you know, naturalness across a different, a number of different voices, being able to pronounce things exactly right, whether it's, you know, customer names, whether it's product names, um, being able to have some amount of control over emotions.

54. 10:42

    And so you might see TTS models that allow you to add these different little tags which says, you know, this is happy or angry or sad, and it's the start, but these models are getting pretty good, uh, at emotional control.

55. 10:53

    And of course, coverage over language continues to matter.

56. 10:57

    Okay, so those are the main components. But just to kind of zoom out a little bit, you know, this is still-- all of these components are part of this larger architecture, which is multiple models, uh, being, being orchestrated.

57. 11:10

    And so the-- there's a few things that we should always kind of keep in mind, which is first, you know, there's a latency and cost budget across these models that we're thinking about.

58. 11:19

    A rough rubric is the LLM is going to take up a majority of it, followed by TTS, followed by speech-to-text, both from a latency and a cost perspective. Um, and so again, just kind of rough, uh, rules to kind of think about.

59. 11:32

    And one piece that we didn't mention, and we'll come back to this in a second, is so far a lot of the, the numbers that we're looking at is just engine latency.

60. 11:40

    How much time does it take the model to produce an output? But actually, when you're calling models that might be sitting in different data centers, there's network latency as well, and that starts to have an impact.

61. 11:50

    But again, we'll come to that in, um, in one more slide.

62. 11:55

    Autoscaling is also, you know, somewhat interesting and tricky to get right for, for agent systems. Of course, you wanna be doing autoscaling to scale up. As demand goes up, scale down.

63. 12:05

    Potentially, you know, night times or weekends, you want to scale down seamlessly. Um- Scaling up, you know, uh, what we've typically seen is people are much more aggressive about scaling up because the last thing you want is, you know, requests to be slowed down or backed up.

64. 12:18

    So you typically might autoscale earlier than you might do with somewhat more asynchronous systems. And scaling down is also tricky because you might actually have these stateful long-lived connections to your models, and so you can't just arbitrarily kill a pod.

65. 12:33

    You might want to wait for conversations to kind of fully finish. So some interesting nuances with autoscaling.

66. 12:39

    And finally, you know, it's like, you know, global deployments are important because you want your models, uh, you want your system to be as close to your end users to shave off latency as much as possible.

67. 12:49

    And of course, you know, if you're building sort of models in Europe or in places where residency matters, you want to be making sure that you have the ability to deploy wherever you, uh, we absolutely need.

68. 13:01

    I know I referenced colocation, um, but here's one way to kind of understand this problem. So the chart on the left-hand side, this is very much, you know, a very optimized, uh, sort of system where you're doing a pretty good job with your speech-to-text and text-to-speech and LLM models, where the engine latency is in exactly the right

69. 13:21

    ballpark. You're doing a hundred to two hundred milliseconds of time to first token or audio. But you might actually end up having your models being sufficiently far away from your agent orchestrator that it's taking seventy-five milliseconds of network latency.

70. 13:36

    Seventy-five milliseconds is really not that much. Like, you know, even in, I think like, you know, U.S. West to Europe would certainly be seventy-five milliseconds, but, you know, depending on networking, it can be much higher as well.

71. 13:48

    And so an interesting kind of direction that we're seeing folks go is, how can you colocate all your models and potentially your agent orchestrator to either be in the same data center or be very, very close to each other?

72. 14:01

    How can you get them literally in the same building? Because that drop from seventy-five milliseconds to five basically gets you a thirty percent reduction in already a fairly optimized voice agent setup, right?

73. 14:13

    So some of these things, um, especially again, with real-time systems, uh, is just pretty important to have like fairly deep observability and, you know, every ten milliseconds matters.

74. 14:24

    Okay. So hopefully that's an interesting picture on sort of this pipeline architecture. Um, but that's not the only way people do it. One of the other kind of directions that is becoming interesting is a pure speech-to-speech model.

75. 14:38

    And so instead of having speech-to-text followed by LLM, followed by text-to-speech, where you're sort of coordinating and orchestrating across a number of different models, um, it's just way simpler if you could have a pure speech-to-speech model that still is responsible for function calling, um, still handles all the complicated instructions, uh, but just a single model doing it.

76. 14:59

    And of course, you know, for folks who've played around with OpenAI's real-time API, they have a single model behind the scenes. Uh, NVIDIA recently launched a model called VoiceChat, again, very similar ideas.

77. 15:10

    The reason why most of these models are not used in production too much is because they h- they still have trouble with instruction following and tool calling. So, you know, the real-world experience often looks like you'll try them, um, and then you'll spend a lot of time just prompt engineering and hoping to kind of fix issues and,

78. 15:27

    you know, eventually move to a pipeline architecture. But as these models get better, which I'm confident they will, um, it has some pretty incredible benefits because suddenly you don't lose anything about the nuances of speech when that speech is getting converted into text.

79. 15:44

    So the model will natively understand what was the tone, what was the emotion, uh, was the user hesitant. That, that stuff will still remain with the model to make the next kind of decision.

80. 15:55

    And this type of model also allows for, um, sort of more full duplex comm-- like communication, which basically means that the model can start producing audio while it's still receiving audio.

81. 16:06

    And this means that, you know, as you're-- as a customer is talking to this model, you can backchannel. You can say, you know, "I see," or "Uh-huh," like similar to what a human conversation would look like.

82. 16:15

    And these models become much better at handling interruptions and margins, which again, with the pipeline architecture, you have to do a lot more complicated engineering workaround. So, um,

83. 16:27

    hopefully this kind of points in the direction of like, you know, what might be coming in the future. But frankly, we have a lot of engineering work ahead for all of these kind of voice interfaces that still have to be built.

84. 16:37

    Um, if you want to learn more about what Together does, you know, uh, here are a couple of links. We're hiring. We also have a booth G1 downstairs. Um, happy to chat more and happy to answer questions. [applause]

85. 16:56

    Um, for voice-to-function calling use cases, what sort of, um, evals do you use? And then what score do you need to get on those evals in order to have something that's good enough for production?

86. 17:08

    Um, that's-- Like the classic answer is like, it depends, y- you know. But, um, I think in terms of evals, like, you know, there's of, of course, like component-by-component evals.

87. 17:18

    So if we're talking about sort of like the pipeline architecture, um, and assuming that speech-to-text is good, text-to-speech is good, and the only thing that we're caring about is sort of function calling or tool calling evals on the LLM, then it's very similar to how one might do evals for tool calling for LLMs broadly.

88. 17:34

    Which is, you know, was the, um, uh, was the tool call correct? Was the output actually parsable? There's a bunch of those. One would imagine that, you know, you'd want the, uh, the tool call structure to at least be very close to a hundred percent.

89. 17:47

    Uh, um, and then the correctness, again, it depends a little bit on what does the use case kind of eventually demand. One of the other things that we are seeing is especially to get around-- to make models better at tool calling, and because we have to stay within that LLM budget of like, you know, the models have

90. 18:04

    to be relatively small, we do see customers fine-tune smaller LLMs with their kind of use case specific data so that they can get tool calling quality to go up while remaining a model that is relatively small

91. 18:21

    Um, sorry.

92. 18:22

    Yeah.

93. 18:23

    When you mentioned about colocation, so in that, what I understand from that is you're using-- you're decreasing the latency, network latency.

94. 18:31

    Yeah. Um, it's literally because the, the machines are closer to each other. So in a-

95. 18:36

    What does that mean in like, uh, if I'm using cloud providers or...

96. 18:39

    Uh, good question. So for example, um, uh, you know, let's say you're using-- let's say you're building a voice agent here in London, right? You have servers here, but you're using perhaps OpenAI's models for the LLM.

97. 18:52

    And now odds are that OpenAI servers might be somewhere in the US. So literally, the data has the-- the network hop has to be from your server here all the way there and back.

98. 19:02

    Uh, instead, if you had-- if you were able to run, let's say, an open source model in the data center that you're running, uh, your voice agent, now it's basically gonna be intra, uh, sort of, uh, d-- like, uh, the data center rather than going, um, let's say, over the Atlantic.

99. 19:17

    That's one way to kind of think about like... And just distance literally kind of like has that big of an impact, aside from other kind of networking related concerns.

100. 19:25

     Cool. Thank you.

101. 19:27

     Yes.

102. 19:28

     Uh, one more question. Um, so we've had, uh, uh, when trying these voice pipelines issue that we had to introduce some guardrailing. So like let's say we have classifier model in between, uh, which just checks that the model is not offering eight percent discount, which is not, uh, which is not, uh, authorized to do.

103. 19:46

     Um, how does it fit into the pipeline-

104. 19:49

     Yeah

105. 19:50

     ... and how you do that without compromising the latency and the, the experience?

106. 19:55

     Yeah, it's a great question. So, uh, you know, the question is like, what if we have other models like guardrail models or other classifiers in the mix? How does this fit into this architecture?

107. 20:04

     Uh, you're absolutely right. Like this is the most kind of basic reference architecture that one might have. But in many production settings, there's actually multiple models that might be in the mix.

108. 20:14

     The guardrail or the classifier, we definitely see people kind of like start to introduce that right before the main LLM as well, because maybe you want to check, is this something that, um, something that goes to an LLM that handles refunds versus something that handles order tracking, perhaps.

109. 20:28

     And so you might have a classifier there. Guardrails at the end of the LLM generation before you produce a response, that also makes sense. And so often this ends up growing as the, um, as the sort of like scope of what you're hoping to achieve grows.

110. 20:42

     Um, and it puts real pressure on sort of like latency concerns and so forth. So, you know, no easy answers except that it becomes, you know, one more or two more components to think about, have very clear sort of guidelines and SLAs on, you know, how much budget can you really kind of associate with them, and then

111. 20:58

     sort of independently scaling them as needed. But unfortunately, no, no easy answers.

112. 21:04

     Yeah. Especially because when you have things like, uh, like the a- the agent already-

113. 21:09

     Yeah

114. 21:09

     ... answered with an answer which was already not what something it should be, but the classifier, the guardrails for example-

115. 21:15

     Yeah

116. 21:15

     ... catches later, you can't take back things that are spoken. You might have to do something, "Hey, sorry, I shouldn't have said that. Uh, I need to revoke that," or something.

117. 21:23

     So this is really a problem I see there.

118. 21:25

     Yeah. I, I think that's spot on. Like catching all of those before the TTS model gets invoked is, is certainly important. Uh, one of the other things, patterns that at least like we've seen is sort of this thinker talker pattern, where you might have a small LLM that is handling, um, all of the conversation.

119. 21:42

     And so as it gets sort of, uh, text from speech-to-text, it produces a response, and the response might look like, "Let me think about it," or, "Let me get back to you."

120. 21:50

     And then it basically issues one big tool-- one tool call to a much bigger model that then has, you know, much, you know, has better instructions, has all the tools associated, maybe more guardrails.

121. 22:01

     And then it produces a much cleaner response that, y- you know, the, the model is much more-- the, the architecture is like, you know, you're much more comfortable with.

122. 22:08

     And so that gets fed into the TTS model, uh, to produce a response. But, you know, in some ways, this is kind of the-- this is the kind of beauty of like all of these architectures is, you know, the components are, you know, you can-- you'll only add components, and so it kind of, you know, pushes more

123. 22:23

     on, um, sort of reliability, having like, you know, sort of detailed kind of observability on every single component.

124. 22:32

     Yes.

125. 22:34

     Hi. Um, I think I, I was just, um, curious about the, the upcoming voice-to-voice, uh, conversations coming out. And I know that this is upcoming, but, um, at the end of the day, all the surrounding infrastructure that we have in the, you know, through Trump, while we have the conversational systems now and now, I guess, with the,

126. 22:53

     um, pipeline approach, uh, how do we-- like how do you do evals in that model and observability? Like we still need to transcribe everything.

127. 23:02

     Yeah. Uh, very good question. So how does sort of observability, logging, evals change? Um, so i- in some ways-- so some parts of it can still remain the same.

128. 23:13

     So sometimes what you might have is, aside from pure speech-to-speech, you might actually have a, a, a transcription model that is running so that you can at least seeing the transcription, uh, at the same time as the audio is being generated.

129. 23:24

     So that gives you some amount of auditability in terms of what audio is coming in and what audio is being produced. But yes, you know, some evals are going to change.

130. 23:33

     There is no, um, there's no sort of real concept of sort of text-to-speech anymore. There's no concept of pure kind of text-to-text here anymore. And so the evals become much more full duplex kind of conversation evals, which is like a much longer conversation and then, you know, evals and metrics, uh, that are sort of focused on that

131. 23:51

     entire conversation.

132. 23:53

     Right. But what I mean is like, is the nature of those models like able to, uh, output the correct respects of the, the conversation in a way that you can evaluate?

133. 24:02

     Is it that native to API or is that something that you do on top?

134. 24:06

     Typically, a lot of the eval stuff would happen on top, on top of the base kind of inference API.

135. 24:12

     I think, I think we're running out of time, but... Okay. Um, well, thank you everybody for the-- for, for being here. [clapping] [upbeat music]