Building & Scaling an AI Agent Swarm of low latency real time voice bots!

Talk transcript

1. 0:00

   [upbeat music] Hey, everybody.

2. 0:16

   Yeah, so we're gonna get started here. Uh, thanks everybody for coming along, um, and thanks to AI Engineers for, for having us. Uh, my name is Damien Murphy. I'm a senior applied engineer at Deepgram.

3. 0:27

   Uh, a lot of people ask me, "What is an applied engineer?" Uh, it's a customer-facing engineer, right? So we work directly with customers, uh, to basically help them achieve their business cases.

4. 0:37

   Um, yeah, about 20 years of experience, full stack developer. Uh, been working in pre-sales, post-sales, uh, really just customer-facing roles for the last 10 years or so. Uh, I've helped hundreds of companies, uh, working with Deepgram to actually build, uh, and scale low latency real-time voice bots.

5. 0:54

   Uh, a lot has obviously changed in the past few, few years around how we actually build these voice bots, uh, and I'm gonna kind of walk you through that evolution as well.

6. 1:03

   Uh, if you're not familiar with Deepgram, uh, we're a foundational AI company. That means we build our own models, we label our own data, and basically train and deploy at scale.

7. 1:15

   Um, we have a lot of models. We have over 1,000 models running in production, so we run a lot of custom models, um, for different use cases, right? So you can think of meeting use cases, drive-through use cases, phone call use cases.

8. 1:27

   Uh, the vast majority of audio today is generated in a call center, and call center data is probably the worst audio you've ever heard, right? It's, you know, 8K, mulaw or linear 16, and, uh, it's just very hard to actually understand for an AI model.

9. 1:43

   So by us building and training models specifically targeted to that low quality audio, we're able to have a, uh, a much better performance. Um, we're research led, so, you know, about 70% of the company is research and engineering, uh, and that means that we, we really focus on, you know, building that foundational, scalable, and cost-effective AI solutions.

10. 2:04

    Um, you know, building a model is easy. Rolling it out into production at a, at a low price point is hard.

11. 2:11

    Yeah, so what are we gonna learn today? Uh, we're gonna build a voice-to-voice AI agent. That means you're literally just gonna send audio data in, and you're gonna get audio data back, right?

12. 2:22

    So you don't, you don't need to hook in, you know, the LLM, the speech-to-text, and the text-to-speech. Uh, you're gonna be able to basically do audio in, audio out.

13. 2:31

    Uh, we're gonna build a simple backend API for the AI agent, uh, that's gonna enable us to, you know, build a front end that shows what's happening, uh, and also, uh, allow the LLM to do function calling.

14. 2:43

    Um, and then we're gonna talk a little bit about how you could scale, you know, these AI agent swarms.

15. 2:50

    Yeah, a couple of prerequisites. I'm gonna get into those first, just so that people have time to kind of, uh, install any tooling that they need. Um, I'll help you understand, you know, the kind of evolution of AI voice bots, right?

16. 3:02

    So how they were previously, how they're moving today. Uh, I'll help you get set up. Uh, we'll go over a little bit of application architecture, how it works, and then we'll, we'll touch on, uh, scaling it as well.

17. 3:15

    Yeah, so you're gonna wanna go to deepgram.com. Uh, sign up for an account. Uh, it's a free account. You get $200 in credit. Uh, that's about 750 hours of, of transcription for free.

18. 3:27

    Um, you'll need Node.js installed on your machine. Um, that will allow you to run a little HTTP server, and if you wanna modify the back end and run your own back end, um, you can also do that.

19. 3:39

    Uh, we recommend Chrome browser. Uh, haven't tested on other browsers, so if you have Chrome, great. Uh, it should work in other browsers, but you, you just never know with browsers these days.

20. 3:49

    Um, you're gonna need a microphone and a speaker or headphones. Uh, you're gonna be talking to this AI agent, so, um, should work fine on a laptop with a, with a speaker on.

21. 3:59

    But yeah, just keep the volume down a little bit so it's not, it's not communicating with other people's agents.

22. 4:07

    Do we need the API key?

23. 4:09

    Uh, I, I've set it up today that you're not gonna need an LLM API key or a Deepgram API key, just to keep it simple. Um, but, uh, after this, you'll have the opportunity to, uh, sign up to the wait list, where you will need that API key.

24. 4:26

    Awesome. Yeah, so the current approach, and, and I've been building these sorts of, uh, voice bots at scale for quite some time, um, and typically they revolve around three key pieces, right?

25. 4:38

    So you've got your speech-to-text. Uh, that's gonna take your audio, give you back, uh, a transcript. And then you've got an LLM. That's going to take that text, uh, that you've detected, uh, process it, and then generate a text reply.

26. 4:53

    Uh, and then you're gonna use text-to-speech to actually speak that back. Um, this has been around for a while, right? It's gotten better and better and lower and lower latency.

27. 5:02

    Uh, we can really bring down the latency on, on both the speech-to-text and the text-to-speech, uh, especially if you run it self-hosted. Um, so one of the things I help a lot of customers with is, you know, co-locating all of these pieces together to bring the latency down.

28. 5:17

    Uh, the challenge with that is it becomes a infrastructure challenge rather than a software challenge. Um, so what we've tried to do, uh, with our new voice agent API is actually offer all of that as a single API, right?

29. 5:30

    So you can send us audio, we'll co-locate all of those services together, um, and then we'll be able to send you back, uh, the response. Uh, we also handle a lot of the complexity of, you know, end pointing, when has the u- user finished speaking, um, a- and those can be challenging.

30. 5:47

    Um, so being able to have all of that in a single API really just makes it easy for the developer to, to build the application that they want to actually achieve.

31. 5:57

    Um, one of the things you'll notice here as well is function calling. So depending on the LLM you use, right, um, you may want to shift your entire infrastructure to a different provider, right?

32. 6:06

    So if you're using Claude, you'll probably go with something like AWS. If you're using OpenAI, you might go with Azure. Uh, you want the LLM and the other pieces of the puzzle, uh, to be co-located.

33. 6:17

    Um, if you're running your own local LLM, right, so Llama 3, uh, Phi, things like that, um, you can just put them really anywhere you want.

34. 6:26

    Uh, and you can see here as well the, the time difference, right? So going from 500 milliseconds for the speech-to-text to 700 milliseconds for the LLM and, and TTS.

35. 6:38

    Um, and you can really bring that all down, right? So a- as you bring those latencies down, you actually start to respond too fast, right? And that's when we start adding in delays and things like that to make sure that you're not, you know, being rude.

36. 6:52

    Um... [laughs] Uh.

37. 6:55

    Hey, excuse me.

38. 6:56

    Yeah?

39. 6:56

    I see a lot of people taking photos. Do you happen to have the slides uploaded somewhere?

40. 7:01

    Uh, I don't have them uploaded, but after the event, I, I can talk to the organizers and get them shared.

41. 7:07

    All righty. So yeah, let's see if the demo gods are with me today and the audio works.

42. 7:22

    Hello? Nope. Gotta allow my mic. Let's try that one more time.

43. 7:35

    Hello?

44. 7:37

    Hi there. Welcome to the Krusty Krab drive-through. What can I get started for you today?

45. 7:42

    Yeah. Could I get a Krabby Patty, please?

46. 7:46

    Sure thing. Would-

47. 7:49

    I think it might be picking itself up

48. 7:50

    ... would you like to... Would you like to go ahead and add anything else to your order?

49. 7:57

    Yeah. Can I get a kelp shake as well, please?

50. 8:04

    I've added a Krabby Patty and a kelp shake to your order. Anything else?

51. 8:09

    Yeah. Could I get a Krus- Krusty Combo as well?

52. 8:13

    We don't have a Krus- Crispy Combo on the menu. Would you like a Krusty Combo instead?

53. 8:19

    Yeah. [laughs]

54. 8:25

    I've added the Krusty-

55. 8:26

    Yeah, so you can... And let me just refresh this so it's not picking me up. Um, so you can see, right, we- we've basically just sent audio to the service.

56. 8:34

    It's gone off and figured out what to do function calling wise. So the LLM is actually making the order. Um, uh, I'm not doing any sort of order making within the app.

57. 8:45

    I'm not parsing the LLM's response. Everything has just happened, uh, automatically. Um, cool. So let's jump back into the slides.

58. 8:56

    Right. So if you have Node.js installed, um, I have two repositories for you, um, which is located here. So github.com/damiandeepgram/deepgram-workshop-client and deepgram-workshop-server.

59. 9:15

    Um, the server itself, um, I'm actually running it on glitch.me. Um, if you want to make changes to the server, you're gonna need it publicly accessible, right? So in order for the LLM to be able to reach out to it, um, you, you'll need that publicly accessible.

60. 9:31

    Um, that's really a stretch goal. You don't need to modify the back end. Uh, the back end I have running should, should be able to handle everything. Uh, if you do wanna make modifications, and we'll go through that a little bit later, um, you can basically spin it up yourself, uh, you know, point the LLM to that

61. 9:46

    new API, um, and it'll be able to then call those functions. Um, hands up, anybody already got Node.js installed?

62. 9:57

    Okay. We've got a good few. Uh, anybody having trouble?

63. 10:03

    All good? Cool. Yeah. So, so once you have that set up, um, you can simply, uh, run the workshop client. Um, so the workshop client is vanilla JS, right?

64. 10:14

    I tried to keep it as simple as possible. Uh, simple HTML, um, uh, page, and then the main.js is, um, where we open that WebSocket connection. Uh, we capture the audio, and then we send it.

65. 10:29

    So, you know, this, this can send audio at a pretty fast rate. Um, if you were running this with a telephone system, uh, you could probably send audio at, like, 20 millisecond chunks.

66. 10:41

    Uh, that's gonna give you the lowest latency. Uh, the browser tends to send larger chunks. Um, so yeah, you can, you can definitely bring the latency down when you increase that chunk rate.

67. 10:50

    Uh, we, we can't process audio faster than you, you send it to us unfortunately. Um, within the config... And, and this is really where you're telling, you know, the, the API what it's actually gonna do, who it is, how it's gonna work.

68. 11:05

    Uh, you can see we have a base URL. Um, so that base URL is just pointing to my WebSocket server, or sorry, my API server. And, um, we have some input parameters, right?

69. 11:16

    So we're sending linear 16 audio. And, uh, down here at the drive-through, uh, speech-to-speech config, um, this is telling, you know, the system, "Okay, I wanna use OpenAI. I wanna use GPT-4.0."

70. 11:31

    These are the instructions, right? So, you know, simple, uh, system prompt. Um, and we have a function call here as well. Let me just bring this down a bit.

71. 11:39

    Yeah. So this function call is telling the system, "Hey, if you wanna add an item to the order, this is the API you're gonna call," right? Um, there's gonna be a call ID.

72. 11:50

    So, you know, when the system starts up, it, you know, generates a unique call ID so that all your order items can go into your particular order. Um, and yeah, it's pretty straightforward.

73. 12:01

    Um, later on we'll actually look at how we can add more of these. Um, and I'll just go back to the slides here.

74. 12:09

    Oh, yeah. So it was... Just gonna talk at a high level. Uh, I probably should have done this before jumping into the code. But, um, we have a user.

75. 12:15

    We have a browser. The browser is generating microphone data, um, and we're sending that microphone data to the service. Uh, so the voice agent API wraps all three pieces together.

76. 12:27

    Uh, the LLM can do the function calling, and so can the, the client browser. So, uh, the LLM adds the items automatically, and then the browser is just displaying what items are actually in that, um, in that order.

77. 12:43

    Uh, any questions on that before I move on?

78. 12:46

    Like, get the repo.

79. 12:48

    Oh, yes. You need that again? There you go.

80. 13:01

    Was everybody able to find, uh, the GitHub repos?

81. 13:05

    It's in the Slack. All good. Okay. Nice.

82. 13:10

    If you missed it, it's on the Slack.

83. 13:14

    Thank you. Awesome. Yeah, so, so this is the agent configuration I was showing you earlier. Um, right now the app is pretty basic. It only has an add item, right?

84. 13:27

    So you can't take items away, can't modify items. Um, and the reason we've limited it to that is, you know, that's a challenge for you. Can you add order modification?

85. 13:38

    Um, we've already got the API there, so we have a remove item API and, uh, a few orders. Let me grab... Yeah. So there's... Yeah, these are all the APIs that are in the server code.

86. 13:52

    Um, so if you want to add that order modification, um, you're gonna need to look at, you know, uh, basically removing an item from, uh, the order. I believe this one here, uh, delete call ID order items.

87. 14:08

    Uh, so we've already hooked up the add item, um, but in order to remove items, your LLMs need to be going to understand what's already in the order, and you'll need to use the get order, uh, call.

88. 14:19

    Uh, but we'll get into that a little bit later.

89. 14:23

    Okay. So we support multiple LLMs. Um, so within the API today, you can call, you know, Claude, uh, Llama 3, Mixtral, uh, supported with OpenAI, Anthropic, and Groq. Uh, we will be adding to that as well over time, but those are the kinda initial ones.

90. 14:40

    Um, this API is, is pre-release as well, so you're basically getting a, a sneak peek to it.

91. 14:48

    Yeah, so we have the, the menu here. So the menu is coming from the menu items API. Um, when I wanna create a new call, so on, on l- loading of the webpage, we basically grab a unique call ID, uh, and we get the menu.

92. 15:03

    Um, the menu itself right now is, is baked into the, um, LLM system prompt. Um, stretch goal would be let's turn that into a function call as well, right?

93. 15:14

    Let's allow items to go out of stock, right? Um, the LLM will need to know when you're out of chicken wings and things like that. Um, and then we have the, the get order for the call API as well.

94. 15:26

    Yeah, so the WebSocket client itself, um, if you are familiar with the Chrome DevTools... Let me see if I can grab it here.

95. 15:38

    Yeah, so in the Network tab of the Chrome DevTools, there's a cool little, uh, WebSocket inspector. So if I start a conversation, um, you'll be able to see the messages as they happen, right?

96. 15:50

    So you can see here I'm streaming audio at a pretty rapid rate. Um, and you know, when the API responds, it's, it's gonna send us back, uh, messages as well.

97. 16:00

    So there's, there's quite a few lifecycle messages that we'll send you too, um, and I can, I can bring those up now as well.

98. 16:11

    Yeah, so the sentence configuration, um, that's one that we send. So we tell it, uh... And this is being truncated as well. There's, there's a lot of, like, system stuff and function calling as well in it.

99. 16:22

    Um, but we're basically telling the system what we want to actually send it, what we want it to be, how we want it to work, uh, what functions we want it to call.

100. 16:30

     Uh, and it's gonna send us back this session ID. Um, and then the conversation text, right? So that's gonna give us a transcript of what the user is saying.

101. 16:39

     Um, that can be useful if you wanna display, you know, text on screen while the person is speaking. Uh, sometimes that feedback is, is really useful so people know it's, it's hearing you as you talk.

102. 16:49

     Um, and then, yeah, the function calling. So right now, um, this is only doing add item, and you can see the argument here is basically the, the Krabby Patty.

103. 16:59

     Um, yeah, so the tool will respond, uh, whether it was a success or failure. So, you know, if it can't add the item, you'll know about it. Um, and then you'll also get the assistance, uh, response back in text.

104. 17:14

     Uh, it can be useful to know what the AI is saying if you wanna do content moderation. Um, so you don't want it offering free Krabby Patties, for instance.

105. 17:23

     You might have a, a mechanism to detect, you know, um, various things, and you can apply that to a lot of different use cases as well.

106. 17:32

     Yeah, so the backend API, um, this is the main one that we're gonna be, be using today. Uh, but yeah, I do recommend trying to, to add a few more.

107. 17:41

     Um, and yeah, on the server log, so, you know, in the server where it's running this API, you're gonna see things like this, right? So there's a new call coming in.

108. 17:52

     You've got that order ID. Uh, you're adding an item to that order, uh, and then you're getting, uh, the updated order here as well. Um, and, and this is essentially what, you know, we're consuming in the front end to display the items as they're ordered.

109. 18:08

     Yeah, so walking through the client code, uh, there's five files. I tried to split them out logically as best I could. Um, so let's go through, uh, each of those and just kind of explain what each of them does.

110. 18:20

     Um, the main.js, this is gonna be, you know, all the kind of front-end hookup code. Uh, config.js, that's gonna be how we tell the LLM and, uh, the, the agent API what to do.

111. 18:31

     Uh, the services.js, that's just a, like a, you know, um, CRUD kinda interface to the backend API. Um, audio.js does some kind of interesting stuff around, you know, audio manipulation and downsampling.

112. 18:46

     Uh, the browser itself actually sends higher, um, higher sample rate audio, um, but we don't, we don't need 48-kilohertz audio at that rate. That's gonna be a huge bandwidth, uh, usage.

113. 18:59

     Um, so we drop that down to 16 kilohertz, which is, you know, essentially what the, the API can handle, um, pretty fast anyways. Uh, and then animations.js is really just animating that little bubble that, you know, kinda responds to speech.

114. 19:13

     Um, and then the server code, super simple Express.js API. Um, if you're familiar with Express.js, you know, it, it doesn't really get much more simple than that. [clears throat]

115. 19:23

     Okay, so let's take a look at the code.

116. 19:29

     Yeah, so this is the, uh, the index.js within the server code. Um, it's just got a few very simple, uh, function calls here. So, um,

117. 19:42

     I'm not, not sure how much I should go into this, but yeah, it's, it's pretty, pretty straightforward. It's just updating, uh, the app state, um, and handling some of the CRUD operations.

118. 19:52

     Um, yeah, so let's go through these top to bottom. Uh, the animation stuff, it's just a simple canvas. It's going to, uh, modify the, the bubble. Uh, and I'll show you the bubble again if you forget what it looks like.

119. 20:05

     Um, yeah, so this bubble here i- is gonna respond to speech. So you can see it kind of get bigger, smaller, bigger, smaller, uh, and that, that's pretty much what that does.

120. 20:17

     Um, within audio.js, so we've got a couple of different, uh, functions in here. We've got, uh, receiveAudio and captureAudio, and then clearScheduledAudio and downsample. Um, and this little function is just, uh, a conversion function that the downsample uses.

121. 20:34

     Um, the reason we need a clearScheduledAudio is because you can interrupt the LLM while you're talking, right? Or, or the agent. Um, and we may not know that on the server side because you're handling it on the client side.

122. 20:48

     So if you've got audio already playing, you're gonna wanna pause that audio as soon as you can. Uh, you could even add a client side, uh, voice activity detector.

123. 20:57

     Uh, Solero VAD is a, is a really good one that I've used before. Um, and that just allows you to do that barge-in. So when you do start speaking, you know, it knows to stop, um, you know, and more advanced systems will help the LLM actually understand whereabouts in its speech did it get interrupted, right?

124. 21:15

     Because it then it, it may not know you didn't hear a part, you know, at the end of its prior response.

125. 21:21

     Um, and then, yeah, receiving audio. So this is basically grabbing the audio from the WebSocket, um, and just sticking it into a buffer. And then to capture audio, this is just grabbing it from the, the media devices on the browser.

126. 21:35

     Um, and then once we get that, we, we call the callback, and that callback is, is what we saw here, uh, which is the WebSocket send data. Um, yeah, so w- walking down through this, um, unload, we're going to prepare the agent's config and, and send that over.

127. 21:53

     Uh, that'll give us the order ID. Um, and then when I click s- Start Conversation on the UI, um, that's gonna call this, uh, code here, opens the WebSocket, and begins sending audio data.

128. 22:05

     Uh, the errors, any WebSocket errors will be handled, handled there. Um, and then this is essentially where we're getting back, um, text-based, uh, status messages. So user started speaking, um, you know, like what the AI said.

129. 22:22

     Um, and then here we're actually receiving the audio. Um, so receiving the audio is, is, is, um, what we looked at in the audio.js file. Um, and then we also have the ability to update voices.

130. 22:34

     Um, so I don't think I showed that actually in the prior example. Uh, have a version running here. Let me just kick it off.

131. 22:54

     Hello?

132. 22:58

     Hi there. Welcome to the Krusty Krab drive-thru. What can I get for you today?

133. 23:05

     Yeah. Can I get a, a Krabby Patty, please?

134. 23:10

     Got it, a Krabby Patty. Anything else for you today?

135. 23:14

     Yeah. Can I get a Krabby Meal as well?

136. 23:18

     Sure, a Krabby Meal.

137. 23:21

     Actually, can I change that to two Krabby Patties?

138. 23:27

     I've added an additional Krabby Patty to your order. Anything else you'd like?

139. 23:32

     Yeah. So we, we have 12 different voices. Um, that voice, that second voice I used there is actually my own voice, um, [laughs] which i- which is pretty handy.

140. 23:42

     Yeah, I had to, I had to tell my parents when I trained the voices. Like, "If you ever get a phone call from me looking for money..." [laughs]

141. 23:52

     Krabby, Krabby Patties.

142. 23:53

     Yeah. [laughs] Cool.

143. 23:54

     Uh, excuse me.

144. 23:54

     Oh. I'm still talking to it, am I?

145. 23:59

     Uh, I, I have a question.

146. 24:00

     Oh, yeah. Sorry, go ahead.

147. 24:01

     Um, where in the stack do you categorize the type of function call? So whether that's add item or, uh, remove item or...

148. 24:11

     Mm-hmm.

149. 24:11

     Yeah.

150. 24:12

     Yeah. That's in the config.js. So you can see here we, we tell it what to call, and we give it a, a base URL.

151. 24:24

     Um, so it's not dynamic based on the prompt?

152. 24:29

     Uh, the LLM will dynamically decide what to use.

153. 24:33

     Okay.

154. 24:34

     Mm-hmm.

155. 24:34

     Okay.

156. 24:35

     Yeah. So there, there's no, like, direct calling of, of it. The LLM is kinda, kinda like, you know, ChatGPT plugins, right? You don't really know if it's gonna use it or not.

157. 24:44

     Um, but yeah, they've gotten pretty good, especially GPT-4o and, um, uh, I think, uh, Mistral as well is pretty good at calling it. Uh, you'll probably have problems with GPT-3.5 and function calling.

158. 24:57

     Um, it's just not really up to the level, um, to do it. Um, but yeah, LLMs are getting better, so, uh, they're able to do it now.

159. 25:07

     Thanks.

160. 25:13

     Actually, um... I'm sorry. Putting on top of what you just said, I don't know if anybody-

161. 25:19

     Sorry, what [clears throat] sorry. What I've realized is to have a good system prompt along with a good function definition, that really works with most, uh, newer models-

162. 25:31

     Mm-hmm

163. 25:31

     ... to consistently get it to use the tool or request the actual tool instead of just coming up with its own stuff.

164. 25:38

     Yeah, yeah, definitely. And, like, it, it really is only those newer models, right? So the, the latest Claude, um, I don't know if Haiku's gonna work super great with function calling, but definitely, like, Sonnet and Opus, uh, work a lot better.

165. 25:52

     Um, and then on Groq, um, they host, uh, Llama, um, Llama 370 billion, I believe, and Mis- Mixtral. Um, you kinda... Your mileage may vary with those open source LLMs.

166. 26:05

     I don't think they've caught up to the function calling level just yet. Um, but yeah, like, you know, shoot for where the puck is gonna be, and I think, uh, a lot of those will catch up pretty soon.

167. 26:16

     Yeah?

168. 26:17

     Um, so curious about your take on, uh, the, the UX of the voice. Uh, in particular, um, what do you have in terms of recommendation specifically for interruptibility of-

169. 26:31

     Mm-hmm

170. 26:31

     ... like, for these models, for, for these kinds of interaction. Uh, I think one place where people get tripped up is, well, this is cute, but oftentimes people think while they are saying something, so oftentimes there are, like, these awkward silences in between where the sentence is not fully formed yet, and midway the thought changes and stuff

171. 26:52

     like that. Um, and, and how does that affect the usability? Likewise, the model is saying something, and is there interaction design that you recommend to interrupt the model in between and say, "Okay, this is going in the wrong direction."

172. 27:06

     Mm-hmm.

173. 27:07

     Sorry, I've got two questions but I'll just-

174. 27:09

     Yeah, yeah, so for anybody who didn't hear, uh, the question is about interruptibility and, and how to handle things like long pauses. Um, a- and that really comes down to end pointing and contextual kinda, um, uh, semantic end pointing is what we call it.

175. 27:23

     Uh, so that's something we're gonna build into, uh, this voice agent API. So, you know, you can imagine a scenario where the user says, "Hang on a minute, let me get that for you."

176. 27:32

     Right? You know, say they're going to get their account number or whatever it is, um, that doesn't necessarily require the LLM to go off on another kind of monologue.

177. 27:40

     Um, the LLM might say, "Sure, you know, let me wait for you to get that." Right? Uh, and that's kind of, uh, semantic end pointing. Uh, the other type of end pointing, which is kinda, you know, traditionally what people used was, you know, a span of silence, um, is used to determine when somebody's finished speaking.

178. 27:56

     Um, but yeah, like, if people are calling out credit card numbers, um, it's pretty common for them to do backchanneling. Um, so when you do backchanneling, you're essentially waiting for, like a, a noise from the other person, like a mm-hmm, you know?

179. 28:09

     So if I do, like, you know, "One, two, three, four." "Mm-hmm." "Five, six, seven, eight." "Mm-hmm." And, and that just kinda gives you that, like, I've, I've captured what you said, um, so that you don't go too fast and then the person, like, falls behind.

180. 28:22

     Um, and a lot of the time with these voice agents, what I recommend to customers is, you know, what would a human do, right? Um, and there seems to be this really high expectation that the AI should be able to understand pretty much anything, right?

181. 28:37

     Um, but the, the reality is, is that, like, nobody can understand my email address over the phone. Um, and I have to call it out, like, you know, D for Damien, uh, you know, A for apple, and, and, and this is with a human, but, you know, with an AI, you need to build in that kind of

182. 28:52

     understanding logic. Um, and I'll go into it a little bit later about, you know, how you can make that composability with these agents, um, because you don't wanna create an agent that does everything, you know, for your entire business, right?

183. 29:04

     Uh, you wanna create an agent that's capable at a particular task, and then build them together, right? So having that kind of multi-agent, uh, system where you can offload parts of the conversation to, you know, a slightly different AI agent that's able to collect credit card numbers very accurately and handle all of the edge cases or, you

184. 29:24

     know, verify account information, you know, versus, you know, taking an order. Um, they're all, they're all very different use cases, but, um, yeah, from what I've seen in, in the market, people tend to wanna make it do all the things in one system prompt, um, and it's just not there yet.

185. 29:40

     Y- you know, even with these large context windows, I don't think it's, it's really good to try to get it to do everything and have, you know, um, a massive system prompt.

186. 29:49

     Um, like, as you increase the system prompt length, you also increase your time to first token. Uh, time to first token is, is really the key metric for an LLM, uh, to respond.

187. 30:00

     Um, so you can start responding as soon as you get, you know, let's say five tokens or 10 tokens. Uh, you can start the TTS playback at that point.

188. 30:08

     Um, if you wait until, like, you know, the 250th token, uh, the latency is gonna be much higher, right? Maybe if you're using Groq, you could wait, uh, that long because it's so fast, but yeah, mo- most LLMs are outputting, you know, maybe 30, 30 tokens per second.

189. 30:23

     Uh, and it's highly variable, right? Like, uh, even GPT 4o can give you, like, 900 millisecond, uh, latency on first token. Um, and you know, that, that's something that's gonna improve over time, but yeah, it's definitely something you have to be aware of when you're building these voice bots.

190. 30:40

     Yeah?

191. 30:42

     So suppose that, you know, I've got this use case, and I'm looking at either using ChatGPT with plugins where I can also give it capabilities for prompting, give it capabilities for calling APIs, versus I'd like to use, you know, more of a bespoke solution.

192. 30:59

     In your experience, what are some of the, the things to consider? You know, do I want, like, a plug and play thing? And also, like, you know, what are the more nuanced maybe ways to cater to a specific use case like you were mentioning, you know, with, like, the multi-agent approach?

193. 31:16

     Mm-hmm. A- a- and just to clarify, so, um, is the question about using this approach versus which other approach, sorry?

194. 31:24

     Yeah, like-

195. 31:29

     The functionality of calling APIs to ask the wrong thing.

196. 31:33

     Mm-hmm. Yeah, so I'm not sure I fully understand the question, but, um, let, let me kind of paraphrase it. So using a ChatGPT, um, today doesn't, doesn't have ears or a mouth, right?

197. 31:46

     So, so you, you just have an LLM text in, text out. So you still have to add the ears and the mouth. Um, and what we're doing here is, is real-time low latency streaming.

198. 31:55

     So the audio is being streamed like, you know, uh, straight into the system, and then audio is being streamed straight out of the system. Um, you know, obviously GPD 4o had that big fanfare announcement the day before Google's announcement, but neither of them have released anything yet.

199. 32:12

     Um, and the reason that they haven't released anything yet is that it's hard, right? Um, we've had real-time voice agents, you know, for, for years and, uh, they've just gotten better and better and better, and, and one of the key things there is the latency from, you know, end of speech to transcript.

200. 32:29

     Um, once you go self-hosted with Deepgram, you can get that down to like fifty milliseconds. Um, in our hosted API, you're gonna get closer to half a second. Uh, and that's just because we, we don't like crank up the compute.

201. 32:41

     Um, so as you increase the compute, like say to 5X, you can get down to that fifty milliseconds. So a lot of these, you know, companies that you see showing real-time voice bots, um, they're using Deepgram under the hood.

202. 32:54

     Um, you know, today I think we're the only option for low latency real-time, uh, speech recognition. Um, that may change in the future, but yeah, today this is kind of state-of-the-art I think.

203. 33:06

     Does that answer your question?

204. 33:08

     Yeah. I'm wondering like what are your thoughts on, you know, with like ChatGPT, right, uh, you can create your own plugins and these... enable these plugins to make API calls.

205. 33:18

     Would you say like that is kind of similar to a part of the workflow that you're showing here?

206. 33:24

     Y- yes. So we're LLM agnostic, so y- you can use function calling with a lot of LLMs. Um, like building a GPT assistant, um, kind of follows a very similar API format.

207. 33:36

     It's a pretty standard OpenAI kind of, uh, interface and, and most LLMs have actually adopted that same interface. So, you know, you can use that same function calling and system prompt with another LLM.

208. 33:49

     Um, so, um, yeah, I, I think that's definitely interchangeable, but there, there's no real difference between a GPT agent and what we're doing here.

209. 33:58

     Yeah.

210. 33:59

     Um, suppose the function call that you're making is like a... is going to be long running. Um, is that blocking to the voice agent? And like, I guess, what are some ways around it, um, if the function call that you wanna make is something that's long running?

211. 34:14

     Mm-hmm. Yeah. So the question was about, uh, long running function calling, uh, and that's definitely a concern. Um, I don't know if you want to do long running function calling with a, a real-time voice bot.

212. 34:25

     Uh, you might hand that off to, you know, a, a secondary system. Um, so you say, "Hey, okay, you know, I'm checking on that for you. Is there anything else can help you with?"

213. 34:33

     And then when, when it comes back, your agent can then offer up the, the information.

214. 34:38

     Is, is this, um, like are the voice agent function calls like pull only? So like in the context of this, uh, demo, there's like a get order function, so like it first adds things to the order, and then like looks at the order.

215. 34:53

     But is there a way to proactively push things to the conversation window-

216. 34:57

     Mm-hmm

217. 34:57

     ... um, as, as part of, as part of this API?

218. 35:00

     Yeah. Yeah. So we're pushing the order items from the LLM. So the LLM is actually making the call to the, uh, add order or add item to order API.

219. 35:10

     Um, so it's doing the pushing, right? I, I'm not pushing anything client side. I'm only reading client side. So client side I'm just polling the order as like, you know, "Give me the order.

220. 35:19

     Give me the order. Give me the order." Um, and that's able to allow me to display the order. Um, but the actual pushing, it's happening all from the LLM.

221. 35:27

     I see. So with respect to like, um, like if we need to add information about the order to the LLM, that is in and of itself a function call-

222. 35:36

     Mm-hmm

223. 35:36

     ... on the part of the LLM to, to pull from the API. Is that correct?

224. 35:39

     Yeah. So what you would wanna do is you would wanna give it a new function. Um, and, and we've already created the functions, uh, to give it, and that's kind of, uh, the next step in this.

225. 35:48

     So you would want to add, uh, a new function here, uh, for get order, right?

226. 35:54

     So this would be like your get order function, and then you can point that to, uh, the API to get the order. Uh, another one you'll probably wanna do is, is get menu, right?

227. 36:05

     So, you know, is, is there an item that's no longer available, right? Because we're pulling from the, you know, the menu ordering system to see if something's out of stock, um, because we know all the orders that have gone through.

228. 36:16

     Uh, and then another one you'll want is actually, uh, remove item. So with remove item, you have the ability to modify the existing order. Um, I didn't implement these in the function calls, 'cause I f- I thought that would be a good kind of learning exercise for people here.

229. 36:30

     Um, but yeah, you could definitely add those and, and, uh, understand a little bit more how to work.

230. 36:35

     Got it. And, um, if, if we did want to have a long running function that ran, um, it would be a non-blocking function call. It triggers a job, and then at some later point the assistant attempts to fetch that information.

231. 36:51

     Is there a way for a client to actually push data into the conversation log, like in... with this API? Is that, is that a possibility?

232. 36:58

     Yeah. Yeah, absolutely. So, um, as a part of the, you know, the, uh, the information that the LLM has access to, um... and I, I don't know if you can see it here.

233. 37:08

     It might be off screen. Uh, you can see here the menu, right? So the menu there is a part of its system prompt. Um, but you could remove that menu from there and add it as a function call.

234. 37:19

     So now anything that modif- Like, you could have a separate service modifying the menu, and it could be a separate LLM, right? And when that menu is modified and it pulls the menu, it's now updated its system context.

235. 37:36

     Yeah?

236. 37:36

     Can I ask you a follow-up just to, just to that?

237. 37:39

     Yeah, sure.

238. 37:40

     Is it... Looking at the workflow as to the function, um, have you had any cases where the agent goes, "Oh, by the way, this long running function call has finished in the background.

239. 37:52

     Would you like me to get more information on it?"

240. 37:56

     Um, I haven't tried it myself, but I, I'd probably imagine you'd want some sort of web hook callback. Um, so when the function call is complete, it would, it would instantly return, but then have a separate handler that would, would know that it's, uh, completed.

241. 38:13

     Um, and then you can prompt the LLM w- from that, uh, web hook handler to say, "Hey, you know, this thing you asked for earlier..." And it would kind of act like a user, um, input as well.

242. 38:24

     As that would... You'd see that as a function call within the third content of JS and it's-

243. 38:29

     Uh, the, the web hook handler would probably run in the back end, not in the LLM itself. So the, the LLM would just say, "Hey, go do this long running task," instantly return and say, "Okay, I've kicked off that long running task."

244. 38:42

     And then when the web hook handler gets fired by the long running task, it would then tell the LLM, "Hey, you know, this long running task has completed." Yeah.

245. 38:52

     And I think you had a question as well, yeah?

246. 38:54

     On the interrupt handling side, how do you differentiate, uh, between a, a noise versus a action speech?

247. 39:01

     Mm-hmm.

248. 39:02

     You know?

249. 39:02

     Yeah. So we, we have voice activity detectors. Uh, and the voice activity detector will only trigger on, um, audio that's generated by the vocal cords. Uh, it will trigger on coughs and humming and things like that.

250. 39:15

     Um, but you may want that to happen as well. Um, so you can put in things in place to detect, okay, did I actually transcribe a word? Um, you know, should I respond to this?

251. 39:25

     Um, so those are things you can implement as well.

252. 39:27

     So on that, will it added latency when I'm speaking, uh, on the phones, and if I interrupt-

253. 39:34

     Mm-hmm

254. 39:35

     ... uh, while you have a audio going on, we'll have to wait for the response?

255. 39:40

     Yeah. So we- we'll send you a very quick, uh, user started speaking event using a server side, uh, VAD. Um, so the voice activity detector's gonna tell you as soon as you get it.

256. 39:51

     Uh, and we actually have code in there as well to, uh, to clear, I believe... Let me see. Where is it? Uh, in here.

257. 40:01

     Yeah. So you see if user started speaking happens, we basically stop the audio playback.

258. 40:09

     Okay.

259. 40:09

     Mm-hmm.

260. 40:09

     What's the latency on that?

261. 40:11

     Uh, it can vary. I, I think it should be in the order of, like, less than 100 milliseconds.

262. 40:16

     Okay.

263. 40:16

     Mm-hmm. Yeah?

264. 40:17

     Is memory being handled at all, or is that just something separate?

265. 40:20

     Uh, s- memory would be kind of like a separate, uh, challenge. Y- you can obviously build up the system context. Um, you know, depending on your use case, like if you wanna handle hour long calls, you probably don't wanna keep building up the system context.

266. 40:35

     You'll wanna use some sort of, uh, memory system. Uh, AutoGen have a pretty good teachable agent, um, if you're, if you're familiar with it. Uh, it has the ability to run a secondary LLM to ask, "Is there anything new or updated, you know, in this new content?"

267. 40:50

     Update, you know, the existing memory. Um, a good example of that might be like, "Oh, I live at 123 Street," and then it comes down later on and it's like, "Oh, actually I live at 456 Street."

268. 41:00

     Right? And you don't want both conflicting in your system prompt. You wanna update, uh, the prior memory of it.

269. 41:08

     Yeah?

270. 41:09

     How do you protect against, uh, code injection or rogue, rogue content into the system?

271. 41:17

     Yeah. And, and that's really the, uh, the reason that we have, um... Let me see this one here. Um, this is why we have the conversation text. So, you know, you'll, you'll hear stories of people getting, you know, Chevrolet cars for, for $0 by, you know, modifying, uh, the system.

272. 41:35

     Um, but what you can do is you can actually have a process on the text that, you know, tries to block certain things, right? Um, and that content moderation's very important to prevent things like that.

273. 41:46

     I don't think you're ever gonna be able to prevent, like, prompt modification, right? Because, you know, if, if you ask an AI bot five times or even three times to, like, break its rules, uh, it probably will, right?

274. 41:59

     Like, the first time it's like, "No, no, I can't do that." And then the second time it's like, "No, definitely can't do that." Third time it's like, "Sure, I'll do that for you."

275. 42:04

     Um, and, and that's just an inherent problem with LLMs. So, you know, you're, you're not gonna be able to stop it on the way in, but on the way out you could be like, "Hey, you know, you've offered something that we've detected is, is invalid."

276. 42:17

     Um, but yeah, it's, it's, it's a hard problem. I don't think anybody's really solved that. I don't think we understand LLMs enough to solve it. Yeah?

277. 42:27

     I'm curious about the TTS and STT side of things.

278. 42:31

     Mm-hmm.

279. 42:31

     Uh, what kind of, uh, what kind of language support is there? What kind of compilation support is there-

280. 42:37

     Mm-hmm

281. 42:37

     ... uh, from, from a, a speech to text perspective? And I guess from text to speech, uh, are there any emphasis or different voices that can make it sound very natural?

282. 42:48

     Mm-hmm.

283. 42:49

     Speaking?

284. 42:49

     Yeah. So the question was about the language support on, um, uh, text to speech and speech to text. Um, so let me just log in here quickly.

285. 43:05

     Yeah. So if I jump over to the text to speech, um, these are the different, uh, voices we have. So we've 12 voices. Uh, today we've got all of our English voices, uh, publicly available.

286. 43:17

     Um, they're, uh, super low latency, uh, and very low cost, right? About 20X cheaper than ElevenLabs. Um, so we're, we're kind of competing at the, you know, the Google, AWS, Azure, um, the voice pricing.

287. 43:32

     Um, but with the quality that's pretty close to ElevenLabs. Um, we're working today on promptable TTS. Um, so that's gonna give you the ability to say, "Hey, you know, say this in an empathic voice," or, "Say it in a pirate's voice," right?

288. 43:45

     Like, uh, the ability to prompt it. Um, and once we've completed that research, we're then gonna r- roll out other languages. Um, the challenge with building them now would be that we'd have to go back and retrain all of the models once the promptable TTS is out.

289. 44:00

     Um, I believe our TTS, uh, launched about four months ago. Um, so yeah, it's pretty, pretty new to the market, but, uh, you can play all of them here as well, which is pretty handy.

290. 44:09

     Deepgram is great for real time conversations. And also you can build apps for think-

291. 44:15

     Deepgram is great for real-time conversa... Deepgram is great for real-time

292. 44:22

     Yeah, and what I've found with customers is, uh, the vast majority of customers want [REDACTED:gender] voices. Um, I, I don't know what, what it is, but I guess nobody wants to be mansplained.

293. 44:33

     Um, yeah, and then on the, uh, on the language side, uh, we have 36 languages supported, uh, today on our Nova 2 model. Uh, we have a few more supported as well on our older models, um, but we're adding languages, you know, e- every month there as well.

294. 44:49

     Um, we've actually got an auto-training pipeline set up, probably the f- first in the world I think, where we have the ability to detect, like, low confidence words and then, like, retrain, uh, based on, uh, low performance.

295. 45:03

     Um, we've also got a ton of other intelligence APIs, so if you wanna do summarization, topic detection, intent recognition, sentiment analysis, uh, you can send all of those off as well, and, uh, I think I have, like, a customer service one here.

296. 45:17

     Um, a- and those can be pretty useful because, like, detecting topics in an actual, um, audio file and where they happen, um, is super useful, right? So if you, if you're in a call center and you wanna understand, you know, at a high level how many calls, you know, of my millions of calls touched on these different

297. 45:36

     things a- and, and which ones to automate. Uh, and we usually say to people that are, you know, automating the call center is, like, you know, step one is analyze all your, all your existing calls, right?

298. 45:46

     Figure out what you've got, and if you've got 40%, you know, uh, phone issue, look at automating, you know, the phone issue first, right? And, um, a lot of them do agent assist, which is, like, bubbling up knowledge-based articles to real people.

299. 46:01

     Uh, and then once they have that built, you know, it's very easy to then just, you know, uh, use an AI. Um, we don't necessarily wanna replace people in call centers.

300. 46:09

     We just wanna take away the work that, uh, can be automated. Um, so, like, what we're seeing now in call centers is that, like, one call center agent using a cloned AI voice can hand off a call, um, to the agent.

301. 46:23

     So let's say it's collecting credit card information. They can just, you know, press a button, let the AI collect the credit card information, and they can do five calls simultaneously.

302. 46:32

     Uh, and then when a call needs their help, they can jump over to the call that needs their help. Um, so you're kind of five X-ing productivity, uh, with that approach.

303. 46:41

     Uh, yeah, question at the back there.

304. 46:44

     How do you go about monitoring, like, if you wanna monitor where the order go- order got wrong or someone tried to jailbreak it or all of those things?

305. 46:54

     Mm-hmm. Yeah, so there's multiple places you can do that. Um, you're probably gonna have the agent actually running on a server. Uh, you know, for this workshop we just put it in a browser, um, but, you know, the vast majority of people will have this agent running directly, like, on a Twilio, uh, you know, phone line.

306. 47:11

     Um, so when, when that agent is doing the work, um, you would handle that in your server code, right? You might have some moderi- moderation code that, um, you know, detects something out of, out of allowment, and then, you know, uh, blocks it.

307. 47:25

     Um, you can definitely add it to the system prompt, but, uh, it's only gonna get you so far.

308. 47:31

     Yep.

309. 47:32

     So what kind of evaluation, uh, strategy that you would optimize in this kind of scenario, like end-to-end or like, you know, text-to-speech, text or function by function evaluation and what kind of metrics you set?

310. 47:47

     Mm-hmm. Yeah, so the, the question is about, like, how to monitor and, and track metrics, is it?

311. 47:52

     Yeah.

312. 47:53

     Yeah.

313. 47:53

     Evaluate if everything works or...

314. 47:55

     Mm-hmm. Yeah, yeah. Yeah, so there's a lot of open source projects out there at the moment for kind of doing agent ops. Uh, one of them is called Agent Ops, uh, which is pretty good.

315. 48:05

     That can give you, you know, uh, temporal debugging, so you can actually debug what was happening throughout the LLM flow. Um, so there's a lot of stuff in that space that's happening right now, but yeah, a lot of your typical, uh, monitoring tools will, will work there as well.

316. 48:21

     Yep. Yep.

317. 48:21

     Um, well, I know of course that's basically a big part of your, uh,

318. 48:28

     company secret, so to speak. Uh, I would be interested in general techniques you use to reach that optimized quality speedup, uh, compared to just the traditional bu- uh, building your own just, uh, as you showed in the beginning.

319. 48:44

     Um, it's voice-to-text-

320. 48:46

     Mm-hmm.

321. 48:46

     ... give that to the LLM and then have it then, um, talk, uh, code and speak it to the agent.

322. 48:52

     Mm-hmm.

323. 48:54

     Uh...

324. 48:54

     Yeah, so a lot of our customers do that today, right? They post all the different pieces, um, and it becomes, like, a large infrastructure challenge, right? You... And we, we'll, we'll touch on that a little bit later as well, uh, as a part of the, uh, the agent swarm stuff.

325. 49:07

     But it's like how do you make sure that you have low latency in different regions, right? So people in the EU don't wanna be hitting a server in the US, right?

326. 49:16

     Um, not just for GDPR reasons, but the latency's gonna be higher. Uh, same with APAC. So now you have to build and scale each of your clusters, uh, in multiple regions, and you have to be able to auto-scale as well, right?

327. 49:29

     Like, um, one of the major use cases for AI agents is peak traffic, right? So, like, you might only need 10 customer service agents, you know, five days a week, but if there's, like, an outage in PG&E, suddenly they need a million agents, right, for one hour.

328. 49:46

     Um, so the ability to actually scale up, uh, same with, um, you know, [REDACTED:phone_number] services, um, they can't take all the calls when there's a large disaster or something happens.

329. 49:56

     Um, so a lot of people actually just, you know, get a, a busy tone. Um, so the ability to, to do that spike up in scale in multiple regions, um, that, that's a huge challenge for a lot of startups.

330. 50:07

     So having somebody that offers that as a service, I, I think is, is pretty useful.

331. 50:12

     Yeah, uh, sorry to add to that. Uh, what I actually meant with the question-

332. 50:17

     If you theoretically have it all running locally, so we completely forget about actual hardware or scaling, et cetera, uh, what kind of techniques you use there to improve the, uh, the inference time, the-

333. 50:30

     Mm-hmm

334. 50:30

     ... time between the different models, et cetera?

335. 50:33

     Yeah. So, so when you run in our hosted API, you're running at a one-second interval, right? So every second you're getting what was spoken, right? Kind of like a metronome.

336. 50:43

     Um, once you run it yourself, you can crank that up and you can say, "You know what? I'm gonna r- run it five times a second." Um, so you're inferencing, like, an ever-growing context window, um, you know, at a much faster pace.

337. 50:56

     Uh, most words are about half a second long, so, you know, you're cho- you're basically inferencing words, you know, partially as well. So the word something might be so, some, something, right?

338. 51:08

     So you're getting this increasing, um, context window. Uh, we run with, like, a three to five second context window in a real-time streaming, uh, which allows us to solidify, you know, every three to five seconds what was spoken.

339. 51:21

     Um, and then as soon as we detect that end of speech, we'll basically, you know, say, "We're not gonna get any more words. Let's, you know, finalize what we have so far."

340. 51:30

     Uh, and that's really how you can achieve those low latencies. Um, but it, it is a lot of compute. Um, yeah. But w- with our system, it's very fast, very, uh, light on compute, so you can actually run, uh, a lot of streams, like on a Tesla T4.

341. 51:45

     Awesome. Oh, one more question.

342. 51:47

     If we are using our own LLM, how can we co-locate with Deepgram?

343. 51:52

     Yeah. So we, we offer, uh, self-hosting. Um, so you can basically, you know, grab our Docker images, models, uh, run it on a GPU. Um, like if, if you can get, like, three GPUs on a single motherboard, um, that's gonna give you, you know, lightning fast, uh, uh, end-to-end.

344. 52:11

     All right, one more question.

345. 52:13

     So working on speech-to-text application and I'm using AWS Transcribe currently. If I want to switch to Deepgram, what three metrics do I see improved?

346. 52:25

     Um, so speed, quality, and price. So yeah, all three. [laughs] [laughs]

347. 52:33

     That's all. [laughs]

348. 52:36

     Yeah, it's, it's, it's easy to sell Deepgram. Uh, okay, I'm gonna jump back into slides 'cause I think some of the other stuff might be of interest as well.

349. 52:44

     Um, so yeah. So some more advanced features that you could play around with, um, make a whole new backend, right? Maybe it's, you know, a table booking API. Um, you know, a lot of businesses have to answer the phone.

350. 53:00

     Um, I don't think they want to, right? They're already busy. Um, handle multi-agent flows, so have a routing agent with two sub-agents. Um, so you could have a booking agent and a cancellation agent.

351. 53:11

     Um, and that can be the same voice on the same phone line. Um, so you know, if I say, "Hey, I wanna make a booking," route it to the booking agent, or, "I wanna cancel a booking," route it to the cancellation agent.

352. 53:21

     Um, tho- those are, uh, essentially how you would build out these, uh, these more complex, uh, systems.

353. 53:29

     Uh, use cases, call center AI agents. I think this is already here. Uh, we're, we're seeing this right now. Like, you know, you, you have companies that have replaced, you know, a, a large proportion of their call volume with AI.

354. 53:43

     Um, and you know, they, they still employ call center agents because, you know, they, they always need to hand off difficult calls that they haven't covered yet with the AI agent, you know, to somebody that wants to handle it.

355. 53:55

     Um, IoT AI devices, right? So wearables, uh, toys, uh, things like that. Um, there's a lot of them out there. Um, and then, yeah, AI worker agents. So, you know, working in a drive-through, taking those orders.

356. 54:09

     Um, you know, the workers that are actually doing that are also, you know, busy preparing food and doing other things. Um,

357. 54:17

     yeah, so multi-agent swarms. Um, so reduced complexity, keep it simple, right? Get something that works really well, really robust, and kinda, like, box it off, right? Like single responsibility.

358. 54:29

     Um, reduce the cost. Use the smallest, cheapest model you can to achieve the use case. Um, like right now it's probably gonna be those bigger models, but I think in time, you know, the price point of those is gonna come down, and the new generation will, will kinda take its place.

359. 54:44

     So, you know, e- every six months we're seeing like 10X drop in cost. Um, and then composability. So you can reuse, you know, a, a sub-agent in multiple different flows.

360. 54:56

     Um, so th- this is a kind of a pretty basic kind of layout. So you have your routing agent that's able to figure out, you know, which agent to use.

361. 55:06

     Um, you have a support agent and a booking agent in this example, and maybe you have a technical support agent and an account support agent, right? Two different, uh, types of agent, but they, they can each kinda help depending on the need.

362. 55:17

     Um, your tech support agent is probably gonna be hooked up to some sort of RAG system. Um, account support and existing booking agent is probably need to... are gonna need to verify, uh, that this person, you know, owns the account that they're calling about.

363. 55:32

     New booking agent might, uh, leverage like a credit card payment agent.

364. 55:37

     Um, yeah, so scaling it, right? We, we talked a little bit about the latency. Um, so distance, uh, kills latency, right? Like, if you call a server in the US from Asia, um, you're gonna see like, you know, a second extra, uh, at least latency.

365. 55:52

     Um, if you wanna do regional scaling, right, you're gonna need to have, uh, the ability to horizontally scale, um, you know, within US East, within US West, within EMEA, right?

366. 56:04

     You want-- You're gonna wanna have redundancy as well. Um, as you add redundancy, um, you increase cost as well. Um, but you know, if you want high availability and your agent to always be on, you're gonna need that, uh, re- redundancy to do it.

367. 56:20

     Um, and then horizontal scaling within your, you know, your regional clusters, um, we support Kubernetes, and we'll give you all the auto-scaling Helm charts and everything. Um, so that can be pretty, pretty powerful.

368. 56:31

     Um, but you can imagine, like, if you wanted to You know, build an agent, do you really wanna worry about all that infrastructure, right? Or do you want to just build the agent, achieve the, you know, the value, the business value, and, and roll it out on, on a large scale?

369. 56:49

     Yeah, sorry, go on.

370. 56:51

     So based on this, um, I was wondering what your outlook is on the embedded models. Um, like, do you think that at some point in time that embedded speech-to-text and text-to-speech models which are on device, do you think they will be good enough, um, that, that we can have them on device and not worry about things like

371. 57:09

     what latency

372. 57:12

     Yeah, I, I think it's possible. Uh, you still have to distribute the models, right? The models tend to be quite large. Um, I think Gemma is, like, you know, two gigs.

373. 57:20

     You can put it in the browser, but it's, it's gonna take you a while to download. You know, as you move to mobile devices, you know, it's gonna be pretty hard to do it as well.

374. 57:28

     Um, like, I, I do believe that there's a lot of use cases where on device makes sense. Um, but those are, like, single, single stream, right? So you have a single stream, you're sending a single request to an LLM, you're getting a, you know, single response.

375. 57:43

     Um, what we're building here is, like, you know, a million simultaneous calls, right, can come in. Um, that's never really gonna work on device, um, just from a distributions perspective, I guess.

376. 57:55

     Um, but there, there's definitely use cases for it. So, uh, like for the wearable use case, uh, there's an open source project called Friend, uh, helped them integrate, uh, Deepgram's real time, uh, speech recognition into that.

377. 58:07

     Um, so, you know, running it on device doesn't necessarily mean the model has to be on device. Um...

378. 58:13

     Yeah. I, I was curious about the, the speech recognition models because LLMs of course they're huge, but the speech recognition models can be effectively cut down to be able to run on device.

379. 58:24

     Yeah.

380. 58:25

     I'm curious, like, is, is, is, uh, is your company thinking about getting the embedded model, getting to the embedded model space, or

381. 58:32

     Mm-hmm

382. 58:32

     ... the cloud model is

383. 58:33

     Yeah, it's something we're looking at. Uh, you can run our current model on a Raspberry Pi. Um, it, it's not gonna be super fast and it's not gonna handle multiple concurrent, uh, requests, but, uh, it will run.

384. 58:46

     Uh, so it can run on CPU and not just GPU.

385. 58:49

     Mm-hmm.

386. 58:51

     I have a question.

387. 58:52

     Yeah.

388. 58:53

     What service do you recommend for the telephony part of the system? We use Twilio for various things, but are there, are there other options beyond Twilio?

389. 59:01

     Yeah, there's a few telephony providers out there. Uh, Twilio, Vonage. Um, we're kind of, you know, telephony agnostic, as long as you can get us the, the audio stream.

390. 59:11

     Uh, usually that's achieved either through, like, you know, hooking into their API or just doing a SIP trunk. Um, so SIP trunk basically just hands off the processing of the call to a different server.

391. 59:23

     Yeah.

392. 59:24

     Um, when GPT-4O eventually releases their audio-to-audio model, is Deepgram gonna be, like, pivoting around that to build applications around that?

393. 59:36

     Uh, I, I don't think we would necessarily use it in, in that regard. Um, so, uh, I think it's great for the space that, you know, they're, they're releasing this.

394. 59:45

     Um, I think maybe in the future, this type of, uh, multimodal model will make sense. Um, but it's yet to be seen what the price point is gonna be and what the latency will be.

395. 59:56

     Um, like even their chat completion API for, for 4O is taking, like, up to a second. Um, so if you add audio into that as well, that's additional processing.

396. 1:00:05

     Um, like it was a really cool demo and, and I loved, like, how the... Like, a single model could have, you know, infinite voices. Um, and I, I think that's where we'll see a lot of, uh, changes in the future.

397. 1:00:17

     Um, at Deepgram, our main focus is, um, like scalable, uh, low cost, efficient, uh, inferencing. So, you know, the ability to run at these price points, um, you know, is probably gonna be a barrier to entry.

398. 1:00:31

     Um, but yeah, I'm, I'm looking forward to see what they, what they release and, and when. Uh, yeah.

399. 1:00:38

     When, when you implement a solution for a client, how long does it take you need to fine-tune your model?

400. 1:00:45

     Mm-hmm. [echoing] Yeah. So for model fine-tuning, uh, the question was how long does it take? So we require between, like, 20, uh, and 50 hours of audio, uh, to fine-tune the speech to text.

401. 1:00:57

     Um, and then we actually human label that. So, uh, we have our, our own team of human labelers, and what they'll do is they'll actually do three passes. So the first pass, um, usually works out at about, like, 12% word error rate.

402. 1:01:09

     So if you, if you give somebody a piece of audio and you ask them to write down exactly what was said, um, they'll have errors in it as well.

403. 1:01:15

     Um, so then we run it through a second pass, they fix the prior errors, and then the third pass is... And they're all different people as well. So the third pass goes in and basically gets us to, like, 99%, uh, label accuracy.

404. 1:01:27

     Uh, and then we train the model. Training the model is pretty quick, right? We could probably do it in under a day. Um, but yeah, getting all that audio and then, uh, labeled, and then we kick off the training cycle.

405. 1:01:37

     Um, for the text-to-speech side, we don't offer cloning, uh, voice cloning today. I think there's a lot of concerns around, you know, what happens when you clone people's voices and, um...

406. 1:01:47

     We do do cloning for certain customers. Um, so if a customer comes to us and says, "Hey, you know, we want to use you, but we need our own voice for our brand," um, we can, we can do that training.

407. 1:01:57

     Uh, but that would be a, a business engagement.

408. 1:02:02

     Yeah. Yeah, question over there.

409. 1:02:06

     Are we going to use the API keys or it will be managed by Deepgram?

410. 1:02:10

     Yeah. So right now, uh, everything is just on the sandbox API. Uh, this API will probably go away after the workshop, um, but we'll have a, a way to actually sign up, uh, for the API wait list.

411. 1:02:23

     So if you do wanna get access to this, uh, and then it will require an API key. Um, and all of the services will be wrapped under a single, um, kind of usage fee.

412. 1:02:35

     So your speech to text, your LLM, and your text to speech will all be under a single, uh, cost.

413. 1:02:44

     Awesome. Any more questions? Yeah, sir, you.

414. 1:02:50

     Yeah. I was just wondering, are there any plans to allow, um, kind of multiple speaker input within this new API?

415. 1:02:56

     Mm-hmm.

416. 1:02:56

     Uh, you know, being able to recognize speaker one, speaker two, speaker three.

417. 1:03:00

     Yeah. Yeah, so we, we have diarization. So if you send multiple speakers on the same, uh, channel, uh, we'll be able to determine, you know, speaker A, speaker B.

418. 1:03:11

     Um, if you're sending us multi-channel audio, uh, that will allow us to inference them separately. Um, yeah.

419. 1:03:18

     So specifically mean more on the single channel.

420. 1:03:21

     Mm-hmm.

421. 1:03:21

     And I played around a little bit, but it's not perfect today. I- I'm just wondering, are there plans to kind of enhance that or expand upon it?

422. 1:03:27

     Mm-hmm. Yeah, we're always improving, uh, diarization. Uh, it's definitely a challenge because, um, to, to understand how diarization works is you're, you're building up embeddings of what people say.

423. 1:03:39

     Um, so like our conversation so far, you, you've had maybe three or four sentences, you know, maybe 20 seconds of audio, and that may not be enough for the model to say, "Okay, this is a unique speaker."

424. 1:03:50

     Right? Um, and it's, it's building out these embeddings in like 512 dimensional space. So, you know, as more data comes in... And we, we typically recommend 30 seconds per speaker, um, to actually generate a solid embedding.

425. 1:04:03

     Um, if, if we were to lower that requirement, we might start like mislabeling people from the same person. Um, but it, it is a challenge, and it's... I don't think it's ever gonna be perfect.

426. 1:04:14

     Um, you know, uh, one of the hardest parts of diarization is actually when, um, is actually when, uh, people actually say like, "Yeah," or, "Mm-hmm." So like if, if you're on a call and somebody, while you're speaking, says, "Yeah," it, it's very hard for the AI with that tiny little, you know, segment of audio to know that

427. 1:04:35

     it's somebody else speaking. Um, but yeah, we, we, we've seen a lot of cases where, you know, if it's a longer call, um, it, it works very well, but those first 60 seconds, it's probably not gonna determine who's who.

428. 1:04:48

     Sure. Yeah.

429. 1:04:49

     Yeah.

430. 1:04:49

     That makes sense. Thanks.

431. 1:04:50

     Yeah.

432. 1:04:50

     And I imagine behind the scenes there's maybe some accuracy percentage, right?

433. 1:04:54

     Mm-hmm.

434. 1:04:55

     Like similarity score. Like is that something that might ever get exposed, or is that exposed today, so we can kind of make a decision ourself, right? Where it's we get something back that says, "It's probably a 5% chance that it's this guy speaking."

435. 1:05:07

     Mm-hmm.

436. 1:05:07

     Um, that would be really helpful.

437. 1:05:10

     Yeah. And one of the things a lot of people ask for is the ability to, you know, get speaker identification, right? So like a unique identifier for a speaker.

438. 1:05:19

     So like if you have a call center agent and you know who they are on every call, you know, could you pass that to us and we'll tell you, you know, were they the first speaker or the second speaker?

439. 1:05:30

     Um, it's not something we expose today. Um, obviously there's, you know, uh, legal challenges around fingerprinting voices and stuff. Um, but yeah, it's something we're thinking about, the ability to at least like identify a speaker, um, and just say like, you know, "This speaker is this person."

440. 1:05:48

     Yeah. Thank you.

441. 1:05:49

     Yeah. You're welcome. Awesome. Um, so I think that's everything I had. Uh, I'm interested though, had anybody achieved order update with the remove item?

442. 1:06:04

     Any- anybody get any, uh, additional APIs up and running?

443. 1:06:13

     No? I got the demo running, but- You got the demo? ... it's a little bit too loud in here. Yeah. It picks up you a lot. Right. Right. Yeah.

444. 1:06:19

     It, um- It works, and it's fast. Yeah. I told it to add lots of items back to back, and it did. Nice. Yeah. And, and this is, again, running on our hosted API.

445. 1:06:28

     So we, we haven't even optimized this for low latency yet. Um, but you can see how quick it is even with those, um, hosted APIs that it can respond in that time.

446. 1:06:38

     Um, like we, we did run a kind of a sandbox environment where, you know, we cranked up that compute and, and it was just so fast that like, it was just like kind of like interrupting you, like the moment you stop speaking, um, uh, which is, which is a pretty funny, uh, challenge.

447. 1:06:57

     But yeah. Uh, thanks everybody for coming. Um, feel free to hit me up, uh, or chat with me after, uh, the workshop. But yeah, hope yous all enjoyed it. [upbeat music]