Voice AI: Your Bot Isn't Special

Talk transcript

1. 0:00

   [upbeat music] Hey everyone, uh, I'm Nick.

2. 0:19

   I'm an engineer at SuperDial, and first of all, big thanks to the organizers. This event has been awesome. I've had a blast talking to you guys, connecting with you guys, and hearing all these great talks.

3. 0:29

   Um, somehow I'm one of the few voice AI talks today and this weekend, so I have a lot to cover. We're gonna dive right in. If you're new to voice AI, I hope I can ver-- provide a nice little framework to think about this very fast-moving space.

4. 0:41

   And if you're building with voice AI already, I'll be sharing some little anecdotes from our own scaling journey that I hope will help yours as well. So voice AI in twenty twenty-five, extremely exciting.

5. 0:54

   We're seeing these new smart, really fast, really affordable LLMs that are supporting a lot more complex conversational use cases. Uh, but you still kinda need some tricks to take your chat agent and turn it into a voice agent.

6. 1:09

   We have these low latency, really realistic, super generative text-to-speech models, but sometimes we have audio hallucinations, and we have to deal with things like pronunciation and spelling.

7. 1:20

   With all the new things that people are building, there's this explosion in voice AI infrastructure, and tooling, and evaluation systems, and a big question becomes, what's actually worth owning?

8. 1:30

   And the big one on everyone's mind are these new speech-to-speech or voice-to-voice models. Uh, and our take is that for a lot of production applications, they're not quite yet ready.

9. 1:42

   And a big reason for that is they start to output things that aren't actually speech, aren't actually, uh,

10. 1:49

    things that you can use to build a reliable conversation. And this-- we saw this when they first came out. They were, like, imitating people's voices. And from the start, that's why we've kinda been favoring, uh, reliability over this sort of realism.

11. 2:03

    So today I'm gonna talk about how we at SuperDial approach agents as a service, how we think about the voice AI engineer, and the last mile problem. So once you have your little voice, uh, MVP, all the challenges that you're gonna face trying to actually make it reliable and put it to work.

12. 2:21

    So at SuperDial, we're in the business of phone calls, specifically one of the most annoying phone calls ever, that phone call to your insurance company. So for mid to large-sized healthcare administration businesses, we sell the SuperDial platform.

13. 2:38

    And with SuperDial, you can build your script, so design the sort of conversation, ask all the questions that you need to get answered over the phone. You send us your calls via CSV, API, or we also integrate with a lot of EHR software systems.

14. 2:54

    And then, you know, within the next couple of hours, in the next day, we send you back your results in a structured format. And this makes for a really interesting agentic contract that we sort of have with our customers.

15. 3:06

    So from their perspective, they're paying for results. They tell us who to call, which questions to ask, and we tell them the answers. Internally, we have a little agentic loop set up so that,

16. 3:18

    uh, we go out, we wait for these offices to be open, we wait for, um, you know, the call centers to open, so we can actually make these calls.

17. 3:26

    We will attempt to make the call with our voice bot, and then if our voice bot needs to bring in a human to complete the call or cannot complete the call after a certain number of attempts, then we send it to a fallback team.

18. 3:37

    And this is something that, of course, we're transparent with with our customers. In fact, it's a benefit to them because it's kind of inevitable with these healthcare phone calls that sometimes you need to bring in a human.

19. 3:48

    So with us, they know that no matter what happens, the call will get made. Whether or not it gets made with a human or a bot doesn't matter to them.

20. 3:55

    They get their answers reliably and in a structured format.

21. 3:59

    Uh, and with all these calls, we try to do our best to learn from them. So we'll update the sort of office hours for the given phone number we're calling and learn from the sort of phone tree traversal that we just tried, so that when we call it again, we get even better at that sort of call.

22. 4:14

    And because these are sensitive healthcare phone calls, we wanna make sure our system always works. So randomly, we'll take out some of these calls, audit them, make sure everything's working.

23. 4:24

    Uh, for a quick little demo, this is actually a prior authorization call. Uh, this is after the point where we've traversed the little phone tree by clicking the right buttons, and now we're talking to a human and trying to get some questions answered for a customer.

24. 4:39

    May I know your first name?

25. 4:42

    Hi, this is Sarah.

26. 4:45

    Are you calling from a doctor's office or from a facility?

27. 4:49

    I'm calling from [redacted] provider's office.

28. 4:53

    Do you have a member ID or a case number?

29. 4:57

    The member ID is [redacted].

30. 5:03

    What is the CPT code?

31. 5:06

    The CPT codes are [REDACTED:generic_id].

32. 5:12

    Okay, hold on. So there's a case on file, uh, that was initiated for the code [REDACTED:generic_id]. It is pending. So this case number is [redacted]

33. 5:27

    . And we have not received any clinicals for this case yet.

34. 5:34

    Okay. What is your name again, and what is the reference number for this call?

35. 5:39

    My first name is [redacted]. You may have the pending case number as a call reference number, and the fax number on where to send the clinicals is [redacted]

36. 5:50

    .

37. 5:50

    Thanks so much for your help.

38. 5:54

    You're welcome. Thanks for calling. Have a great day.

39. 5:58

    So that's it. Uh, if that call was really boring to you, thanks-

40. 6:04

    If that call was really boring, that's kind of just how these things go. A boring call for us is an excellent call 'cause it turns out a lot of work is boring. [laughing]

41. 6:14

    Uh, so with this system, we've been able to save over a hundred thousand hours of human phone calling time, and we're on track to save millions more in twenty twenty-five.

42. 6:21

    And what's really incredible about voice AI today is that we did this with a really lean team of four engineers. So building the whole full stack web application, these EHR integrations, the bot you just saw, all while bringing on new customers, supporting new conversational use cases really quickly.

43. 6:38

    And a big part of why that was possible was because we really all embrace this role of a voice AI engineer. So let's kind of uncover what's unique about a voice AI engineer today and what hats they may be wearing.

44. 6:51

    So starting from Switch's, like, original graph, we can kinda see that a voice AI engineer is gonna deal with multimodal data. So MP3s, audio bytes, in addition to transcripts.

45. 7:01

    You're dealing with transcription models, voice models, speech-to-speech, all that sort of thing. The application you're building, it's in real time. Latency all of a sudden matters so much more.

46. 7:12

    You're gonna be dealing with async in Python a lot more than you probably wanted to be doing. And the product constraint here is almost always gonna be a voice conversation.

47. 7:21

    So people have really high expectations of how these sorts of conversation goes. Uh, for us, like, we're slotting ourselves into an existing, uh, sort of like business interaction, and people expect us to be conversational and fit into that use case.

48. 7:37

    So to grapple with all these challenges, we kind of have two sayings at SuperDial that we've been saying over the past year and a half. Say the right thing at the right time, and build this plane while we fly it.

49. 7:49

    So the trickiest part, uh, for us is customizing all these scripts and all these use cases for each customer individually, and then we really rely on this kind of like horizontal voice AI stack to help us out with all those other problems.

50. 8:06

    And this is kind of how we think about the voice AI engineer today and its unique roles. And in the larger context, we're really at this inflection point where it's so easy to build out an MVP for these sorts of applications that ultimately what is going to make your voice bot unique isn't its voice or its interruption

51. 8:26

    handling or how realistic it sounds or how it does turn-taking. Ultimately, it's gonna be in the conversational content and the design there and the vertical integrations around it that make your agents' work actually valuable.

52. 8:40

    And if you're like me, and your favorite classes in college were the AI ethics ones, everything I just said about moving fast, building with generative AI could raise a few red, uh, or like raise some alarms.

53. 8:54

    So it's not hard to imagine how voice AI apps specifically could be biased against people with certain accents, people with certain dialects, or be really spooky when they sound so real and then say weird things.

54. 9:07

    So in the US, we both like enjoy and suffer from a lack of AI regulation, and that leaves the onus ultimately on the AI engineers and leaders in this room to think about these sorts of problems.

55. 9:20

    This is not gonna be like a talk on like AI safety and ethics, but I think for voice AI specifically with how it's such like a new modality of interaction with artificial intelligence today, I think it's really important how we go about building it.

56. 9:35

    So for AI engineers, when we go about making tooling and infrastructure choices, uh, remember that like developing AI should be really accessible and collaborative, and the work that AI does should be for everyone.

57. 9:49

    And a key part in making sure that's the case is choosing tooling and infrastructure so that a really diverse set of stakeholders can be involved in that process from the start.

58. 9:58

    So with the role of the voice AI engineer kind of scoped out now, let's dive into some of the last mile problems in voice AI that we've been dealing with.

59. 10:07

    So when we started out, we had a really scrapped together pipeline of like a transcription model and an LLM and then a text-to-speech model. Uh, this was awesome to get started at [chuckles], but you know, we faced a lot of problems very quickly.

60. 10:21

    And a lot of what we were learning was not new at all. So though the voice agents we see today are better than ever, voice UI itself is not that new.

61. 10:30

    So when we were just getting started, uh, around a year and a half ago, I had the chance to speak to Cathy Pearl, who is a close family friend and has been working on, uh, the UX of Gemini.

62. 10:41

    She's been in the conversation design game for like twenty years or something. Uh, and back in the day, like voice UI was lots of phone tree design, and then it becomes th- these Alexa and Siri-type things, and now we're just in this whole new world.

63. 10:56

    But a lot of the principles remain the same. And one of the biggest things that's changed with developing voice UI is the shift from prescriptive to descriptive development. So we no longer prescribe what we want our bot to do over the course of the conversation by mapping out every possible direction that it could go.

64. 11:16

    Instead, we describe what we want to do and then kind of pray to the generative gods that it happens. [laughing]

65. 11:23

    And for this, you know, there's a lot of things I'd talk about with conversation design, but it comes up really quickly when that becomes your main interface. One thing for us is when we're asking these questions, you know, should we be really open-ended with it or kind of constrain the user into selecting from a list of choices?

66. 11:39

    And for us, because these are existing conversations, we find it's often better to just go general, hope the call center representative gives us a ton of information, and then instead of trying to prevent them from saying the wrong thing, we try to adapt to whatever they say.

67. 11:55

    So Cathy's recommendation was hire a conversation designer. If you're thinking about these sorts of problems, there are experts in this. And if you're just a voice AI engineer and you wanna get started in this kind of thinking, a great recommendation is to do little- Table reads.

68. 12:13

    So have one person pretend to be the bot and the other person pre- pretend to be a user, and the sort of like transcript that you may write out by hand, immediately

69. 12:23

    the sort of gaps and awkwardness of it comes out when you say these things out loud. So knowing all these things, we were really excited to work on our conversations, but we had-- kind of had to deal with the tech debt, debt of the orchestration framework that we had built.

70. 12:37

    So we really hit our stride when we started using Pipecat for our voice AI orchestration. This is an open source framework maintained by the guys at Daily. It's really easy to extend and hack upon, which is important for our use case when we need to do transfers and stuff.

71. 12:53

    Um, and we make really long phone calls. These can be like an hour and a half long. So a big decision for us in choosing Pipecat was that we can self-host it and deploy it and scale it how we want.

72. 13:05

    So with some of our like voice orchestration headaches dealt with, we really wanted to get back to focusing on our conversations. And everything in this slide for us is really not unique to voice UI, uh, and AI.

73. 13:19

    So I'm gonna kinda speed over it. Two interesting decisions we've made here, because we just have, you know, an LLM in the backbone. Uh, we chose to own our own OpenAI endpoint.

74. 13:30

    We find this leads to a better interface with a lot of these new voice AI tools. So behind our OpenAI endpoint, we can kind of route to different models that are maybe more, uh, latency sensitive.

75. 13:44

    For all of our generative responses, we route them through this tool called TensorZero. TensorZero is relatively new. They have this nice framing of LLMs. Uh, if that quote interests you, I recommend you look them up and talk to them.

76. 13:57

    They're awesome. Uh, this is like a little open source tool, so you can do whatever you want with it. They give us kind of structured and typed LLM end-endpoints that we can then experiment with in production.

77. 14:08

    So that's our gateway to our LLM. And then all of our logging and observability, we self-host LaneFuse, and we self-host these things also because these are like healthcare calls.

78. 14:19

    We have to be HIPAA compliant. That's often an easiest-- an easier way to deal with, you know, the rapid growth of this space. So there we do like anomaly detection, evals, and datasets.

79. 14:30

    So with a good plan in place for our LLM sort of work, another big challenge is our text-to-speech system. So when you make these sorts of phone calls, your password is basically your name, your date of birth, and then your member ID or something, which is like a twelve-digit long string of characters that you have to be

80. 14:49

    able to communicate over the phone. And something we quickly realized was that what our LLM is outputting is not necessarily what we wanna shove through our text-to-speech engine, and neither of those things may actually match what's in the recording.

81. 15:04

    So a little example of this, and this is like a personal last mile, is that if you're building me a personal voice UI application, it should say my last name correctly.

82. 15:14

    So my last name is pronounced Caryotakis. Most people and most models will say Caryotakis. But with a lot of new tools out there, this is the syntax this company called Rhyme uses.

83. 15:26

    You can spell out the exact sort of pronunciations you want, and then for things like spelling, where you may have kind of an intuition for like the sort of pauses and breaks you might wanna use to say a really long word, you can use something like this little spell function.

84. 15:42

    Um, and then with all this stuff, like, because this is outputting audio bytes, we usually review recordings to make sure that this all sounds okay, in addition to checking the transcripts.

85. 15:53

    And to start wrapping things up, I have a couple little mini last mile problems that we've had to deal with. Oh, and, you know, with voice-to-voice models, all this sort of rule-based stuff gets a little more complicated.

86. 16:05

    So some little mini ones. Uh, we used to be called SuperBill, and we called our bot Billy 'cause we thought that was a fun name. Turns out that's an awful name over the phone because we would constantly have these conversations where people were like, "Hey, nice to meet you, Billy."

87. 16:21

    And we would say, "It's Billy, not Billy." [laughing]

88. 16:25

    So yeah, think about your persona a lot. Dial that in early.

89. 16:31

    Uh, if you're just starting, don't build from scratch. What's gonna make your bot unique is the conversation, and there are so many new tools out there like Pipecat that you can use to get a quick jump start.

90. 16:41

    Track latency everywhere. Time to first byte for each of your little processors is the new most important metric, and it's something you always kind of have to keep an eye on.

91. 16:50

    Uh, upgrade paths. This is a big one for us when we need to make sure we have really high transcription accuracy. So we use Deepgram for our speech-to-text engine, and we know that whenever we kinda wanna improve that part of our system, we can work with them to fine-tune a better model.

92. 17:08

    Have fallbacks ready. It really sucks when OpenAI goes down for a little bit, and all of a sudden, all the concurrent conversations you have are just down the drain.

93. 17:17

    So have fallbacks ready for each part of your stack. It's really easy to set that up with something like TensorZero. There are lots of other tools that'll help you figure that out.

94. 17:26

    And then end-to-end testing. This is pretty unique for voice UI and-- or voice AI. Uh, it seems like people are kind of settling on telephony as a boundary layer to test your bot with, like, an external service.

95. 17:41

    We do a couple different things. The easiest test for us is to create a kind of fake phone number that just plays an MP3. If your bot can't talk to an MP3, then you probably have bigger problems.

96. 17:51

    Next, we can kind of create, uh, a simulated voice tree with like different, uh, like phone tree building tools and have our bot pseudo-navigate it. And then there's lots of generative services like Koval and Vocera, where you can have your bot talk to another bot.

97. 18:07

    So some takeaways for a, a what I call a vertical voice AI, AI engineer. Choose your stack wisely. The better decision you makes-- you make here, it will allow you to focus on the things that are really truly unique to your conversational experience.

98. 18:22

    Laser focus on the last mile because this is where ultimately you can provide a lot of value and put your agents to work. And then ride the wave. There's so much new stuff happening in this space, and whenever new models come out, you wanna be able to use them quickly, and you also won't-- wanna be able to

99. 18:38

    use them safely. So thank you very much. I'm excited to talk to you all and hear about what's so special about your conversations. [clapping] [upbeat music]