Shipping an Enterprise Voice AI Agent in 100 Days

Talk transcript

1. 0:00

   [on-hold music] Today, I'm gonna be talking about Fin Voice.

2. 0:18

   And Fin Voice is a voice agent for phone support. Um, and we designed it to be a frontline teammate for inbound calls, so it picks up the phone, answer customers' questions, and then escalates to a human agent when needed.

3. 0:31

   And we built this experience in about 100 days. So today, in this talk, I'll share what it took to get there. And I'll also talk about why I believe voice is the next big frontier in AI for customer service.

4. 0:45

   So first, a little bit of context of my company, Intercom. Um, so we're a customer service platform, um, and also an AI agent company. And you might be familiar with us because of our messenger product.

5. 0:56

   Uh, seen it... You might have seen it in the mobile app or on the website. Uh, but yeah, it's been our foundation for years. Uh, but we evolved over time.

6. 1:04

   We became a complete customer service platform a few years ago, added robust tooling for other channels like, like email, WhatsApp, and, uh, and phone.

7. 1:11

   And then two years ago, right after the launch of GPT-4, we launched Fin, uh, which is an AI, uh, uh, agent, uh, via on text, on chat. And Fin's growth has been incredible.

8. 1:24

   Uh, we have over five thousand of, of custom- customers, and also in terms of performance, it's reaching average resolution rate of fifty-six percent, and for some customer it's reaching seventy, eighty percent.

9. 1:35

   Um, and this is defined as a percent of interactions handled by Fin that are resolved without human intervention.

10. 1:43

    And Fin is also a full system for continuous optimization, so it's not just as the agent, but also tooling for analyzing conversations, training the behavior of the agent, and also testing and deploying changes.

11. 1:56

    Uh, but yeah, up until now, we didn't have the voice channel, and that's what we're changing with Fin Voice. It's the same system, but now it can answer phone calls.

12. 2:04

    And a few thoughts on, uh, why voice? Why are we investing in this channel? Um, so for all of users, voice is simply the preferred way to get help.

13. 2:15

    Uh, so when an issue is urgent or sensitive, uh, they don't necessarily wanna type, they just wanna talk. And if you look at the, uh, some of the top of our data, over 80% of support teams, uh, still use, uh, phone support.

14. 2:30

    And if you think about all the conversations globally or customer service interactions, over one-third of them are, uh, happening over the phone as well. So it's not a legacy channel that's going away.

15. 2:40

    It's still widely used, and it's also quite costly. Uh, if you think about the average cost of handling a, a phone call in the US with human support, it's about between seven and twelve dollars.

16. 2:52

    And with voice AI agent, it's gonna be at least five times cheaper.

17. 2:56

    And a few more benefits of voice AI in, in customer service. Uh, so first, availability, twenty-four/seven support. Uh, you can call your bank on the weekends. Uh, no wait time, so instantly available.

18. 3:08

    There's no need to be... to stay in... be on hold in a queue. Uh, no IVR menus, no need to press one to go to support, press two go to payments, uh, because all is...

19. 3:17

    everything is happening via natural speech. And also multilingual, so, uh, AI agents can su- support 30, 40 plus languages, so obviously it's better for the, for the users. And on the business side, major cost savings and also scalability as the business grows or as every- when you need to handle peak times, um, AI agents are much better

20. 3:36

    for that. Um, so how we built Fin Voice. Uh, over the next few minutes, um, I wanna cover, uh, seven main areas that had the biggest impact on how Fin Voice came about.

21. 3:48

    I'll try to be more practilal-- practical, focusing on the some of the product decisions we made and the challenges we faced. Um, um, and yeah, the first one is, is the use case or the starting point for, for Fin Voice, uh, then also the scope of our MVP, uh, the tech stack behind it, the how we approached

22. 4:06

    the conversation design, how we integrated it with the support teams, and also, uh, how we thought about evaluation and pricing.

23. 4:17

    So starting with the use case, um, if you look at, uh, all of the voice AI startups in the space, they typically start with, uh, a narrow problem space, something like scheduling a dentist appointment or, uh, booking a table at a restaurant.

24. 4:32

    And we looked at the, at the, some of those options, but eventually decided to go for a more flexible, uh, knowledge-based agent. So an agent that can answer helps, uh, help article questions like, "What are your pricing plans?"

25. 4:45

    Or, uh, "What's your returns policy?" Uh, why did we decide to go this way? So first, uh, we had a strong evidence from chat. Fin, over chat, has been handling those kind of conversations for years, and our customers have con- constantly told us that they're seeing the same type of issues over the, over the phone as they

26. 5:02

    see on chat. And we also validated this for e- extra analysis of, of call transcripts, and this confirmed that a very large percentage of all the queries could be solved with the, with the knowledge base, with the help articles content, rather than, say, with the API integrations.

27. 5:19

    And we are also thinking about the initial wedge use case. So like the what's the lowest possible risk way for companies to integrate voice agents? And we looked at the in-office hours and outside-of-office-hours use cases.

28. 5:32

    So w- we pitched out-of-office hours as the initial wedge because essentially it allows the, the team to not affect their main workflows and try out this technology, build up more confidence over time, and later deploy it for their, on their, in their main office hours.

29. 5:46

    But in out-of-office hours, it just replaces their voicemail experience.

30. 5:51

    And there's also a few other use cases we looked at. Uh, authentication, so verifying user identity on another channel. Uh, info gathering, so the agent getting stuff like, uh, order ID, account ID, and also, uh, smart routing to the right team.

31. 6:06

    So these use cases are, uh, still very high leverage because they can save a lot of time for the support team agents. Um, uh, but they're not necessarily solving the issue end to end.

32. 6:17

    So we're still focu- fo-focusing on those, but they weren't necessarily the primary use case for this initial version of the product.

33. 6:24

    So now moving on to, uh, what we shipped first. Um, uh, when we, uh, wh-when we started, started this, like the biggest challenge there was to ship something meaningful as soon as possible.

34. 6:37

    We had access to a lot of customers, uh, because we, we already had thousands of customers using our native, uh, phone support product. So it's mostly about how can we test as quickly as possible.

35. 6:47

    So we focused on the three main experiences: uh, testing, deploying, and monitoring the agent behavior. So first test, uh, this is what we call the Fin Voice playground. And here was like a, as a, uh, lightweight test environment for the customer service managers to go in and simulate a few sessions, uh, ask the questions based on the

36. 7:06

    knowledge base, get those answers, and get an idea how this product actually works. And it is also may-- we shipped this probably within the first four weeks of the project.

37. 7:14

    Uh, so it was like the, the fastest possible way to get, get some feedback from customer service managers. This is how it's actually performing, so we can do optimization, not just based on our internal views, but also based on customer feedback.

38. 7:27

    Um, then the deploy experience. This was to allow, um, uh, customer service managers to actually deploy it on their phone lines and include some, uh, basic configuration in terms of agent behavior and also how it should interact with the customer service team's workflows.

39. 7:44

    And lastly, observability, uh, or monitoring. Uh, really wanted to provide some visibility into, into what's actually happening on those calls with an AI agent. Uh, so we had those experiences that just show, uh, the transcripts recording and also the transcript summaries and call outcomes to, uh, customer service agents.

40. 8:06

    Cool. And now moving to the tech stack. Uh, I'm not gonna go for the technical detail of everything. I'm sure that there's gonna be a few more talks on, on this that you might attend as well.

41. 8:14

    Uh, but I'll mention some of the core components, uh, in, in Fin Voice. Uh, so there's the main, uh, chained loop for, uh, for the voice agent STT, LLM, TTS.

42. 8:26

    Uh, so speech-to-text, converting, uh, speech into text. LLM for, uh, uh, actually generating the response, and text-to-speech for converting text back into audio. Uh, but there's also another approach with the voice-to-voice models, where, uh, everything is processed, uh, directly in audio while skipping the text layer entirely.

43. 8:44

    So, uh, and the voice, voice-to-vo-voice approach has the benefit of, uh, potentially faster and natural, more sounding speech, but also gives you less control over the output. Uh, so in our approach, we did start with real-time API by itself from the get-go, and it allowed us to test very, very quickly.

44. 9:01

    Uh, but eventually we did evolve our stacks, but we're still using real-time API as part of the, of the, of the core architecture. Uh, there are two other components I want to mention, RAG and telephony.

45. 9:12

    So RAG obviously is super critical for all of ex-agent experiences. Uh, but obviously it's, it is important for the, uh, agent answering questions based on the knowledge base. Um, and then telephony.

46. 9:23

    Um, uh, so as actually being able to put the agent on the phone lines. And we had a bit of a head start because our agent on chat already had the RAG set up and alread- we already had a native phone support product, so we got some of those things for free.

47. 9:41

    Now, once, uh, uh, we have the technical foundation in place, uh, there's a question about how do we actually design the conversations for voice? Um, so, um, Intercom has a background in chat, but we knew from the get-go that the approach for voice will have to be a bit different, um, and that voice is not necessarily just

48. 10:01

    chat with sound. So there are like three key differences I want to mention. There's obviously many, many more, but there's a few that I, I thought it's worth mentioning.

49. 10:08

    So latency, um, uh, on chat, it's actually, I think, okay to wait for a few seconds for a response, or at least from the user's perspective, that there is a lot of tolerance.

50. 10:18

    But obviously it doesn't really work on voice. If the, the agent goes silence for, for a second or two, or maybe a bit longer, uh, the user might assume that something has gone wrong.

51. 10:28

    So in terms of our approach for the simple queries, uh, we got it to about one second, so we didn't need to do anything extra. Uh, but for more complex queries, when they, they're running a bit longer, three, four seconds, then we added injected filler words.

52. 10:39

    Something like let, let me look into this for you, let me look it up, uh, to maintain the conversation flow while we generate the answer in the background.

53. 10:48

    Then the thingy-- the second one is answer length. Uh, so again, on chat, it's actually probably desirable in those agents, uh, customer service agents to provide a bit of a longer response to, uh, uh, uh, provide as much context as possible to the user, and it's easy to skim through the answer to find the right information.

54. 11:05

    But again, it wouldn't work for on voice. You don't wanna wait there for a minute or two listening to, to the agent. So for more complex responses or for responses with multiple steps, we're breaking down the answers into multiple chunks and deliver it, uh, uh, uh, chunk by chunk.

55. 11:20

    And after each, we ask the user to confirm whether they would like to listen to the next step. And this works really well for something like troubleshooting when you have a few steps, uh, to follow.

56. 11:29

    And lastly, the user mindset. So what's something interesting in then during our early testing of real phone calls is, uh, some customers would interact with Fin Voice, like with an old school IVR.

57. 11:41

    So we'll be just using single words like support, uh, password reset, yes, no. But then throughout the conversation, I, I've listened to a lot of those calls, is like they change their behavior during the call and actually start using full sentences once they hear the agent using full sentences.

58. 11:57

    Um, and one of my colleagues like, uh, summed it up nicely that crazy how the human speaks more like a bot and the bot speaks more like a human. [chuckles]

59. 12:05

    Uh, and I think this will change over time, uh, to some extent it change over chats. And it's about, uh, obviously voice agents are getting better. And people more, being more, uh, common as a technology, people are gonna get used to it.

60. 12:18

    But for now, for now, it's on us to make those, uh, conversations sound as natural as possible so we help with this transition.

61. 12:27

    Now thinking about how Fin integrates into support workflows. Uh, so this was super important and definitely surprising for me, uh, when, um, when we got to this point, is that majority of the feedback wasn't about the voice, uh, about the model or about the latency, it was actually about how does it work with the support team workflows.

62. 12:48

    And don't get me wrong, I do think that all those core model experiences are super important, but this actually became a bigger blocker for all of those teams. Uh, so we put all our focus that the integration spo- points are as smooth as possible.

63. 13:00

    So we did a bunch of other things, but I'll just mention two here. One is the escalation paths. So getting the calls, um, uh, ge-getting configurations for how the calls get escalated to the human support team, and also the context handoff.

64. 13:13

    So after every AI agent call, we generate a transcript summary that gives a bit more context to the human agent that gets the call to what, what happened on the call.

65. 13:22

    And yeah, these are not like super flashy features, but they were absolutely essential to get this from this demo stage to the deployment stage for larger customers.

66. 13:32

    And then how do we know it's working? Uh, so, uh, there's a few topics I wanted to touch on. So, um, one is the manual and automated evals. Uh, so we had a test, uh, a set of, uh, test conversations that, that would be running through on every ma-major code de-- uh, code change.

67. 13:48

    Initially, it was mostly manual. It was in a spreadsheet, but over time we added some automation. Number two is internal tooling, and this was super critical, um, uh, for troubleshooting.

68. 13:58

    So essentially, we built some internal, just streamlit web apps to review the logs, the, the, the transcripts, the, the recordings. So any, any time one of our customers raise an issue, we can review in detail what happened in the conversation and actually troubleshoot it with the, with the logs.

69. 14:12

    Um, number three, resolution rate. So this is our North Star metric, which actually tells us whether we're delivering value for our customers. So we define it as, um, uh, essentially e-either the user confirming on the call that the issue was resolved, or the user disconnects after hearing at least one answer, um, and then doesn't call back within

70. 14:34

    twenty-four hours. Um, and yeah, this is the main, the ma-main metric, metric, metric we track. Obviously, there's more in customer service, but this is kind of the main success metrics that we have.

71. 14:43

    And lastly, LLM-as-a-judge. This is more experimental, but we're using another LLM to analyze the call transcripts to help us identify issues or opportunities for improvement.

72. 14:54

    And lastly, uh, how to price it. Um, so just wanna touch on the cost and, uh, some of the pricing models. The typical cost ranges between three and twenty cents per minute, and the cost here will depend on the, uh, the complexity of your queries, uh, but also on the providers that you, that you choose.

73. 15:12

    And in terms of the pricing models, so the, the main two dominant on the markets are usage-based pricing and outcome-based pricing. Probably usage is still the most dominant right now.

74. 15:20

    Uh, so user-based pricing, very simple. It's like per minute or per call, very predictable, but it doesn't necessarily capture the quality of the agent. Um, so, uh, the incentives are not very well aligned between the providers and the customers.

75. 15:34

    This changes with outcome-based pricing because you only charge if you actually resolve something for the customer. So it has a lot of benefits, but also it also introduces risk because for a very long call and the call that you unresolved, the, the provider actually needs to take the cost.

76. 15:49

    Um, so, uh, so yeah, so there is risk there, but over time, I do expect the market will converge toward outcome-based pricing because those incentives are way better aligned.

77. 16:01

    Cool. Um, and a few final thoughts. Um, so to recap, uh, we built a, a voice AI agent and shipped it in about one hundred days. We got several enterprise customers to use it on their, on their main phone lines.

78. 16:16

    And when I think about like some of the main takeaways from this ex-experience, um, is actually like getting to the right performance and those like latency and the, and the, uh, res-resolutions outcomes, obviously super important.

79. 16:29

    But it also is not just a model problem, it's also a product problem. So it's about picking the right use case, designing for the realities of those phone conversations, building the tools, both your internal and external, uh, integrating with the support team workflows and actually building trust with them because they ultimately gonna be decision makers whether they

80. 16:46

    wanna release it. Um, and yeah, and it's about making it feel effortless even if there is a lot of complexity behind the scenes. And that's everything for me. Uh, thank you very much.

81. 16:56

    And yeah, if you're building in this space, would love to chat with you. [audience applauding] [upbeat music]