Cooking with fire without burning down the kitchen

Talk transcript

1. 0:00

   [upbeat music] Thanks everyone for joining. Let's start with a quick raise of hands.

2. 0:16

   How many of you in the last two years got into AI?

3. 0:21

   Okay. That's a, that's a pretty good amount. And how many of you were in the room when your boss was like, "Okay, so what are, what is our plan about AI?

4. 0:28

   What do we do about AI?" All right. That was, that was pretty much me last year. Um, so in the next twenty minutes, time, time permitting, should be exactly twenty [chuckles] minutes.

5. 0:40

   Um, we're gonna talk about our team at Twilio, our history, lessons we've learned aro-around, uh, you know, our team and how we operate within the company, the role we play, and to ensure sort of that Twilio can keep up with the rapid pace of AI.

6. 0:56

   Or in other words, how do we, you know, cook with fire without burning down the kitchen and, uh, upsetting our customers? Um, as Peter said, my name is Dominik Kundel.

7. 1:07

   I lead product and design within a small team at Twilio called Emerging Tech and Innovation. Um, we're a team of 16 people, uh, across engineering, product design, and go-to-market, and we try to operate as self-contained as possible, um, outside of our two more traditional Twilio business units around communications, that is more focused on sort of the APIs

8. 1:27

   that Twilio is known for around SMS, voice, email, et cetera. And then our customer data platform that is more focused on providing a holistic, uh, overview of your customer, so you can appropriately engage with them.

9. 1:40

   And our team itself is focused on exploring what the future of customer engagement is gonna look like, um, with an eye towards emerging technologies like AI.

10. 1:49

    But we're not the AI team within Twilio. Um, [chuckles] at Twilio, we sort of see AI as a feature that spreads across all of our products to help engage customers to better-- Uh, it helps our customers to better engage with their customers.

11. 2:06

    And, um, you might have seen sort of our, like, customer AI billboards around. That is really sort of the vision around that is, how do we use AI to better empower people u- uh, to engage with their customers?

12. 2:19

    So we have innovation in the name, though, so are we the only ones innovating at Twilio then? Um, the answer is also no. Um, generally, the every-- it's everyone's job to innovate.

13. 2:30

    But I think there's two types of innovation. If you've read The Innovator's Dilemma, this seems familiar to you. But basically, uh, there's sort of the sustaining innovation and the disruptive innovation.

14. 2:40

    Sustaining innovation is really teams trying to in-innovate on top of their existing products and their existing customer needs. That is sort of often more the iterative one. It's like Sony trying to make a mirrorless camera better and, like, really innovating in that space.

15. 2:55

    Um, while disruptive innovation is often initially in its in-infant stage, much worse in quality. It's actually a characteristic of disruptive innovation is the, uh, poor quality that often does meet the need of a niche audience, though.

16. 3:11

    So think about cell phone cameras when they initially came out. Nobody would leave their camera at home, go on vacation and be like, "I'm gonna take pictures on this, like, Motorola."

17. 3:19

    Um, but these days, like, most people don't even have a camera, a camera anymore. They... Like, I just went on vacation and only took my phone. Um,

18. 3:28

    the exciting thing about AI is that innovation is really happening on both ends of the spectrum. We see both sort of augmentation of existing workflows, like, you know, Photoshop's Generative Fill.

19. 3:42

    Um, a lot of what Apple announced was baked into exi-existing products, right? It wasn't a, "Hey, here's a new AI thing," but it was more built into the features.

20. 3:51

    So that's more on the sustaining innovation side. While I would say agents especially, and GPTs in general, are more on the disruptive side. Um, because the reality is agents are not ready for enterprise prime time, um, primarily because the quality is not there yet, right?

21. 4:06

    Like, we've, we've all seen sort of, uh, agents going rogue, partially even RAG-based chatbots, like, not even agents. Like, you know, hallucinating things, like the example of selling, you know, a Chevy Tahoe for a dollar or, you know, Air, Air Canada's chatbot making up a policy.

22. 4:23

    And so because of all of that and overall the cl- uh, lack of clarity in the regulatory space, often within enterprises, the adoption right now is dictated by, uh, legal more [chuckles] than product teams.

23. 4:37

    Um, and so overall, we're seeing sort of a lack... Like, like, it's not ready for the prime time. The other aspect of it is, um, for a lot of enterprise, they see AI as a cost-saving method, and agents are not cheap yet, right?

24. 4:51

    Like, we can do increasingly better, like, RAG-based chatbots, um, for low cost, but, like, agents are still not in the cheap level if you want high quality, and so they're at odds there.

25. 5:03

    The difference, though, is for startups and SMBs. They're really your niche these days where for them, they actually see agents as a way, as a value add, not as a cost saving.

26. 5:13

    They already know that their quality of customer support and, and sales and stuff is not where you want it to be. And so for them, it's much more an opportunity to really add value 'cause, you know, it can't be worse than the current experience.

27. 5:30

    So why am I talking about the difference between these two types of innovation? Um, it's very natural for a business actually to focus on sustaining innovation, especially if you're the incumbent in the market.

28. 5:40

    It's, you already have an established customer base. They're already paying you, so that's great. Um, the problems are well-defined, and so overall, it's much more predictable and you can, you hopefully you understand your target customer.

29. 5:53

    While for disruptive innovation, it's sort of really the opposite and you don't know yet what it's gonna be, and, like, you often are not gonna make money initially. Um, but ignoring disruptive innovation right now can be more dangerous than ever before because the biggest difference between sustaining and disruptive innovation is the quality, and the quality changes every

30. 6:12

    day, right? Like, every day right now with AI, there's massive leaps, and so something that wasn't quite there from a quality perspective yesterday can be there tomorrow. And in a similar way, a company that is not your concern tomo- yesterday can be tomorrow.

31. 6:26

    I think the best example of this is the impact that ChatGPT and Perplexity and similar things had on, uh, how Google approaches search, right? Now we have AI Overviews and things like that.

32. 6:37

    Um- And so our team right now is focused on this disruptive innovation aspect. That's sort of where our innovation as- uh, perspective comes from, is we want to understand what is going to be disruptive to Twilio and w- how do we, um, prepare for that, react quickly, or be the disruptors in the space and inform the rest

33. 6:58

    of the company about our learnings. But that wasn't our original agenda. Our original agenda was we were founded as a three-person skunkworks team last year that was really focused on w- uh, special projects and trying to rapidly prototype the bridge between our customer engagement-- uh, like our customer data platform and our communications platform to create this customer

34. 7:21

    engagement flywheel that I was talking about, and to do this in a way of quickly wrap-- uh, quickly prototyping solutions, showing them to customers, getting feedback, understanding what is their vision, where are they gonna go, not just their current problems, and then iterate on that.

35. 7:37

    And we found GenAI was actually a natural co- like fit for us,

36. 7:43

    uh, because the biggest thing about customer engagement is you're sort of sitting between unstructured communications data and structured customer data, and nobody wants to chat with a JSON object and, like, your unstructured data is not very helpful to-- for analysis.

37. 8:00

    And really large language models are perfect to bridge this. They're great at creating this connection and translating structured to unstructured and vice versa. And so as a result, we dove into pro- uh, prototyping our first two solutions in that space, and that's what got us into the AI space.

38. 8:18

    Uh, we had conceptualized essentially two features that we showed to customers, iterated on, and ultimately handed off to R&D to take to market. Um, one of them was the AI personalization engine, which is a RAG on top of this customer profile that is within Segment, and then the AI perception engine that was more focused on how do

39. 8:38

    you take communications data and then translate that into a customer profile, because within a conversation, there's so much rich information that you want to remember about a customer, right?

40. 8:49

    And sometimes a human agent will take notes somewhere, but I realistically want to have one central spot. So we conceptualized these two systems, and sort of together they make this customer memory.

41. 9:00

    Um, but as we showed them to l- leadership and they handed them over to R&D, we did hit our first challenge, and that was that we had conceptualized something that wasn't sustaining innovation, it was disruptive innovation.

42. 9:12

    It was, uh, a problem that where the quality was not quite there, the cost was not quite there, and primarily we're solving a niche use case. Like, what we had made up was, uh, primarily helpful for this up-and-coming wave of agents and generative AI solutions in general, but there wasn't a large market for this yet.

43. 9:32

    And so co- even though we had great feedback, we didn't really have the momentum that helped us to prioritize this among the sustaining innovation and, uh, profitability focus that our R&D teams had.

44. 9:47

    And so we learned from that that even though our customer obsession was key because it helped us understand that we were on the right track, we knew from talking to them that this wasn't just a problem they were trying to solve today, it was a problem that they were going to solve, uh, tomorrow.

45. 10:02

    What wasn't working was we had to build ideas out further to be able to gain that traction and, uh, be able to hand it over to R&D.

46. 10:11

    And so we started working on our next project called AI Assistants, which is an agent builder that is built on top of this customer memory concept, um, and is designed to really allow you to build omni-channel customer engagement chatbots.

47. 10:25

    Uh, and we started with the same principles. We tried to really rapidly prototype, quickly get something into the ha-- uh, like to show to customers, get feedback, get a working demo, continue to iterate on it as we're getting feedback both internally and externally.

48. 10:41

    But we did a couple of things differently this time. One, um, we decided we have to get this into the hands of people as soon as possible. What we had built previously was primarily demo-driven.

49. 10:50

    It was, what can we show to customers and get feedback based on that? And so we changed our mode and ran internal hackathons, tried to give it, uh, give really rough-- access to really rough prototypes with all the rough edges to customers coming into the office, and we're like, "Here's for one day you have access to this

50. 11:08

    prototype. Let us know everything that is wrong with it and what you want it to do." Um, and this was both terrifying and incredibly, uh, you know, uh, insightful for us.

51. 11:19

    And, uh, additionally, we tried to find more opportunities to do dogfooding. We tried to find ways we can solve problems internally. We started with a low u- uh, low-risk use case around IT helpdesk.

52. 11:30

    We knew it wasn't the perfect target customer that we would expect to buy our solution, but it was a way for us to gather data and see what the quality challenges were and how we wanted to structure things.

53. 11:42

    And then we started to establish an engineering organization that would actually be able to set the foundation for us to actually take this into the hands of people. But that brought us to the next challenge because at Twilio, we've worked hard over the last couple of years to really build the foundation to, um, in our software development

54. 12:01

    life cycle, to really create trust with customers in what we're launching. And GenAI really changed how you develop AI products, right? It's actually exciting but also tricky for larger companies because GenAI turned sort of the entire product development life cycle on its head.

55. 12:19

    Where in the past you had to gather a lot of data, you had to make sure you had enough data to train a model, you train the model until you reached a certain level of quality, then you released it, and then you started optimizing.

56. 12:31

    These days, you can actually use GenAI to build a prototype or often an MVP very rapidly without having that data and release it out to some customers to get feedback and then continue to iterate on that.

57. 12:45

    But that puts us into a dilemma because we have a system now that we know is not quite there from a quality perspective, but we need to get it in front of people to make it better.

58. 12:57

    And so how do we do that, and how do we keep up with the pace of technology without really upsetting customers with something that doesn't meet our normal quality bar?

59. 13:08

    And so that's where we looked around the industry for other examples. Uh, the two that came up for us were GitHub Next, uh, the creators of GitHub Copilot, and then, uh, Cloudflare Emerging Technologies and Incubation that created things like Workers and, uh, D1 and Workers AI.

60. 13:23

    And both of these were really focused on setting the appropriate expectations, uh, arou- for users around reliability, capabilities, and availability. And so our solution for this was to create our own sub-brand called Twilio Alpha that was very focused on, um, you know, setting the right expectations with customers, but enabling us therefore to ship early and ship often,

61. 13:46

    engage the interest from customers outside of those that we would have to otherwise track down, and enable customers to come to us. Um, and so that's learning number two, to ship early and ship often so that we can regularly, like, get rapid feedback from customers.

62. 14:00

    And the slides are off now.

63. 14:02

    Oh, no.

64. 14:02

    Yeah. Do they just stop? Yeah, literally, like last slide.

65. 14:12

    I have a screen.

66. 14:12

    If someone's asleep, usually we push it awake. It's happened again the last few times.

67. 14:20

    Mine still thinks it's connected.

68. 14:24

    There we go. One.

69. 14:24

    Nope, now it's gone.

70. 14:28

    Yeah. There we go. No. Yeah.

71. 14:39

    Yep. Cool. Um, so that got us to our second learning about shipping early and shipping often, um, because by having a sub-brand, it allowed us to open wait lists for developer previews, show our offerings, and onboard people quickly, as well as have internal POCs to learn while we're onboarding people.

72. 14:59

    But everyone talks about shipping fast, so, like, you know, how fast, like, es- especially with sort of the, the rapid pace of AI, like, how do we ship fast enough?

73. 15:08

    And the honest answer is we haven't figured out yet what is fast enough, but we put some base principles in place that help us move fast. One was around how we grew the team.

74. 15:17

    As we started hiring people, we focused on cr- uh, curiosity and creativity because at Twilio, we've always valued the creativity of developers to solve not just coding, but business problems.

75. 15:31

    In fact, our co-founder, Jeff Lawson, wrote a whole book, Ask a Developer, about it. And so for us, using that, uh, like, having natural curiosity within the team was more important than existing AI experience because we wanted everyone to be able to solve problems rather than relying on product to figure out what is the thing we're building.

76. 15:49

    We wanted to give problems to c- uh, to the engineers and be able to figure out what we're gonna build. Uh, the other thing we put in place is flexibility, both in terms of our systems.

77. 15:59

    We assume that every model that we're using is gonna be redundant tomorrow. That doesn't mean that we're jumping on every model that comes out in every paper, but instead, we understand what are our known limitations and how do we-- and every time a model comes out, how do we quickly validate whether the next model is something that

78. 16:18

    helps us in that moment? So for example, when 3.5 Sonnet came out, we were able to figure out within a day whether we wanted to pay any attention to this for now or whether we're putting it in the backlog.

79. 16:29

    The other side is flexibility of roadmap. Having a new team that is specifically focused on innovation helped us not have any pre-existing commitments to customers and allowed us to really have flexibility about the roadmap, and we try to defend this flexibility as much as we can with, uh, with c- uh, current initiatives around this.

80. 16:49

    The big part around this I would recommend is to understand that expectations are shifting at any moment currently, meaning figure out what are expectations that are staying, because those are worth putting on the roadmap.

81. 17:02

    Those that are ever-changing, you need to have more flexibility around. Think about the difference between trust and safety, where people are, uh, you know, always gonna be focused on versus multimodality, where, um, every time some new mo- a new mode comes out, people are gonna change what they want it, uh, want to have.

82. 17:19

    Like, most people don't talk about image these days anymore, but, like, that was the hot thing like a few months ago. Now everyone talks about voice. Um, so you will constantly have to iterate on your roadmap, and you have to be okay with failures.

83. 17:32

    Um, but if you're okay with failures, especially in the environment right now where everyone in the company, in most companies, like, has to provide value at all times, how do you provide that value i- uh, while you're waiting for your eventual home run?

84. 17:47

    Um, that's where we made sort of the last mistake that I wanna call out, where, as I mentioned, our team is self-contained within the company, um, including having our own go-to-market.

85. 17:58

    And in the first year, we took that self-containment a bit too far. Um, we basically just operated in our own corner quietly and shared what we were working on on a need-to-know basis, both with customers but also internally.

86. 18:11

    And that resulted in, uh, often teams not even knowing that we existed as a team or what we were working on, and it was on us to find potential conflicts or collaboration opportunities.

87. 18:22

    And at the same time, our customers were having the same questions that we had around what are we doing about AI, you know? And it really, especially for a B2B company, you have the opportunity to help your customers be thought leaders in their space and build a stronger bond.

88. 18:38

    So our third learning was to share things as we go, both internally and externally. Internally, to find opportunities to share all that enormous learnings, the, the enormous learnings that we had with others to empower them, but also externally to enable others be, to be thought leaders in, in their space.

89. 18:58

    So these are our biggest learnings from the three, uh, you know, from the last year, and we actually ended up turning them into our base principles for the team.

90. 19:06

    So being customer and developer obsessed to make sure that we talk to customers early and often and understand not just, you know, their current problems and their current problems with Twilio, but understand their business, their challenges in general, and their vision for customer engagement so that we can anticipate their future needs.

91. 19:24

    Uh, shipping early and often so that we can set the right expectations, um, with them but still get things into their hands and get feedback from them. And creating a team that is curious and owns problems so that we can actually build things quickly.

92. 19:40

    And then lastly, share as you go, uh, share, share as we go, so we're both sharing internally as much as we can, but also sharing externally, um, to help our customers be thought leaders.

93. 19:50

    And with that, um, thank you so much for your attention. I will put, like, the slides are already on that URL if you wanna check them out. As Peter mentioned, I will have to run to the Twilio booth to raffle away a prize.

94. 20:03

    But if people do have questions, please come to the Twilio booth, and it's, like, literally over there. It's the only one that is red. Um, [laughs] and I'm happy to answer any questions there.

95. 20:11

    But thank you so much for having me, and have a great day. [audience applauding] [upbeat music]