How to talk to statues — Joe Reeve, ElevenLabs

Talk transcript

1. 0:00

   [upbeat music] Can I get a vibe check of the room?

2. 0:17

   How's everyone feeling? Are we like, "We really want to hear what Joe has to say about statues,"? Or are we like, "I just wanna like chill and just sit in a r- quiet room"?

3. 0:26

   How are people feeling?

4. 0:27

   Statues.

5. 0:28

   Statues. [laughs] Okay, that's good to hear. You're much more lively than statues, which I've spent a surprising amount of my, my life with recently, so. [laughs] Um,

6. 0:36

   this, I'm gonna, uh, well, actually, I'll quickly introduce myself. I'm Joe. I work in the growth organization at ElevenLabs. Hands up if you've... Actually, I've got a slide for that.

7. 0:45

   Hands up if you've ever used/heard of ElevenLabs. Okay. Um, so th- this next slide, you'll probably be familiar with a lot of it. Um, ElevenLabs does, basically we're an audio AI foundation model company, so everything from text-to-speech, you put some text in, you get some speech out, um, transcription, the other direction.

8. 1:04

   Music, we've got the first- [coughs] ... um, commercially legal, uh, AI music generation. You know, we licensed all the, all the training data, uh, behind an API. Sound effects. Create voices.

9. 1:15

   This one, by the way, just like pro tip in case you're ever using ElevenLabs at a hackathon or something, this section, creating and editing voices, is like the thing that people really just don't use ElevenLabs enough for, and this is the thing that the Statue app that I built, that we'll talk about, um, is built, built on.

10. 1:30

    And then agents. Uh, this is our fully managed agents deployment platform, which sounds like a mouthful of, of SaaS, and it sort of is, but it's also very cool in, in various ways.

11. 1:40

    So who saw the Statue app? It went quite viral, at least here in London. Raise your hands if you saw it. Okay, that's fine, because I'm gonna play you a video, 'cause, I guess people like me love playing videos- [laughs] ...

12. 1:52

    of our own voice out loud. Um, it's-- I'll just play the first 30 seconds or so. I made an app that lets you talk to any statue you want using AI.

13. 2:02

    So we've come to the British Museum to see how it works. [trumpets playing]

14. 2:09

    I am Pharaoh Amenhotep.

15. 2:11

    I am Demeter.

16. 2:12

    Hoa Hakananai'a.

17. 2:14

    I am Hans Sloane.

18. 2:16

    The Guardian Lion. I am the Young Rider.

19. 2:19

    And I am the War Horse.

20. 2:22

    First, take a picture. Okay, so now I'm about to explain that bit of me explaining. So we, uh, basically what this statue does is it lets you take a picture of a, of a statue, sorry, this app.

21. 2:32

    It lets you take a picture of a statue. Um, it then does a, an OpenAI deep research, uh, on, on the identity of the statue, generates a bunch of the sort of historical knowledge and prompts for what it thinks the, the, the voices of those individual statues would've been if they were alive.

22. 2:50

    Um, it then uses our Voice Design API, that really underutilized API where you can put in a description of a voice and it will go and generate something that matches.

23. 2:59

    And then it creates an ElevenLabs agent, and then starts a phone call. And that whole thing works in like 30 seconds. So you take a picture of something, you get all the like search research back from OpenAI, generate a voice, and start talking to an agent, to a statue within 30 seconds, which is pretty fun.

24. 3:15

    Um, what sort, uh, if you, if you're interested in like reading more about the details of how it was all built, you can scan that QR code. It's just a blog post.

25. 3:22

    Um, this was attached to the, the initial tweet that I made, um, which basically had the prompt for one-shotting. This whole thing I built in Cursor in two hours.

26. 3:33

    Um, it, it sort of, sort of was. So I, I built this, um, in, in two hours on a Sunday 'cause I was, I was sort of tired and bored, and then published the, the prompt through the ElevenLabs blog.

27. 3:44

    Made this video that you just saw, posted it on a Tuesday or something, and got 50,000 impressions. It's like pretty, pretty good, not bad. Um, people on, on Twitter kinda liked it.

28. 3:54

    And then three museums, or people who represent groups of museums, and a bunch of other like businesses, including like TripAdvisor c- competitors and stuff, who were coming and saying, "We've been"...

29. 4:05

    Well, actually, one of them, the CEO called me. He found my WhatsApp phone number somewhere, and called me up and said, "I've had a team of 10 people working on this for a year.

30. 4:12

    How did you build this?" Um, and so then the next day I, I reposted saying, "I've had a bunch of like interesting people. I vibe coded this in, in like two hours."

31. 4:22

    Not as a brag, just as a, this is interesting, like vibe coding is so powerful, and if you are sort of experimenting with these, these interaction patterns, you can actually do something quite big.

32. 4:31

    Surprisingly big. Um, it then went completely viral and went from 50,000 on the first day to one and a half million on the second day. Uh, and it was in part because it got kicked off by the vibe coding, and then suddenly I got all these artists and creatives and everything from, from portrait, uh, museums to like

33. 4:50

    Bonham's and Christie's reaching out saying, "We want to have people be able to talk to the items we wanna sell." Um, I think what I would sort of... Oh, and then, and then this led to something called ElevenHacks, which we can talk about later maybe.

34. 5:01

    Um, there are loads of different things in this story we can talk about. We can talk more about the Statue app and about ElevenLabs. I sort of wanna get, get more from you.

35. 5:10

    Um, we can talk about ElevenLabs generally. We can talk about what it means to do sort of growth, particularly API growth, uh, at one of these companies, um, from the growth engineering point of view.

36. 5:19

    And, and, uh, we can talk about the sort of implications on culture, which I think is something that we're not, as a, as an industry, we're not really looking at as seriously as, as we could be.

37. 5:29

    Um, vibe coding generally, and again, what that impact is having on society, um, voice interaction patterns or making viral videos or anything else. So it's sort of... Yeah, please.

38. 5:39

    I mean, it's really easy to prototype these days, right?

39. 5:43

    Yeah.

40. 5:43

    But it's like pushing it into production that's the hard part. So maybe, I don't know, I'm curious about how you would take your really successful prototype to like how can you scale it to millions of users maybe?

41. 5:53

    Yeah. Well, that's one of the things, uh, this is gonna sound very ElevenLab salesy now. The nice thing is that pretty much all of, like what I've done is stitched together existing APIs which are designed to scale.

42. 6:02

    So yeah, if I wanted to start doing user management, that sort of thing, that's relatively, I think, well understood, and you can u- buy in other third parties for that.

43. 6:10

    But the, the hard bit, the maintaining the agents and the voice design, that's all APIs that there's no way I can make a dent in the API volume, even if this, this goes absolutely gangbusters.

44. 6:20

    So from that point of view, I think, and I think that's something that vibe coding is really showing, is that the glue pieces and, and telling a good story about the glue is the im- in part the most important thing of the project rather than the solving hard technical problems.

45. 6:34

    So yeah, obviously, there's a lot more work to do to make it actually production ready, uh, which is something that I've been talking to a bunch of the, the museums about.

46. 6:41

    Uh, we might be doing that as like a nice ElevenLabs gives something nice to the, to the museums. But it's-- the hard bit is actually not the user management, and like Cursor, again, can pretty much...

47. 6:51

    You need to, to check it all, obviously. Can pretty much one-shot that with Supabase or whatever else for, for logins and, and magic links and that sort of thing.

48. 6:58

    It's, it's mostly the relying on our, our APIs and our, our agents platform to do the heavy lifting, I suppose.

49. 7:05

    What about evals?

50. 7:06

    Evals, I get-- So that's, that's one of the big things. I think the, the take a photo and just get your research back is not really the long-term solution for the, for the museums.

51. 7:15

    Really, the important bit, the hard bit is going to the curators and saying, "Curators, can you figure out what is the actual narrative? Let's not just take random things you found from Google.

52. 7:23

    Let's actually, like, put some thought in, in, in design to the content." So that's, I think, the piece that's the longer tail. The nice thing is a lot of the museums have-- they see their core IP as these databases.

53. 7:35

    Um, so they have APIs often, um, and s- we can pull that information out. The V&A has a public API for, for their, their stuff, so...

54. 7:45

    Yeah. I guess on this topic, um, maybe related to AI culture and voice interaction patterns, uh, how-- what is the interface then that you enable the curator to design the experience?

55. 7:59

    Is it Sandra or?

56. 8:00

    Yeah. So right now, there's no, no... I haven't designed anything for that. I mean-

57. 8:04

    Oh

58. 8:04

    ... at best, they could log into a, to a, a dashboard, the ElevenLab dashboard, and make, make edits to the system prompt and the knowledge base files that are in there.

59. 8:12

    Um, the-- I th- I think probably in, in this sort of d- information management, the best case or the best interaction pattern is probably editing text, uh, rather than speaking.

60. 8:24

    Although, obviously, if you're choosing a voice, you need to sort of manage that. There's an interesting question there. I've been talking to, uh, a chap, Jago, who used to be the head of the Americas at the British Museum and now runs the Sainsbury Centre, which amusingly is the location for the, uh, the Avengers headquarters in the Avengers

61. 8:42

    films. Um, he is going through this big, long academic process of figuring out what should a voice sound like for an inanimate object. So things like, where did the materials originally come from?

62. 8:54

    It came from some mountain in China, or it came-- And then, so it, it... The, the rock was shipped to Vietnam, and then it was carved in Vietnam, and then it spent the last two hundred years living in a British museum.

63. 9:05

    So like, what should it sound like? It would have a little bit of a maybe a Chinese origin with some Vietnamese twist in there, but then it's just lived around people with British accents, but maybe also not because it's, it's lots of tourists.

64. 9:15

    So actually thinking through from a, a much more philosophical point of view, what should objects sound like? And that's something that, from my point of view in, in, in ElevenLabs, is really interesting because I, I think we have the opportunity to give all sorts of things voices, like elevators, right?

65. 9:29

    A lift should probably-- I mean, they do have, have voices. They're often quite discordant with what a lift is, I, I find. But if-- It's quite likely, I think, that we start walking into lifts and saying, "I want to go to this floor, please," using voice to, to, to, to, to interact with them.

66. 9:42

    So what should they sound like? And it's beco- becoming a more important question. I, I don't know exactly what the answer is, but smarter people are doing that. So...

67. 9:50

    Sorry.

68. 9:50

    Sorry. Yeah.

69. 9:50

    Yeah. So it was re- really interesting to hear, uh, about your thoughts about, uh, voice as a, an interface to application.

70. 9:58

    Mm-hmm.

71. 9:58

    But in general, what, what, uh, problems arise? Uh, what do you think of how do users i- uh, uh,

72. 10:06

    interact or how, what, what do they expect from this voice agent, and when does it not work? Uh, what problems do we, do we expect if we are going to be embedding the voice interface?

73. 10:17

    Yeah. I think currently voice interfaces, voice interactions have quite a large range of problems currently, but a lot of them are solvable. One of them is you basically have a binary.

74. 10:28

    You're either interacting with voice, or you're interacting in some other way, and I still feel like the sort of interactive or generative UI plus voice is something that we still haven't seen.

75. 10:38

    Particularly, we've got... So sometimes it's, and this is something I've experimented with, is like if you think about a coding agent, you've got Lovable or something. I want to be able to talk to my app, talk to Lovable, uh, but not talk to the coding agent part.

76. 10:51

    I want to talk to like a product manager agent that then goes off and triggers my coding agent to go and do things. So what does that... So you-- The voice interaction there is not direct.

77. 11:00

    The, the thing that I'm talking to it or the thing that's doing the work is the thing I'm talking to. I sort of want to be talking to a, to a, a halfway house person.

78. 11:08

    So I don't-- What are the problems? Aside from often the thing you, you, you end up talking to is not the thing you actually want to be talking to.

79. 11:15

    There's also the, what are the parallel sort of interaction patterns, uh, in UI? And I think actually you were just showing me your app earlier, which does show the, the stuff that the, the voice agent's thinking and, and sort of extracting from the conversation and allows you to, to interact with that at the same time.

80. 11:29

    That's something that I, I think we're gonna see a lot more of, the sort of multimodal conversations where it's voice and visual. Um, the other thing is people don't like interrupting, uh, voice agents 'cause they're too polite.

81. 11:41

    People are too polite, and I'm starting to learn to just like interrupt agents much more aggressively, and that actually makes the experience much better. But we-- I don't know how to give people permission to interrupt.

82. 11:50

    Sorry.

83. 11:50

    How do you, how do you, uh, solve the problem of like prompt guidance or skill loading? So I mean, with a typical coding agent, you can give like skills, this, this, this.

84. 11:59

    You can throw anything at it, right? But I don't think it's the same interface or... Is it the same interface for-

85. 12:06

    So the, the ElevenAgents platform doesn't really support the concept of skills, though it could. It does support the concept of knowledge files, which do get loaded in. So you could do it in that way.

86. 12:14

    We also support MCP calling, so you can have knowledge em-embedded in those, um, or skills embedded in, in the MCPs. The, uh-

87. 12:23

    I, I don't think that's core to voice or not. Uh, I think that's mostly down to, like, the, the interaction patterns of, of coding agents are quite... l- lend themselves to skills.

88. 12:34

    Um, but you could have a voice agent that then has the ability to use skills, and you can add voice capabilities to an, to an existing coding agent that, that...

89. 12:42

    in, in that way. So I don't know if that's related directly to skills, but, um-

90. 12:46

    I think w- basically I want to talk to Claude code as a, as a human voice.

91. 12:51

    I see. I see. Well, actually, some people, some people have built the, particularly with OpenClau actually, there's ElevenLabs is, is quite a common interaction pattern for OpenClau. People have built phone, phone numbers they can call, and it'll call them back and that sort of thing.

92. 13:01

    Um, and then obviously you can, if you say, "Well, I want you to be able to load in skills," you, it just, like, learns how to do it and, and adds its, adds that capa-capability to itself.

93. 13:10

    So that's for sure possible. Um, and people are doing it with their sort of OpenClau setups and, and some Claude Code setups. But, yeah.

94. 13:19

    In, in this experience, one, the AI culture and the vibe coding-

95. 13:22

    Mm

96. 13:22

    ... thinking about new interaction patterns to engage with history, with our built environment, like, where or how do you see that starting to... Again, like, I mean, I mean, we're so empowered now with this vibe coding, but it's still-

97. 13:36

    Mm

98. 13:36

    ... there's a barrier, but how do we build that engagement and, and do it well? I don't know what the museums are telling you. I mean, I'm sure this is so new, but.

99. 13:44

    Yeah. I think, to be honest, the, the museums, I, I, I've met with the CEO of the science museum, or co-CEO of the science museum, and they're asking the same questions.

100. 13:54

     They don't really know the answers. They're saying, well... I mean, and, and they, the science museum, for example, is really good at going and trying stuff. So they've got all these big tablets that kids can go and interact with.

101. 14:03

     But in my personal opinion, a lot of the time that can sort of feel like sticking technology onto the, the th-

102. 14:09

     Yeah

103. 14:09

     ... rather than it being a, a core part. So some of the stuff we're experimenting with here is, obviously, you've got to take a picture of a statue and you talk to it.

104. 14:15

     We're commissioning a, a statue to be made that has the technology inside of it and a speaker and phone, um, and microphone, so that you can then talk directly to the statue without having a piece of technology in the way, without it feeling tacked on.

105. 14:27

     And that's something like with the, the red phone booth you may have seen here, um, on floor three. Uh, there's a, a phone you can pick up inside of a, of K6, um, red London phone, phone booth or British phone booth and talk to an agent, talk to, um, Sir Michael Caine.

106. 14:42

     So that's trying to put it into the real world a little bit rather than having it go through a screen. Um-

107. 14:46

     So I almost imagine, like, with vibe coding, like you're where you see kids making their own games.

108. 14:51

     Yeah.

109. 14:51

     Like, you imagine a kid go to a science museum, they have their tool-

110. 14:54

     Mm

111. 14:54

     ... and they just start creating whatever experience they want as they engage. Like that-

112. 14:59

     Yeah

113. 14:59

     ... it, it just sort of explodes the possibilities.

114. 15:01

     Yeah. Yeah. I mean, I think even more generally there's a question of what's the vi- vibe coding I feel like still hasn't really gone consumer mainstream.

115. 15:09

     Right. No, it's still, yeah.

116. 15:10

     It's like even, even Lovable sort of feels like it's targeted at consumers for building, effectively building B2B SaaS apps. You know? It's like have Supabase and, you know-

117. 15:20

     Yeah

118. 15:20

     ... standard, standard design components. But I... This is why I really love vibe coding, evening vibe coding events. They feel like the OG hackathons.

119. 15:28

     Mm.

120. 15:28

     Because people show up, and they have, they've never even thought about writing code before. Sometimes I go to them and I talk to people, and they're like... I say, "So what's your favorite app?"

121. 15:36

     Or like, "Who made the app?" They're like, "Wait, people make apps? I thought they were just there on my phone, or on the App Store." Right? They hadn't even thought through the fact that people have to make them.

122. 15:44

     So that's something that I, I find vibe coding events really fun because people come in and they type in. They have no idea what, they have no idea what a hamburger menu is or, or like a, an accordion is.

123. 15:53

     So they just say, "I want this and I want this, I want this," and they get something completely wacky because the LLM just says, "Yeah, okay, I'll try it."

124. 15:59

     Right.

125. 15:59

     Uh, when if they were talking to a software engineer, I would've said, "Ah, you want one of these and one of these."

126. 16:02

     Yeah.

127. 16:03

     Um, so I, I, I think at some point we're probably gonna have, like, what's the Instagram filters moment for vibe coding?

128. 16:09

     Yeah.

129. 16:09

     Or the TikTok moment for vibe coding. I think we're gonna have something like that that makes social vibe coding much more. But I d- I don't know what it's gonna look like, but worth experimenting.

130. 16:18

     Do you see anyone doing it well? Like in that sort of experimentation?

131. 16:21

     There's, there's Spielwork. There's sort of a, a, an app, a mobile app for vibe coding games, and it's TikTok swiping. Um, and there's, I think, I can't remember what it...

132. 16:33

     There's a London-based game vibe coding tool that is, again, focused on games. I don't know that games are really the thing because they're quite complex. Um, but it, the- there are a few people experimenting, but I don't think there are that many people really deeply pushing the boundaries of, of what's possible.

133. 16:48

     They're mostly like Lovable, but on your mobile phone.

134. 16:51

     Yeah.

135. 16:51

     Is my opinion. Um, does anybody else have any opin- any sort of, see anyone doing good consumer vibe coding?

136. 17:01

     Okay.

137. 17:03

     Vibe coding in what end? In terms of like creating things or copy or for their work or just in any, any sort of sense of-

138. 17:10

     I guess content creation

139. 17:11

     ... it seems like maybe vibe building or something-

140. 17:13

     Yeah

141. 17:14

     ... vibe working.

142. 17:14

     Yeah.

143. 17:15

     Vibe, you know, interacting with digital systems or firms, whatever it is. Like-

144. 17:19

     Well, well, well you, you, in your game, you can like spawn things in with voice, right? And that's like a great-

145. 17:23

     Yeah

146. 17:23

     ... it's not quite vibe coding, but it's still interacting and-

147. 17:25

     Because I think that's, that's what I... Like vibe coding still means you have an understanding of certain primitives. Like even what you just said-

148. 17:31

     Mm

149. 17:31

     ... thinking about databases. I think when this goes mainstream, you're not, people aren't thinking about it this way.

150. 17:35

     Yeah.

151. 17:35

     They're, they... And that's what's so exciting for me when you think about culture and what you build, is when it gets to a point where just things that us as engineers wouldn't even...

152. 17:45

     It's not even how we would approach the problem, and that's where some incredible creativity is gonna come.

153. 17:48

     Mm. So the, the pattern that I think is closest to being a winner in this space is the Facebook Instant Games API, which doesn't exist anymore.

154. 17:57

     Oh.

155. 17:57

     Um, they, they deprecated it, but it was in Facebook Messenger. You could play these games, and they, it had these primitives for social gaming. They, they tended to be quizzes or like Fruit Ninja, and you compete with your group chats and things.

156. 18:09

     I built, I bought for 15 pounds, um, a Fruit Ninja clone off a website, instrumented it with the, the Facebook Instant Games API, which was this beautiful like JavaScript async await.

157. 18:21

     Had a get user information, get friends, uh, create a leaderboard or get your position on the leaderboard. Um, some very basic data storage, key value storage. And, um- Async await at show an, uh, a rewarded ad and show an interstitial ad.

158. 18:37

     And so those things allowed you to make very quickly and easily a social graph enabled, ads enabled experience for consumers. So I bought this thing, 15 pounds, instrumented it with Facebook Instant Games API, and went to bed.

159. 18:52

     The next day, I woke up with 15 million users on this random... Yeah. I mean, it didn't make very much money, but there were like 15 million users in Vietnam.

160. 19:01

     Because obviously Facebook, they test everything out in, in the sort of lower value advertising regions and then rolls up. But that was an amazing-- Because you've got the social elements of people instantly just sharing it around, and that, I think, is probably the template that's gonna, something along those lines is probably gonna be the thing that wins

161. 19:17

     on, on the social vibe coding. But I don't know if that actually answers the question.

162. 19:21

     On the kind of like vibe working, vibe building kind of, uh, you know, thread, I kind of feel frustration whenever I get a, a response back in voice. Maybe it's just me, but like it's almost like the, the, the input, like the information density per second isn't quite high enough.

163. 19:36

     Like I-

164. 19:37

     Mm-hmm

165. 19:37

     ... I use a lot of voice out to just get things out of my head, like far, far from the time or whatever-

166. 19:41

     Or voice input, I guess is the, yeah

167. 19:43

     ... voice input. But then I, I still feel like I need, I don't know, diagrams or text or some like really high density, um, thing back from the system.

168. 19:51

     So I don't know whether, how you guys think about that, whether I should curiously get other people's views on that, whether they agree with me or disagree. It sounds like there are some nods.

169. 19:58

     But that's what I find myself leaning towards, where it's like information-rich input, and then I can just speak my thoughts as they come out, and it's almost semantically understood, put into that in-information rich format, and then my intent is like, you know, spawned out and, and across the system itself.

170. 20:13

     So that's the pattern that at least I'm seeing and feeling. And what I almost want to vibe interact with now, my email or like everything, is that I feel, I feel like that longing.

171. 20:23

     It's not quite there yet, but I, I, I, I feel like I, this inclination to build in that direction and to try to, um, get in-information rich, you know, get-

172. 20:33

     Response. Yeah

173. 20:33

     ... get that back, but also sort of like the input being very freeform and just my sort of raw intent.

174. 20:39

     Yeah. That's something that I feel. I absolutely feel. I, you know, I want to speak, speak easily and quickly, and then receive maybe a little bit of voice. But mostly there's like generated, maybe it's a UI, maybe it's just diagrams, maybe it's whatever app I'm in context of.

175. 20:53

     But yeah, I want to have parallel input, uh, parallel output, but like single input with my voice.

176. 20:59

     I can really talk to that. So I, I find that the, what, I don't actually get that much information necessarily from voice, but what I do get is like companionship.

177. 21:08

     Mm-hmm.

178. 21:08

     So it kind of like triggers that. It lessens the loneliness feel somehow if I'm like talking to something and I'm getting some information. I don't feel... I might be learning more if I'm looking at diagrams or text, but I, I don't feel as, I feel more motivated to kind of like continue tinkering.

179. 21:23

     So there's like some interesting modalities where you feel different things if you get the information coming in from all these different modalities, at least I find. So curious to hear how other people think about whether they, they think differently or whether that's something that they also feel as well.

180. 21:37

     Um, yeah, this is at least my point of view.

181. 21:39

     Well, the visual cortex is much older than-

182. 21:42

     Yeah

183. 21:42

     ... voice and, and text or whatever. So-

184. 21:43

     Yeah

185. 21:44

     ... seeing something is much-

186. 21:45

     Yeah.

187. 21:46

     Also in text-

188. 21:47

     Yes

189. 21:47

     ... when you ask it to be concise, I don't feel offended if it gives me a concise answer. But in speech, if it gives a concise answer, it's like, "Whoa, relax, man.

190. 21:54

     We're just asking a question here." [laughing] I asked it to be concise, and-

191. 21:58

     Well-

192. 21:59

     ... it just sounds rude.

193. 22:00

     That's, that's an interesting-

194. 22:01

     Yeah.

195. 22:01

     Um, may-maybe this is possible, maybe it's not. But what does, what does skim listening look like?

196. 22:06

     Yeah, exactly.

197. 22:06

     May-maybe, maybe like listening actually should also have two buttons, like forwards and backwards, and I can just tap, tap, tap, tap, tap, go forwards half a sentence until I'm...

198. 22:15

     I don't know. Maybe that's, maybe we should build that. Who w- who wants to vibe code something with me straight after this?

199. 22:21

     How would it work? So you'd be, you'd, you'd just easily, yeah, back, back and forward probably with, with audio. Is that kind of what you had in mind?

200. 22:27

     It's like a speed dial like on a podcast or on 2X.

201. 22:30

     Yeah. We a- or like the old iPods where you can sort of spin forwards and backwards. That's-

202. 22:34

     Right.

203. 22:35

     I don't know. That's, that's probably quite a nice listening interaction. Co- And you sort of, I guess, you want to scroll forwards in concepts, not necessarily in sentences, right?

204. 22:43

     Yeah.

205. 22:43

     Like what is the thing that your eyes look at when you're skim reading? It's probably, it's not the sentence structure, it's like the words, I guess.

206. 22:49

     Yeah, but like you move on to the next thing.

207. 22:50

     Yeah. Yeah.

208. 22:50

     You get that topic, move on to the next topic.

209. 22:52

     Yeah. Yeah. And that's actually, oh, yeah, this is, sorry, this is very interesting and exciting for me. Like, because if, if I'm talking to an agent and it just starts rambling about, you know, sometimes you get back three paragraphs of stuff, and I'm like, "No, not this one."

210. 23:04

     But I want to interrupt it and say, "Go to the next one." But then it's effectively saying it's a new prompt, right? So then it's effectively saying, "Yes. Okay.

211. 23:10

     I'll focus on that next one," and it'll write me three paragraphs about the second paragraph. You know, that's not really at all what I wanted. Unless I say, "Be concise," then it says something rude to me. [laughing]

212. 23:18

     Or could you like make a summary for each paragraph, and then like if you hold the button-

213. 23:23

     Yeah. Yeah

214. 23:24

     ... it will, it will expand and, or you can like...

215. 23:26

     Yeah. I think the Claude app has done some interesting stuff on the voice interactions because they, they show you something different to you here, and they show the higher level sections, and then it goes into each one, and you can tap on them.

216. 23:40

     So that's, I guess, getting a little bit closer to the... But I think that also means you're not interacting as though you would with a, a human conversat- I don't know.

217. 23:48

     Yeah. I, I, I'm thinking about that. Like why do we not have this issue when we're talking to humans, right? Like how do we-

218. 23:56

     Mm.

219. 23:57

     Well, there's so many other cues.

220. 23:58

     Yeah.

221. 23:58

     Like right there in-

222. 23:59

     There are visuals

223. 24:00

     ... there, there vision is like a cue. We have a sense. There's just so many other things.

224. 24:02

     I think there are visual cues as well-

225. 24:04

     Yeah

226. 24:04

     ... that you're lacking on voice. So that's something, 'cause it's like if I'm, as, as was mentioned before, it's like if there's an agent response coming in audio, I don't know how long that's gonna be.

227. 24:12

     Mm.

228. 24:12

     I just think what one was mentioned, like is it gonna be like, you know, a minute or is it gonna be ten seconds? I almost wanna know, and if I know it's like really long and waffly, I'm just gonna like cancel it.

229. 24:21

     I have, I have some other-

230. 24:23

     But also you can tell when I'm about to interrupt you, so you go faster, and you like move maybe... You, and you can tell if people are listening or not.

231. 24:30

     It's more like a phone experience, right?

232. 24:32

     Yeah. Yeah.

233. 24:35

     Because basically-

234. 24:35

     I guess, oh, there's another thing which is interesting here, which is the interrupting. Sometimes I don't want to interrupt. I just wanna say, "Yeah, yeah, yeah," or, "Oh, but no, go back."

235. 24:43

     And like, it's like you c- you're always listening for the, "Uh-huh. Yeah, yeah, yeah." But you can't do that with an agent.

236. 24:49

     I think, I sort of, I'm building sort of, you know... I mentioned this to Joe before, but, but briefly, she, I showed you this yesterday, but briefly, the, the ability to sort of interact with a, sort of a PS5 game, and also create what you want in that world as you're playing it.

237. 25:02

     So you're, you know, you're on like a, you know, a shooter battle. I'm playing you. We're trying to like, you know, kill each other, you know. Apologize language, but like if I'm, if I'm like trying to be creative and, you know, generate a new getaway vehicle or, or a helicopter, I could just sort of say that experience

238. 25:18

     and have that appear in the game straight away, and now I can sort of fly away. So that's sort of the work that I'm doing and how I met Joe, actually.

239. 25:24

     But I think the, the challenge that I faced is that relying on just the, like, voice interruptibility is quite unreliable. So I've just got round this with just a simple whisper flow type-

240. 25:34

     Push to talk type. Yeah

241. 25:35

     ... where you push to talk, and then ho-hold to talk, and then let go to finish, which it, it augments, like, purely this audio stream with some other cue.

242. 25:43

     And in the same way that I think you almost need some very light nudge interface on top of the audio that you're receiving. And maybe as you say, like, you know, you're saying something, and then you see a little ye- um, sort of like circle appearing, being like the agent wants to ask you a question.

243. 25:59

     That would be an interesting experience to feel, right? You're, you're, you're not being interrupted-

244. 26:02

     Mm.

245. 26:03

     ... but you're being kind of like, you know, the agent wants to talk, and then maybe you either stop or say, "Okay. What, what, what, what idea do you have?"

246. 26:09

     Because I even can feel you doing that to me right now, Joe.

247. 26:12

     Yeah.

248. 26:12

     You, you want, you want to say something.

249. 26:14

     Well, I'm just getting loads of ideas. This is great.

250. 26:16

     Yeah. I just sort of like... I, um, but we're, we're almost like having an information, you know, communication on that level.

251. 26:21

     Yeah.

252. 26:21

     But in audio, if I'm just listening to audio, I'm not, I don't think I'd, I'd know that as if I was an agent or if I was just like, just listening to the audio, not looking at the, at the, at the video stream.

253. 26:30

     Okay.

254. 26:30

     I'll let you talk. Sorry. [laughs]

255. 26:32

     Well, I'm just imagining, or I'm j- just imagining on that point that the agent wants to respond to you. It could be showing, "I want to interrupt and t- tell you about this thing or this one."

256. 26:39

     And, and then suddenly that becomes what we've just done, but with even more context than doing it with a human because it's signaling the topic it wants to talk about.

257. 26:48

     Yeah.

258. 26:48

     Yeah.

259. 26:48

     And-

260. 26:49

     I have just, just one question regarding the product. Because regarding this topic, uh, you need some sort of, uh, um, tool talk, tool calling in the background to be able to-

261. 26:59

     Mm

262. 26:59

     ... actually understand that I want to interrupt. Uh, and, uh, and, uh, how would that, uh, tool call be, uh, handled? Will you, will you sort of be streaming the audio directly to back of the RTC to the, to the client, or through a back channel and then, uh, getting some information?

263. 27:17

     Do you update the, the system prompt? Or-

264. 27:19

     Well-

265. 27:20

     How can you work that?

266. 27:21

     Obviously, this hasn't been, as far as I'm aware, hasn't been built yet. The way I would probably approach it is looking at the transcript, and just keep analyzing the transcript over and over again, and say, "Do you have anything to, to add?

267. 27:30

     Do you have anything to add? Do you have anything to add? Or what would you add?" Um, rather than being p- a tool call or being part of it, I'd do this as an asynchronous looking at the transcript.

268. 27:39

     Are you, are you changing original prompt? Because the original prompt has sort of a plan. You want to do this. Um, and halfway you sort of change the plan, and you go in and sort of...

269. 27:49

     How do you steer the original a- agent then?

270. 27:52

     Well, this is actually an interesting thing with agents. They... Often we see agents as things that you can't really interact with the internals of. But effectively, an agent is, is the sort of logic that says, "What's the next message?"

271. 28:05

     But it's also a transcript, and that transcript of the conversation is completely malleable. So maybe some other thing we could experiment with is actually allowing the ability to do the interruptions where it's talking, and then I can say, "Uh, uh, uh, yeah, yeah, yeah."

272. 28:18

     And currently, if, if I have the, it, the way most agents platforms will work is it'll generate its full text thing and then start generating the audio. And then if I talk, even while it's partway through, it's still reading out the audio, uh, the, the text.

273. 28:32

     It will just append my message to the end of that full message. But we do have the timestamps. We know how far through the, the audio is played. So we could actually just go and edit the transcript that's coming back and say, "Well, no, they interrupted at this point, so we're gonna forget that the LLM even generated

274. 28:45

     more text."

275. 28:46

     Yeah. So if you sort of, you have, you have the system message and then the user message.

276. 28:49

     Yeah.

277. 28:50

     And sort of you need a, sort of a second type of user message that you are pro- uh, are passing in. I, I certainly would take the, of course, it's a user message, but-

278. 28:59

     Yeah

279. 28:59

     ... it's also some other kind-

280. 29:02

     Yeah

281. 29:02

     ... of steering here. Yeah.

282. 29:04

     Yeah. I mean, why don't you... Let's go to the ElevenLabs booth at the expo floor and just vibe code something and see. Like this, this is great. I, I can't wait.

283. 29:11

     Now it's getting easier and easier to make tools, and there's so many different ways to execute your ideas. And it seems like the main kind of differentiator now is getting your word out there.

284. 29:21

     Like, how did you approach making those videos, and like, um, how long did it take you for the statue video-

285. 29:27

     Mm

286. 29:28

     ... to make it?

287. 29:29

     That's a good question. Totally se- totally separate.

288. 29:31

     Separate, yeah.

289. 29:32

     Um, so this is sort of the inspiration of this ElevenHacks thing. Um, I learned to make videos. I'm not particularly good at it, you know. It's like still relatively janky.

290. 29:43

     Um, I learned to make videos through doing other, like, politics related campaigning stuff before I worked at, before I even knew ElevenLabs existed. Um, the, it turns out that with videos, random things go viral, or with content, random things go viral and random things don't.

291. 29:58

     Like, things I think are gonna go viral don't, and the things that do, do. Um,

292. 30:03

     I think it's basically about practicing. Like, uh, it's, it's, editing a video, I find, to, it's like the t- 80/20 rule. Editing a video to the standard of the statue app is actually quite relatively easy.

293. 30:16

     But then to go beyond that, you suddenly-- 'Cause I edited that on my phone. The editing itself took about 20 minutes, 25 minutes. Going beyond the quality of that suddenly means thinking about using a desktop editing tool, which suddenly makes everything...

294. 30:31

     Even using CapCut on desktop, to me, is like three times harder than using it on my lap- on my phone. And it's also three times more expensive, the subscription on the laptop than on the phone.

295. 30:39

     So I think a lot of it is just- Doing stuff and trying and iterating. Big things that I found, adding captions really help, um, to the video. Having a hook in the first...

296. 30:50

     You can, I mean, you look at your various platform, um, uh, analytics once you've posted a few videos. My videos tend to get between 6 and 12 seconds is the aver- the median view time, and then people drop off.

297. 31:04

     So you need to get your hook in there because most people are gonna drop off if they don't buy the hook. Um, so that's, that's an important piece, front-loading the, the, like, interesting piece.

298. 31:15

     Uh, adding music makes a massive difference, and this is something that was, was much harder, but now with ElevenLabs music generation, I will make the video and then add...

299. 31:26

     Sometimes I'll make the video with the narrative and, and everything, and then just experiment with completely different genres until I find music, and I'll just put the music on.

300. 31:32

     Does it work? Yes. No. And then you can edit the, the different sections so that it times up with the sections of the speech, so you don't need to, like, figure out mu- music first, find a piece of music, and then match your speech to it.

301. 31:44

     The other way, sometimes I will have a vibe I want to get across, and I will generate the music first and then figure out what's my speech that matches that vibe.

302. 31:54

     Um, so it might be like an, uh, uh, an excited theme or a, a... I, I actually chose the music for, for the statue app before I, um, before I made the, the video itself because I thought this is a fun piece of music for a statue type thing.

303. 32:10

     It's like a bit of im- imperial, outside the British Museum. It sort of made, made sense as a, a- an attention grabber. So, but I think music is a massive thing that people underrate, um, 'cause you just put it...

304. 32:20

     It's relatively quiet, but it makes a massive difference to the feeling.

305. 32:23

     So you just did it on CapCut, and did you think-

306. 32:26

     It's literally on my mobile phone. I've got-- I borrowed my wife's lapel mic. [laughs] Bluetooth la- lapel mic, which cost 200 quid from DJI, um, which makes the audio much better.

307. 32:35

     And then, uh, yeah, just edited on CapCut. Super, super simple.

308. 32:39

     So you didn't think about, like, this is the shot I want, and then, like, I, I wanna take this photo or...

309. 32:45

     Um, I mean, the, the, the f- the one out the front of the British Museum, yes. I... Thank you. Um, yes, I did want that one. But the... And it was actually the second time I recorded the video.

310. 32:57

     I, I went once and got some stuff that was, like, a bit boring, and then I went back and just took a bunch of silly photos, and that's what ended up being the video.

311. 33:04

     So yeah.

312. 33:06

     Very good.

313. 33:06

     Thanks.

314. 33:09

     Yeah.

315. 33:10

     Yeah. Cool. Thank you very much.

316. 33:12

     Thank you. [upbeat music]