Building voice agents with OpenAI

Talk transcript

1. 0:00

   [upbeat music] Awesome.

2. 0:16

   Well, hi everyone. My name is Dominik. I work on developer experience at OpenAI, and I'm excited to spend the next two hours to talk to you all about voice agents.

3. 0:26

   Um, the QR code was already on the slide. If you just entered the room, you wanna try to download the dependencies as soon as possible. So head over to that QR code or to that starter repository and follow the instructions to install.

4. 0:41

   That might take a while with the internet right now, so please do that as soon as possible. You have, like, fifteen minutes of me rambling about stuff before we get started with, uh, with actually coding.

5. 0:53

   Um, so I said we're gonna talk about voice agents. I wanna first put everyone on the same page because I know we all have different, uh, definitions of agents, and there's gonna be a lot of definitions flying around at this conference naturally.

6. 1:04

   Uh, so when we're gonna talk about agents, we're talking about systems that are gonna accomplish tasks independently on behalf of users. And most importantly, um, they're gonna be, uh, essentially a combination of a model that is equipped with some set of instructions that then has access to tools that can both be used to work on that goal.

7. 1:23

   And then all of that is encapsulated into a runtime to manage that life cycle. And that's an important definition because, uh, today we launched the OpenAI Agents SDK for TypeScript.

8. 1:34

   If you've heard of the one for Python, today we basically released the TypeScript equivalent. Um, and so we're gonna use that, and it maps those exact patterns. So if you're unfamiliar with the Agents SDK, it's basically a SDK that provides you with an abstraction based on the pr- best press-- practices that we learned at OpenAI to build

9. 1:55

   agents. Um, and they-- it comes with a couple of different base foundational features, including things like handoffs, guardrails, streaming input and output, tools, MCP support, built-in tracing, so you can see actually what your agents did and how they interacted with each other.

10. 2:11

    And then additionally to those features that are coming from the Python SDK, the SDK we launched today in TypeScript also includes human-in-the-loop support with resumability, so that if you need to wait for human approval for a while, you can deal with that.

11. 2:26

    And most importantly, native voice agent support. What that means in practice is you can use that same-- those same primitives that we already have in the Agents SDK, and you can build voice agents with that that handle handoffs, have output guardrails to make sure that the agent is not saying things it's not supposed to, uh, tool calling,

12. 2:43

    context management, meaning keeping track of the conversation history, so you can use it in other applications, and build and traces support, um, so that you can actually replay conversations, listen to the audio of the user, and properly debug what happened.

13. 2:59

    Plus native interruption support. If you've tried to build interruptions, you might know how hard this is. If you haven't, be glad you don't have to. Um, both WebRTC and WebSocket support, meaning it's actually can communicate both on the b-- uh, on the server for things like Twilio communic-- uh, like phone call, voice agents, or directly in the

14. 3:21

    client in the browser. That's what we're gonna use today using WebRTC.

15. 3:26

    But first, why, why would we be interested in voice agents in the first place? Um, one of the things that I'm most excited about is it makes technology much more accessible to people.

16. 3:38

    There's something magical about being able to talk to a voice a-- uh, to, like, a voice agent and just have it, uh, like, see it do things. It's also, um, much more information dense.

17. 3:50

    I can convey information much faster, but also it can contain a lot of information through the type of tone and voice that I'm using, the emotions. So it's much more information dense than sort of just base, base te-- uh, basic text is.

18. 4:05

    One of the cool things is also it can act as like an API to the real world. You can have a voice agent go and, like, call a business for you and, like, have a conversation with them where maybe there isn't A-- uh, there isn't an API for that business.

19. 4:20

    And so when we talk about building voice agents, there's essentially two types of architectures that have emerged on-- when building these. The first one is based on your traditional text-based agent and just sort of wrapping it into a chained approach where we have a speech-to-text model that is taking the audio and then turning it into text so

20. 4:40

    that we can run our basic text-based agent on it. And then we take that and we, uh, run that agent, take the te-- uh, text output, and run it through a text-to-speech model to generate audio that we can play again.

21. 4:55

    This has a lot of strengths. One of the most common reasons why we raise this is it's much easier to get started with if you already have a text-based agent.

22. 5:06

    You can take that, wrap some audio around it, and you have something that you can, you can interact with. But, uh, eh, the other aspect is that you have full access to any model.

23. 5:16

    Text is the main modality that any LLM has, and so you can use really any of the cutting-edge models. It also gives you much more control and visibility of what the model did by being able to actually look into the exact text that went in and out of the model.

24. 5:33

    But it also comes with some challenges. Turn detection is one of the, uh, one of the big ones where you need to now take into consideration what did the user hear by the time that they interrupted some-- uh, interrupted the voice agent, then translate that part back into text, make sure that, like, your transcript is appropriately a-adapted

25. 5:54

    so that the model doesn't think it told the user something that it didn't. Um- Chaining all of these models together adds latency on every possible level, and so that's another big challenge.

26. 6:06

    And then you're losing some of that audio context, right? You're transcribing the audio, and if you've ever tried to convey a complicated topic over a text, you know it's a bit harder than dealing the-- dealing with the same thing using, using your own voice.

27. 6:21

    And so an alternative to that chained approach is a speech-to-speech approach, where we have a model that has been trained on audio and then takes that audio, uh, to directly interact on the conversation and, uh, make tool calls, meaning there's no transcribing in the process.

28. 6:40

    The model can just natively deal with that audio. And that translates into much lower latency because we're now skipping the speech-to-text, text-to-speech processes. We can also now have much more contextual understanding of the audio, um, including things like tone and voice.

29. 6:58

    And all of that leads to a much more natural, fluid, uh, level of conversation. But there are some challenges with this. One of the most common ones is reusing your existing capabilities.

30. 7:08

    Everything is built around text, so if you already have some of those existing capabilities or a very specialized agent for a certain task, uh, it's harder to reuse those.

31. 7:18

    Also, dealing with, like, complex states and complex decision-making is a bit harder with these models since they've been really focused on improving on the audio conversational tone, less so on being very complica-- uh, complex decision-makers.

32. 7:33

    But there is a solution that we can get around with this. Um, again, taking inspiration from what we do with text-based agents, we can actually create a, um, delegation approach using tools where we have a frontline agent that is talking continuously to the user, and then that one uses tool calls to interact with much smarter reasoning models

33. 7:55

    like o4-mini or o3. Uh, actually, let me at this point give you a quick demo and see how the internet goes here. Um, so I have a real-time agent here that we built with the Agent SDK.

34. 8:08

    It's gonna be very similar to what you're gonna build later on. Um, but when I start talking to this...

35. 8:16

    Hello there.

36. 8:19

    Hello there. How is the AI Engineer World's Fair going?

37. 8:23

    So we can now give it ta-- like,

38. 8:27

    give a task to, like, call tools that I gave it. Like, "Hey, what's the weather today?"

39. 8:34

    Let me check the weather for you. One moment, please. Transferring you to the weather expert now. Actually, I can directly help you with the weather information. Could you please specify the location you're interested in?

40. 8:46

    Oh yeah. What's the weather in San Francisco?

41. 8:50

    So you can see here it's actually dealing with the interruption.

42. 8:53

    The weather in San Francisco today is sunny. Enjoy the bright and pleasant day. Is there anything else I can assist you with?

43. 8:59

    No, that's it. Thanks.

44. 9:02

    You're welcome. If you need anything else, feel free to ask. Have a great day.

45. 9:07

    And so in a similar way, we can actually trigger the more complicated backend agents as well. So I have a tool for this to handle refunds that will call out to o4-mini and evaluate the refund.

46. 9:18

    So, hey there. Um, I have one more thing. So I recently ordered this skateboard that I tried to use, and it seems like I'm really bad at skateboarding, so I wanna return it.

47. 9:29

    Um, it is slightly scratched though.

48. 9:34

    I'm here to assist, but it sounds like you need customer service for that. I recommend contacting the company where you bought the skateboard. They can provide you with the re-

49. 9:44

    Oh. Looks like I didn't add the tool. Maybe I did. Oh, I asked for-- I didn't ask for a refund. Um, let's try this once more. Hey there. Um, I bought a skateboard recently that I tried, and apparently I'm really bad at using it, so I wanted to return it.

50. 10:03

    It is slightly scratched, though. Can you give me a refund?

51. 10:07

    Hello there. How is the A... Hello there. How is the AI Engi-

52. 10:13

    The joys of internet. Um... Hey, um, I recently ordered a skateboard from you, and it, uh, failed. Like, I ca-I can't use it. I'm struggling to use it. It's slightly scratched.

53. 10:28

    Can you give me a refund, please?

54. 10:32

    Hello there. How is the A... I'm going to assess your request for a refund.

55. 10:39

    There we go.

56. 10:39

    Let's get started.

57. 10:40

    It is slightly struggling with this, like, weird echo that we're having here.

58. 10:44

    Your refund request has been approved. The skateboard arrived damaged, and you're eligible for a full refund. We'll process that for you.

59. 10:51

    All right. Um, but you can see here that, uh, it was able to call that more advanced tool and, um, actually process that request. And one of the nice things is that, like, while time to first token is often a really important thing, the longer a conversation goes, like, your model is always gonna be faster than the

60. 11:11

    audio that has to be read out. And so this, this is, like, a really helpful thing where by the time that the model was able to say like, "Hey, I'm gonna check on this for you," it already had completed that LLM call to the o4-mini model to get the response there.

61. 11:29

    All right. Um, let me-- Oh, one more thing. Since we talked about traces, one of the nice things now is we can actually go back here into our traces UI, and this launch today, um, you'll be able to actually look for any of your Realtime API cases, look at all the audio that it dealt with, um, and

62. 11:49

    all the tool calls. So we can actually see here that the tool call was triggered, what the input was, the output. We can listen to some of the audio again-

63. 11:57

    I'm going to assess-

64. 11:59

    Um, to understand what happened and then because both this and the backend agent use the Agents SDK, we can go into the other agent as well, which was the o4-mini one, which we can see here.

65. 12:12

    And we can see that it received the context of the full past conversation, the full transcript, um, as well as additional information about the request, and then generated the response here.

66. 12:23

    So this allows us to then get a full complete picture of, like, what happened both in the front end and the back end.

67. 12:30

    Let's jump back into the slides and cover a couple of more things before we get coding. Um, and that's about best practices. So I would group when you're-- uh, group the best practices of, like, building a voice agent into three main things to keep in mind.

68. 12:45

    Uh, the first one is to start with a small and clear goal. This is super important because measuring the performance of a text-based agent, you'll hear a lot about evals at this conference, uh, is already hard enough.

69. 12:58

    But with voice, uh, voice agents, it's gonna be even harder. So you wanna make sure that you're very focused on, like, what is the first problem you wanna solve, and keep it focused on that, and give it a nu-- like, limited number of tools so that you're fully centered on this.

70. 13:12

    The Agents SDK makes this really easy because you can then later on add additional tools to additional agents and deal with, like, handoffs between them. Uh, but this way you can kind of really stay focused and make sure that one of your use cases is great, and then hand off other ones to human agents, for example.

71. 13:31

    The second one is what I la- elaborated on, which is building evals and guardrails very early on, uh, so that you can feel both confident in what you're building, but also confident in, um, that it's actually working, so that you can then continue to iterate on it and know when it's time for you to, like, grow the

72. 13:49

    complexity of your voice agent. Uh, as of today, you can use the Traces API for-- uh, the Traces dashboard for that. Uh, but alternatively, some of our customers have even built their own dashboards like Lemonade to really get an end-to-end idea of the customer experience, and then even replay some of these conversations with their agent as they're

73. 14:10

    iterating on it. The other thing that I'm personally super excited about with these models is, uh, both our speech-to-speech model and our text-to-speech model are generative models, meaning you can prompt them the same way that you can prompt a LLM around tone and voice, and you can give it emotions, roles, personality.

74. 14:30

    Uh, we built this little microsite called openai.fm. It's a really fun website to play around with, where we have a lot of examples of different personalities and how that, uh, that style of prompt can then change what is being read out by our text-to-speech model.

75. 14:46

    And so that's a great way for you to not just limit-- One second. Uh, limit the experience of your model, um, or, like, the personality of your model by the voice that you picked, but also by the prompt and instructions that you're giving it.

76. 15:02

    There was a question there. Would you mind using the mic that is right behind you, just so that it's on the recording?

77. 15:11

    Hello, sir. So my question regards to the previous slides on Lemonade. So you're displaying how they have this dashboard where they can show all of this. Is this a dashboard that OpenAI provides and Lemonade just integrates as like an iframe or something?

78. 15:28

    No. So in this case, they built their own, um, solution for it.

79. 15:32

    Okay. And does OpenAI then provides all the JSON or the data structure that we can just plug into the-

80. 15:38

    So the, the way the real-time API under the hood works is that you get all the audio data, and you can do whatever you want with that, basically. You're getting all the necessary audio events, so you can use those data structures.

81. 15:49

    So we're not storing them by default. You can use the Traces AP-- uh, the Traces dashboard. We don't have an API for it yet, but you can use the Trace- Traces dashboard to, um, get a, a basic look of, uh, of that, but it's not iFrame-able.

82. 16:04

    But you mentioned it's only audio data. This shows not just audio, but also the transcription and all of that as well, right?

83. 16:10

    So the, the Traces dashboard, if we go back to it, um, does show, um,

84. 16:17

    all of the transcripts and stuff as well, as long as you have, um,

85. 16:23

    transcription turned on, which I don't seem to have turned on for this particular one. Um, but it should-- Like, you can turn on transcription, and you should be able to see the transcripts as well.

86. 16:33

    Okay. Thank you.

87. 16:34

    You're welcome. All right. Um, let's go back to this. Um,

88. 16:44

    the other part with it is, as I said, you can prompt both the personality. You can also be very descriptive with the conversation flows. One of our colleagues, um, found that giving it conversation states, and so this JSON structure, is a great way to help the model think through sort of what p-processes and what steps it should

89. 17:03

    go through, the same way that you would give a human agent a script to operate on.

90. 17:08

    If you're struggling to write those scripts, though, uh, we also have a custom GPT that you can use to access that. And I'll share all of those links and a copy of the slide deck later on in the Slack channel.

91. 17:19

    So if you're in that, uh, you should be able to access those.

92. 17:23

    But with that, um, that's primarily what I wanted to talk through from a, from a slides perspective. So from here on, what I wanna do is, um, build with you a voice agent.

93. 17:36

    We'll see how that goes with the internet. Um, also, if you have headphones, now is a great time to bring them out. It's gonna be really weird when we're all gonna talk to our own agent.

94. 17:45

    Um, but we're gonna try this and see how that goes. Um, [chuckles] so if you came in later, please scan the QR code, go to that, uh, GitHub repository, and set that up.

95. 17:57

    Install the instructions. There's no code in it yet other than, like, a boilerplate Next.js app and a, um, empty, like, package JSON that installed just, like, the dependencies that we needed so that we are not all trying to run NPM install at the same time.

96. 18:13

    But what I wanna do is build a first agent. Um, so if you want, you can just straight up copy the code that is on here, but I'm gonna, um, actually go and type it along with you all so that you get a feeling, uh, for what's happening and we have a good idea of, like, timing.

97. 18:35

    So if you wanna take a picture now, just code ahead, do that, and otherwise, uh, I'm gonna switch over to my code editor and we're gonna do this together.

98. 18:45

    Oh, and if you're running into trouble, the Slack is a great way to post questions that are technical questions, and, and Anoop, who's over there, um, is gonna try to help you.

99. 18:57

    Um, alternatively, raise your hand, but it's a bit easier if you're just Slacking the messages there and we can kind of multi-thread the problem. Um, all right. Let's go and build an agent.

100. 19:10

     So if you cloned the project, you should see an index, uh, .ts file. Go and open that, and you should be able to import the agent class from the openai/agents package.

101. 19:26

     That's what we're gonna use to create the first agent. Yeah.

102. 19:29

     Can you blindfold the lights?

103. 19:31

     Oh, yeah. Good call. Um... Is that better?

104. 19:44

     Yep. Yeah.

105. 19:46

     Cool. That seems a bit-- Seems worse on my side than yours, but I think as long as you all can read that, I'll be fine. Um, all right. So what y- what I want you to do is go and import an agent, and we're gonna define our first agent.

106. 20:01

     And as I mentioned, um, primarily the-- an agent has a c- like, a few centerpieces, the first one being a set of instructions on what to do. So we can give it instructions.

107. 20:12

     I'm gonna say, "You're a helpful assistant." It's sort of the most boilerplate thing you can do. We do need to also give it a name, and that's, uh, so that we can actually keep track of them in our traces dashboard.

108. 20:24

     I'm gonna say My agent here. This can be anything that helps you identify it. And then we need to actually execute this agent. So we can import a run function here,

109. 20:36

     and then we can await the run here. I'm gonna

110. 20:44

     run this agent with just, "Hello, how are you?" And then log out the results. And with the results, we get a lot of different information because, uh, w- essentially when we run an agent, it's gonna do a lot of different tasks from executing all necessary tool calls, if there are any, to validating output, guardrails, et cetera.

111. 21:04

     But one of the most common things that you just wanna log out is the final output. That's whatever the last agent in an execution said. Um, so in this case, it's gonna be, uh, a set of text, and then you should be able to run npm start.

112. 21:21

     Um, oh, npm run start zero one, and that should execute it, and then you should see something like this, um, depending on what your model decides to generate.

113. 21:36

     And by default, this is gonna run GPT 4.1, um, as the model. But if you wanna experiment with this, you can set the model property here, and we can set it to

114. 21:49

     o4-mini, for example, and then rerun the same thing. So this is the most basic agent that you can build, but one of the things that really makes something an agent is if it can execute tools.

115. 22:02

     So we can import a tool here. There we go. And we can define a get weather tool. Um,

116. 22:14

     one of the things here is you have to specify what arguments the model's gonna receive, and one of the ways that you can do this is through a tool called Zod.

117. 22:25

     If you've never heard of it, it's essentially a way to define schemas. Um, and what we'll do is we'll both use that Zod schema to inform the model on what the parameters for this function call are, but we're also gonna use it to validate then what are the actual arguments that the model tried to pass in, and

118. 22:43

     do they fit to that schema? So we get full type safety here if you're a TypeScript developer and you care about that. So in this case, we have, um, a get weather tool, and then we can give that tool to the agent, and we can change this to, What is the weather in Tokyo?

119. 23:01

     Uh, is what Cursor wants to check. So if I run this again,

120. 23:06

     let me move this slightly. We can see it's gonna take a bit longer now, and that's because it ran some tools. Um, and now it's telling me the weather in Tokyo is sunny.

121. 23:16

     And if you're wondering, well, did it actually run a tool? We can go into

122. 23:25

     our traces dashboard here and look at the trace. We have a My agent here.

123. 23:35

     And in there, we can see it ran,

124. 23:40

     tried to call the tool, executed the tool, and got the weather s- in Tokyo is sunny back, and then took the response to generate the final response. So the traces dashboard is a great way for you to see what actually happened behind the scenes.

125. 23:57

     How are we feeling? Can I get a quick temperature check? Are able-- Are people able to follow along? I see Zeke is giving a thumbs up there, so. Um, all right.

126. 24:06

     So this is a text-based agent. I wanted to show you this just to get a bit familiar with the, um, overall Agents SDK so that we can jump into building voice agents.

127. 24:17

     Uh, the first one we're gonna build-- or, like, the first

128. 24:21

     thing we need to understand about a voice agent is the slight differences between a voice agent and a, and a what we call a Realtime Agent. Essentially, a Realtime Agent is just a specialized version of an agent configuration.

129. 24:34

     There's just a few fields you can't pass in. But they can be used in what is a, what is called a Realtime Session, because with voice agents, there's a lot more things to deal with than just executing tools in a loop.

130. 24:45

     Um, one of the most important things is you need to deal with both the audio that is coming in, uh, process that, and then, like, run the, ru-run the model with that, and then gener- uh, deal with the audio that's coming out.

131. 24:57

     But you also need to think about things like guardrails, handoffs, other lifecycle things. And so the Realtime Session is really dealing with all of that. So let me show you how that works.

132. 25:09

     Um, for this, what we're gonna do is we're gonna go in the same project. There's a zero two, and it has a page TX- uh, TSX in there. Um, this is a Next.js app that really I just gutted to have, like, the bare minimum in there.

133. 25:27

     But this is a great way for us to just, uh, build both the front end and the back end part of the, um, voice experience, because this voice agent that we're gonna build is gonna run in the browser.

134. 25:39

     In order to make sure we're not leaking your API credentials, one of the important things is you need to use an ephemeral key. That is a key that is short-lived and is gonna be generated by your server and handed off to your client so that they can use that to interact with the Realtime API over a protocol

135. 25:56

     called WebRTC. Um, for that, you should see a token.ts file in your repository that just calls out to the Realtime API to generate a session and then return a client secret, which is that ephemeral key that we can then use to authenticate with the SDK.

136. 26:15

     You do not have to do this if you're building a Realtime Agent that is running on your server, for example, in the case of a Twilio app or something else where you can just directly interact with the, uh, OpenAI API key.

137. 26:31

     But if you're running anything in the browser, then you actually need to generate this client key just so that you're not, you know, giving your API key to the world.

138. 26:40

     So with that in here, we can actually go and build our first Realtime Agent. So similar to previously, we're gonna import a, an agent class here, but in this case it's gonna be a Realtime Agent.

139. 26:57

     And we're gonna import it from the realtime package,

140. 27:00

     uh, which is just a subpath in the same package, so you don't need to install a different package here. But now we can define a Realtime Agent that works the same way.

141. 27:10

     We have a name, we give it instructions, um, just sort of going with the default suggestion here. And now we actually need to connect that agent to a Realtime Session.

142. 27:21

     So I have a Connect button here for running this example. Let me start it up here with npm run start zero two. That command should be in your README as well.

143. 27:33

     Uh, it's gonna start a development server, and we can go over here, reload this, and you can actually see

144. 27:41

     it just has like a little Connect button that right now doesn't do anything. So let's connect that up. I don't need this anymore. Um, so let me just move this to the side.

145. 27:55

     Um, so in this onConnect function that gets triggered whenever we press the button, we wanna deal with that connected state. So we're g- what we're gonna do here is we're first gonna fetch that token, and this code, what this basically does is it's gonna import that server action, which is a Next.js concept that just makes sure that,

146. 28:17

     like, this code is gonna run on your back end. If you're using a different framework, you, you should be able to just go and fetch this key from your back end server.

147. 28:27

     Um, and then once we have that token, we can go and create a new Realtime Session. So what we're doing here is we're gonna give it the first agent that should start the conversation up.

148. 28:40

     Uh, I'm gonna specify the latest model that we released today along with the Agents SDK. This model is a-- If you've used the Realtime API before, it's an improvement, especially around tool calling.

149. 28:51

     It's much better on that front. Um, we have a couple of different customer stories on our Twitter if you wanna check that out. Um, and then I'm gonna give it...

150. 29:02

     Not there. I don't know why cursor insists on that. Um, the last step that we need to do is we need to connect to that session. So this is where we're gonna give it that API key so that the mo- uh, that we can connect to the Realtime Session under the hood.

151. 29:19

     Just so that it's easier for us to deal with all of this, I'm also gonna close the session. But I've got one thing here that is an oddity of, uh, React.

152. 29:29

     We do not wanna generate that session every time, so I'm gonna...

153. 29:35

     On every re-render. So I'm gonna create a what is called a ref here. Again, if you're new to React, um, this basically is just a variable that's gonna persist through re-renders.

154. 29:46

     So we need to slightly change this here, where we're gonna assign that to session.current so we can maintain that. And then that also allows us to say if there is a session.current set, uh, we wanna actually close that connection when we press the Disconnect button.

155. 30:02

     That just makes sure that we're disconnected from the audio again.

156. 30:07

     So I'm gonna leave that on the screen for a second, and then we can test this out. But if you already typed this, go into your browser, refresh, press Connect, and you should be able to talk to your agent.

157. 30:29

     All right. Let's try mine. Let me move this to the other side so it's not blocking your code.

158. 30:38

     Hello?

159. 30:40

     Hi there. How can I assist you today?

160. 30:44

     All right. So you can see, um, now it's just a few lines of code. We didn't have to deal with things like figuring out how to set up the microphone, how to set up the speakers.

161. 30:52

     By default, if it's running in a browser, it will do-- deal with all of that automatically. If you do wanna pass in your own microphone source or other things like that, you can do that as well.

162. 31:01

     Um, if this is running on a server, you have a, um,

163. 31:07

     both a send audio function that allows you to send an audio buffer in, or you can listen to the audio event, which is gonna emit all of the audio buffers that are coming back from the model so that you can pass it to whatever your source is.

164. 31:28

     Um, so that's our first basic agent. Any, any questions so far?

165. 31:38

     Please open it to Reffo. [laughs]

166. 31:40

     Hmm?

167. 31:41

     Can you send that code to the Reffo? Try put to their both side.

168. 31:46

     He wants you to send the-

169. 31:46

     Can you push it?

170. 31:47

     -that code to the Reffo.

171. 31:47

     Can you push it? Can you push it up?

172. 31:49

     Oh [laughs], I can push it, yeah.

173. 32:04

     Or copy to drive.

174. 32:04

     Good call. Thank you. Um, all right. So now that we have that, let's go and actually give it a tool. So this is really where the benefit of the, um, Agents SDK comes in.

175. 32:19

     We can actually use that same tool definition that we did earlier. Um, so I'm just gonna follow the autocomplete here. We should be able to just give that tool now to our agent and save.

176. 32:34

     Oh, I need to import Zod again to do that schema validation. This is especially important on the real-time side because the real-time model currently do not-- does not support strict mode, so the JSON might not fully comply with your, um,

177. 32:53

     um, schema unless you're running-- you're giving us a Zod schema, and we'll go and validate that this actually fits that schema. So that makes your code a bit easier.

178. 33:03

     So with that, we can go back. Hey, what's the weather in San Francisco?

179. 33:12

     The weather in San Francisco is sunny today.

180. 33:15

     And we can disconnect it here. Um, also this does now deal with interruptions. So, hey, can you slowly count to, like, twenty?

181. 33:26

     Sure, I'll count to twenty for you. One,

182. 33:31

     two, three, four, five, six-

183. 33:34

     Okay, stop.

184. 33:37

     Sure, I'll stop counting.

185. 33:38

     How far did you count?

186. 33:41

     I counted up to six.

187. 33:44

     It's pretty, pretty close. It's, it's always hard with the timing to, like, perfectly get it, but, uh, normally that's enough to, um, deal with the context. But it is super crucial to have that interruption timing so that, like, your model doesn't think it read out, like, the full customer policy, um, but the customer interrupted it halfway through,

188. 34:03

     for example. Um, all right. Um-

189. 34:07

     Quick question. You don't have to, um, manage all the events to actually do that anymore?

190. 34:12

     No. Um, so the, the real-time session will handle all of those events. That is a great proxy tool. If you are curious about these events, um, that you're alluding to, what we can do is listen to the

191. 34:30

     transport event and... Let's do this later. Uh, this will log out all of the events that are happening under the hood. So if we open the dev tools here and rerun this.

192. 34:49

     Hey.

193. 34:50

     Hey there. How can I help you today?

194. 34:53

     So you can see all of the events that normally you would have to deal with are being dealt with. You still have full access to them, so you can both read them, but you can also send events yourself.

195. 35:03

     So it's gonna handle all of the things, but continue to par-- uh, pass them on to you if you wanna do your own logic on top of that.

196. 35:14

     I'm gonna push that code for you so you can pull it. [laughs]

197. 35:28

     Cool. Um, all right. Since we already have this commented out code, the other part of this that typically is a request that you wanna deal with is I wanna show, like, the transcript.

198. 35:41

     I wanna see what sort of is being transcribed. And the important thing here is I'm using the word transcribed because even though the speech-to-speech model is dealing with the audio directly, um, and there is no transcription sp- uh, step in between, by default, we're gonna transcribe all of the conversation at the same time.

199. 36:00

     Um, you can turn that off if you want to. Um, if you're using the API directly, you have to actually turn it on. Um, in the Agents SDK, it's turned on by default because it's such a common request, and it enables us to do a couple of additional features that we'll cover later on.

200. 36:16

     But this is gonna give us that whole history every time. So I'm just gonna, um, log that history here.

201. 36:26

     Or rather, I'm gonna... There we go. Import that. I'm going to set that as a variable, and then because it's React, we can

202. 36:37

     Let's create a list here. We're gonna go over all of this. I need to filter 'cause it has both tool calls and messages, and I only want to show the messages for this.

203. 36:49

     Um, so should be able to... It doesn't-- Why does it want that? Um, let's see.

204. 36:58

     Close this. Refresh. Hey.

205. 37:02

     Hello. How can I assist you today?

206. 37:06

     How's the weather today in San Francisco?

207. 37:10

     The weather in San Francisco today is sunny. Anything else you'd like to know?

208. 37:15

     So you're automatically getting that conversation. If you are interrupting the model, one of the things that happens is the transcript is gonna disappear, and that's because, uh, the c- the model currently does not adjust that transcript, and instead it's gonna be removed, and we're gonna remove it from that object as well, just so that you get the

209. 37:35

     most accurate representation and you're not thinking that, like, the model read out a certain piece of text. And again, with everything that we're doing here, can actually go back into traces, and we can see that same representation here with the weather call and everything, so again, helps with the debugging.

210. 37:56

     Gonna go briefly back to the slides. So we covered-- We set up our first agent.

211. 38:02

     Question.

212. 38:02

     Yeah.

213. 38:03

     Um, how do you store the conversation history?

214. 38:06

     The question was, how do you store the conversation history? Um, so it's currently fully stored in memory. Um, so basically there is gonna be a bunch of events that are-- that I logged out that are being emitted by the Realtime API.

215. 38:21

     All of those are gonna be sent over to the client and then stored in memory in a conversation, like in just like an array, essentially. So you can do whatever you want with that by listening to that history updated event.

216. 38:33

     So if you do wanna store it somewhere, you can store it. Um, the other part is the traces part is automatically gonna be stored on the OpenAI, uh, platform as long as you both enable that tracing.

217. 38:47

     You can disable it, um, by default in the Agents SDK it's enabled. And then, um, the other aspect of that is if you are a ZDR customer, so a zero data retention customer of OpenAI, you don't have access to that traces feature.

218. 39:02

     Um-

219. 39:03

     How about the conversation context? Like the session, how, how, how, uh,

220. 39:12

     how many previous messages of voice conversation is passed?

221. 39:17

     Um, so the question was how much of the, like, voice context, how much of the con- previous conversation is being used? Um, that's gonna depend and sort of like dealt with directly by the Realtime API.

222. 39:30

     So, like the Realtime API, when you d-- start that session, um, that holds the source of truth for that whole conversation session. So what you're receiving on the, on the client side is just a copy of whatever is happening at that point.

223. 39:45

     It's not the source of truth of what we're gonna adapt to pass into the model.

224. 39:51

     So the inference, uh, costs for token, does that include all the conversation that happened before? All the voice, uh, that happened before? Like if I have the before-- Like if I have a conversation and then AI, AI initiates, I will have the all the conversation from the beginning, all the inference?

225. 40:11

     Um, the question is how does it work with like the inference cost and like whether you're passing, like passing in that whole conversation? Um, Anup is nodding. Um, he's the bigger expert there, but yes, um, we're actually like you can log the, the, like we're keeping track of the usage.

226. 40:30

     There's an event that you can like log out to see your token cost, um, so you have an idea of like what is being actually passed in. So like with every-- If we're going back here to this example, you can see these response done events.

227. 40:45

     Um, I don't know. Where is the, um... Shouldn't it be on the response done?

228. 40:52

     Uh, it is being sent over. I just do not know right now. It's not showing. Oh, there. So you can see here, um, it outputs the detailed information of your tok- token usage at any point in time.

229. 41:08

     So while you don't have like access to like what is exactly what was passed into the next response generation, um, you can keep track of the cost as it's happening.

230. 41:20

     You're welcome. Uh, yes. Do you-- There's a microphone right over there. That might be easier than you yelling across the stage to me.

231. 41:27

     I, I see that the format that you're using is PCM16. Is there a way in which we can modify the output formats of the audio files so we can save in memory?

232. 41:37

     Um, yeah, there are different, different audio modes that you can use, um, including like for example, U-Law for-- that is like helpful for phone calls, for example.

233. 41:48

     Okay.

234. 41:50

     Sorry. Uh, another question on the usage. Um, does that like final assistant response roll up all the tokens from like all the intermittent tool calls as well?

235. 42:01

     Does that make sense? Like the agent needs to like kinda reason through and then format tool calls, so I'm assuming it's not just the output tokens for only the assistant response, right?

236. 42:11

     It-- Like every tool call is a response in general as well. So like it, it works the same way that like the responses API works, for example.

237. 42:22

     Okay. So like each-- Right? 'Cause we're using this and we have like tool calls and tool call outputs, right?

238. 42:28

     Yeah.

239. 42:28

     And I couldn't find the like usage attribute on the tool call output. Is it somewhere in those like raw events that are outputted?

240. 42:35

     Do you, do you know, Anup?

241. 42:38

     Okay, no worries. I, I know it's, like, kind of early on time

242. 42:40

     We can, we can follow up on it.

243. 42:41

     All right. Thank you.

244. 42:41

     You're welcome. Uh, yeah, do you wanna head over to that microphone that is right behind you? It just makes it a bit easier.

245. 42:50

     Yeah, there.

246. 42:50

     Oh, yeah, there's... In the meantime, do you wanna-

247. 42:52

     Uh, just a quick question. Can I go back to the slides explaining the different modes of the, uh, the audio agents, like the text in, text out, that's the first one.

248. 43:02

     Oh, yeah.

249. 43:02

     Text to speech, that's the second one. Uh, I didn't get the third one. And, uh-

250. 43:08

     Oh, you mean this?

251. 43:11

     Yes. Yes. It... Uh, when you just showed us the GPT-4 real time, that one.

252. 43:21

     Yeah.

253. 43:22

     That one. Is that, uh, this PPT, this slide ab- is about?

254. 43:26

     Yeah, exactly. So, like, where it's like when we did the refund, um, it kind of followed this pattern where it performs a tool call. Like the, like real-time API agent can perform tool calls.

255. 43:39

     It performed a tool call to trigger a separate agent, that was the refund agent that, in my case, used o4-mini to execute that task and then hand that back.

256. 43:50

     Okay. Got it. Thanks.

257. 43:52

     You're welcome. Yes.

258. 43:55

     Um, I'm currently using, like, a regular, uh, OpenAI agent. So what will be the challenge that I will face when I will-- when we want to change my regular, my regular agent to a real-time agent?

259. 44:08

     So there's a couple of different challenges. Like, one is, like, anything that you're doing around latency, li-like anything you're b- uh, doing around, um,

260. 44:19

     voice, latency is always king. So, like, you wanna figure out what are the best ways to... I actually have a slide around this. Um, like when you're-- when it comes to things like tool calling, you wanna find ways to do things like buying yourself some time.

261. 44:35

     So you'll typically see some prompting around like, "Announce what you're about to do next," um, before you're doing it, and that's to do that little trick around, uh, while the, while the previous audio is still being read out, the agent can already perform the tool call and wait for the tool call to come back.

262. 44:54

     Because similar to a text-based agent, the model can't do-- like can't receive additional data as like, like do another thing, um, outside of like we can interrupt the response, but it can't finish that response, if that makes sense.

263. 45:09

     And so you wanna do these sort of like buying time. The other thing is, like, if you're building a real-time agent, the longer your prompt gets, at one point it increases the likelihood that it gets confused, so you wanna make sure you're properly scoping those use cases and, like, through what, uh, what we call handoffs, where you

264. 45:29

     have different agents that are more specif- uh, scoped to specific steps in your experience.

265. 45:36

     Thank you.

266. 45:37

     You're welcome. Yes.

267. 45:41

     Can you speak a little bit more about memory? Earlier, you said that some of that permission will be stored in memory. Is that short-term, long-term, session memory? Like what-

268. 45:51

     Yeah. So you're, uh-

269. 45:52

     Like a memory tool on top of that, you think?

270. 45:55

     Yeah. So the question is about memory. Um, basically, right now, the, the, ig- the... Let me correct this. Um, when we go back to this demo, what you're seeing here is essentially just like a copy of the events that we're receiving back.

271. 46:14

     So this is like helpful as a visualization of the history. Uh, that being said, the actual, like, memory in the sense of like an LM agent memory is the session context that is happening on the real-time API side.

272. 46:28

     There are events that you can use to update that. Um, we actually have an update history event that you can pass in what you want the history to be.

273. 46:37

     But what that does is essentially like fire off events to the real-time API to say like, "Delete this item from the history" or, "Add this new item," um, and you can give it a previous item ID.

274. 46:48

     So like you can, for example, like slot messages into a specific spot if you wanted to. Um, does that make sense? But there's like no, no like advanced like long-term memory solution like you, like you were alluding to.

275. 47:04

     Cool. Yes.

276. 47:06

     Hi. Do you have tips for handling input from low fluency users? Like say someone who's just learning a language, and they have like multilingual input and maybe broken grammar, and their pronunciation's not so good.

277. 47:19

     Uh, I don't think I have any, like, best practices right now that I could share.

278. 47:24

     Can it handle it just off the shelf or-

279. 47:26

     Um, it can handle like switching languages and, and things like that.

280. 47:31

     Okay. But it might not be able to handle low fluency.

281. 47:34

     Um, I don't know if we have any use cases.

282. 47:44

     Yeah. We have, we have some customers that are like language learning companies, so there, there is some that are using it that way, but I don't think I have any like best practices that I can share.

283. 47:53

     Okay. Thank you.

284. 47:53

     You're welcome.

285. 47:58

     Sorry. Back in the code, is there a callback for the interrupt, and does it include the last transcription?

286. 48:06

     Um, there is a call for the interrupt, but, uh, there is no,

287. 48:11

     um... Oh. There's no actual like event that it-

288. 48:18

     There's no param or E that comes with it or anything like that?

289. 48:20

     No.

290. 48:21

     Okay.

291. 48:21

     There's currently no, um, transcript. So what you can do is if you're getting this, you can call-

292. 48:28

     Get history or something?

293. 48:30

     Uh, there's just history.

294. 48:31

     Okay.

295. 48:31

     Um, so like this always is up to date.

296. 48:34

     Cool.

297. 48:34

     Um, so you can-- you have access in that moment.

298. 48:37

     Okay.

299. 48:37

     The d- the thing that we do have is, um-

300. 48:42

     For tool calls specifically, um, you're getting some additional context, and that context has a, um, history parameter that you can, like, push into.

301. 48:55

     Okay.

302. 48:55

     Um, it's more documented in the, in the documentation.

303. 48:59

     In the API.

304. 48:59

     If you wanna check that.

305. 48:59

     Okay, great. Thank you.

306. 49:01

     You're welcome. Awesome. Um, let's move a bit on and show a couple of other things. So we talked about tools. As I said, like, one of the benefits is you can reuse the same syntax that you're doing with text-based ones.

307. 49:16

     Um, it's also a good way for you to then communicate with your back-end systems using HTTP. Um, follow sort of a, um,

308. 49:28

     general practice around, like, keeping both the tool calls as low latency as possible. Like, send out a tool-- Like, for example, if you know a task is gonna take longer, start the task, give it a task ID, and have the agent have a tool to check on the status, for example.

309. 49:47

     Like, that helps getting back to it because, again, while the tool call is going on, the model is sort of stuck. So you wanna, you wanna make sure to, like, get, get back to that as soon as possible.

310. 49:59

     Um, one of the other things that you can do is human appro-- uh, hu-- uh, human approval. I can show you that quickly. There's essentially a use, uh, it's a needs approval that, um, you can either specify as a function that will be evaluated before the tool ever gets triggered.

311. 50:18

     This is a great way if you have, like, a more complex logic on, "I need approval for this." You can also give it just straight up, "I always need approval," at which point there is a-

312. 50:32

     another event here, um, tool approval requested, and then that gets a, um, event here, so we can do things like, um...

313. 50:51

     Good old prompt. Um, and then we can go and approve that tool call again.

314. 51:02

     I don't know why the autocomplete is not working.

315. 51:07

     Um... Yeah. Approve. There we go. And then that...

316. 51:21

     Why is it... [clears throat] Oh, this is wrong. And this is where I go into the docs because I do not remember why this is autocompleting the wrong way.

317. 51:41

     But everything I'm showing you is in the docs, um, [chuckles] so we can just... Oh, took the wrong thing. Right, the first...

318. 51:55

     There we go. Approval request. And then we can do-

319. 52:07

     Approval request.

320. 52:08

     Approval request. Thank you. It's like the classic thing when you're on stage, and you can't really...

321. 52:16

     There we go. So in this case, I'm just gonna always approve. But if we now go in, "Hey, um, can you tell me the weather in Seattle?"

322. 52:34

     So we can, in that case, approve it. It's always gonna approve right now because I'm not actually checking the status. But, um, that means you can build, like, a human-in-the-loop approval experience.

323. 52:45

     This is really convenient, especially if you're running it in the browser and you just wanna have, like, a confirmation if, like, the tool is hallucinating things before the customer actually submits it.

324. 52:54

     And does it do it directly? Can it actually say, "Are you okay if I do this?" Or no?

325. 52:59

     Um-

326. 53:00

     Or do you have to do that and then do voice-- text-to-voice and all that stuff?

327. 53:03

     So the, the-- Basically, this is happening-- Uh, so the question is, does it automatically do this? Like, the-- What we're doing and the reason why this is separate is the model is asking for this tool to be executed, but we're intercepting this, um, basically before we're ever generating or executing the response.

328. 53:23

     This is intentional so that, like, you don't have to deal with fl-- Like, we want you to think through why should this tool need approval, as opposed to doing that somewhere p- halfway through your tool execution, and you have to, like, deal with the consequence of rolling back every decision that you've made, for example.

329. 53:41

     And so by default, if this is just needs true, it cannot get past that until the execution was approved, at which point it stores it in the context that is stored locally and then bypasses that security.

330. 53:54

     So this is not happening on the, on the model level.

331. 54:00

     Um, I'm gonna remove that again. Uh, so the other thing we talked about already, but I wanna show it in practice, is handoff. So a handoff is essentially just a specialized tool, uh, call that resets the configuration of the agent in the session so that, um, we can update the system instructions, we can update the tools, and

332. 54:23

     make sure that we can nicely scope the tasks of what we're trying to, um, solve. So what you cannot do, I know people are probably gonna ask about this, is you can't change the voice of the agent mid-session.

333. 54:39

     You could define different voices on different agents, but the moment that you're-- Like, the first agent that starts talking That's the voice that we're gonna stick with throughout the entire conversation.

334. 54:51

     So that's a caveat to just keep in mind. Um, but they're still very helpful to, let's say, have a weather agent here.

335. 55:00

     You can have a naked swear even.

336. 55:06

     Sorry?

337. 55:06

     You can have a naked swear. [laughs]

338. 55:09

     Um, and what we're gonna do in this one is call the weather agent. Um,

339. 55:22

     um, [keyboard clicking] let's do... We'll do this, and then what we can do is we can actually give it a handoff description. So if you don't wanna have this in your system prompt, but you just wanna help the model understand when to use this, you can say, like, uh, "This agent is an expert in weather."

340. 55:46

     And then this one's gonna have the weather tool. We're gonna remove it from this one, and we're gonna give it a handoff instead to that other weather agent. So now if I'm gonna restart this.

341. 55:59

     Hey, can you tell me the weather in, uh, New York?

342. 56:05

     The weather in New York is sunny, so you might wanna grab your sunglasses if you're heading outside. Enjoy the day.

343. 56:13

     All right. That's, that's the model's best attempt at a New York accent. Uh [laughs] [laughs] We'll take it. Um, but you can see there that, like, it automatically handed off from that first agent to that second one and let it handle it.

344. 56:26

     You can, through prompting, do things like, do you want it to announce that it's about to hand off? Do you not wanna do that? Sometimes it's a bit awkward if you're forcing it to always do it, so, like, I would not necessarily try it, but maybe that's the type of experience that you wanna have.

345. 56:41

     Um, so that's handoffs. Let me do you a favor and push that code.

346. 56:47

     Push, baby, push. [laughs]

347. 56:50

     So, uh, the agent can change voice when passing to another one, but it can't change accents?

348. 56:57

     Yeah, so, uh, [laughs] that's a good question. The voi-- uh, the, the, the agent can cha- can't change the voice, but it can change the accent. Um, that's because the, um...

349. 57:10

     Like, again, this goes back to, like, the model is a generative model, so you can prompt it to have different, like, pronunciations, tonality, like, voice in that sense, but it cannot change the voice model that is actually being used to generate that output.

350. 57:29

     So may- maybe as a, you know, the-- Is it, like, the whole realtime request body that can't be changed or just the voice? So can I, like, create a tool that could adjust the speed if someone was saying it's talking too fast for the, the noise reduction?

351. 57:47

     You should be able to change-- I, I have not tried the speed because, uh, speed parameter changing at mid-session because it literally came out today. Um, [laughs] uh, I don't know, Anup, if you tried this.

352. 57:58

     No. Um, but, like, you can-- Like, essentially, a handoff does change the session configuration. Like, if we look back at, like, um, the,

353. 58:10

     like, one of the transcripts here, like, now that we have a handoff. Um, let's go to this trace. So you can see here that, like, it called the transfer to weather agent, um, but then, like, the-- this-- these instructions were talk to, uh, talk with a New York accent.

354. 58:30

     Um, [clicks tongue] so in this case, it did change the instructions midway through the session. In the same way, like, when that handoff happens, we take away the tools, we give it new tools.

355. 58:41

     So you can change those tools. You could have a tool to change the tool, but my recommendation would be, like, use a different agent for that. Um, but then, like, the speed control, like, you could-- you should be able to send off an event, but I have not tried that.

356. 58:56

     Yeah. Or, or maybe, like, the, like, the background. Like, um, basically like if you had something and someone was, like, in a noisy environment-

357. 59:04

     Mm-hmm

358. 59:04

     ... like, "Hey, you seem to be getting interrupted. Could you adjust, or could we catch in the background and start adjusting that parameter so just the voice is protected as far as..."

359. 59:14

     Yeah. So the, the question is like, for example, if someone is, like, in a noisy environment, like, could you have the agent detect that and then use, um, like adjust some of the session configuration to deal with that and just the voice is protected?

360. 59:30

     I don't know honestly which parameters are protected or not. The good thing is the, like, the API will throw an error if, if a thing didn't work. Um, so it's a good way-- it's a good thing to experiment with.

361. 59:42

     Um-

362. 59:42

     You could do that in Python before.

363. 59:45

     Hmm?

364. 59:45

     With the previous ones, you could do that in Python.

365. 59:47

     Oh, yeah. Um, well, with the Python, uh, agents SDK, uh, we're doing the chained approach. We don't have a real-time build in yet. So...

366. 59:59

     So just calling the old API in Python, you could change-

367. 1:00:02

     Oh, all right. Yeah, yeah. Um, then it should work, yeah. Uh, if you can do it in Python, like, it should just work.

368. 1:00:10

     Cool. Um, so the other thing we talked about, um, is this delegation part. So that's what I, like, had mentioned earlier that was in the diagram. So this is essentially where you wanna be able to have certain complex tasks dealt with by a more intelligent model.

369. 1:00:32

     Um, and the way we can do that is essentially just creating another, um, agent, except on the back end. And because it's-- because the TypeScript SDK works both on the front end and back end, we can do that through, um-

370. 1:00:48

     Uh, I think I have a-- Let's see if we have a file here or not. Uh, we can do that using the same SDK. So I'm gonna create on the, in the server folder here a new, uh, file I'm gonna call just agent.

371. 1:01:05

     And in here, we can build our regular text-based agent. So this is essentially the same code that we've done before. Um, and we can say this is a, I don't know, um,

372. 1:01:20

     call it the Riddler. Um, "You are excellent at creating riddles based

373. 1:01:31

     on a target demographic and topic." Um, and we'll just give it a model of o4-mini.

374. 1:01:47

     Also a reminder for those, if you are trying to follow along, um, and you run into troubles, post in the Slack, and Anoop can help you with that. Um, so we have that new agent here.

375. 1:01:59

     We're not gonna give it any tools or anything. Uh, and then we can export a function here

376. 1:02:09

     that we just call runAgent, and this is just going to take some input and then return that output. And we can go back into our front end code, create a new tool here, create riddle.

377. 1:02:27

     Um, and this one, we're just gonna have take, like, two parameters, the demographic and the topic, and then call out the runAgent function that is gonna run on the server.

378. 1:02:43

     We can pass in-- Actually, realize I didn't specify this. Let's do demographic and topic, and then create an input here of

379. 1:02:55

     this. The other thing you wanna do when you're using server actions in, uh, Next is put that use server at the top. That makes sure that this file executes on the server.

380. 1:03:09

     And then we can pass in that demographic. And again, if you're using a different framework, this is just the equivalent of a fancy fetch request. So, like, if you wanna do an HTTP request to your services, if you wanna maintain a WebSocket connection to your own back end, you can do all of those things to, like, talk

381. 1:03:27

     back to other systems. So with that, we can give that to our main agent.

382. 1:03:38

     Create riddle. Oh. Tools. And then what you wanna do in these cases is, like, um,

383. 1:03:55

     you can tell it, like, announce when you are about to do a task.

384. 1:04:03

     Down say you are calling a tool. Things like that can be helpful to, like, buy itself some time.

385. 1:04:14

     Hey there, um, can you tell me a riddle, uh, for, like, a [REDACTED:age] Star Wars fan?

386. 1:04:26

     Hey, are you there?

387. 1:04:28

     I'm still here. I'm working on creating that Star Wars riddle for you. It should be ready in just a moment. Here's a riddle for your little Star Wars fan.

388. 1:04:39

     I'm not-

389. 1:04:40

     So you can see that, like, because it announced that, like, what it's about to do, the tool call came back before it even finished what it previously said. And so, like, that's again, one of the benefits of, like, if you can get your agent to, um, balance out that and, like, buy itself some time, this is a

390. 1:04:59

     good way to, uh, you know, deal with the more complex tasks. And, like, it also means that you can like, for example, take all of the, like, more reasoning heavy workloads and take it out, out of the voice agent model.

391. 1:05:15

     Uh, um, for delegation-

392. 1:05:17

     Yeah

393. 1:05:18

     ... is it possible to delegate to more than one agent, like simultaneously, or is it just one in the current SDK?

394. 1:05:25

     Um, you can-- It's, it's tool call. So, like, I think you-- Like, you would have two options, right? Like, you could do par-- like, parallel tool calling, or you could, um, like, have one tool that then triggers running multiple agents, right?

395. 1:05:42

     So, like, my recommendation would be that part potentially, uh, so that you're not relying on the model making the right decision of calling multiple tools at the same time.

396. 1:05:52

     Like, you wanna make the decision-making for the voice agent always as easy as possible.

397. 1:05:58

     All right. Thanks.

398. 1:05:59

     Yeah.

399. 1:05:59

     Previously, when you did-- Can you go back to the previous page of-

400. 1:06:03

     Yeah.

401. 1:06:03

     Oh, no. Sorry. Your example.

402. 1:06:06

     Which example?

403. 1:06:07

     Uh, the one you were running where it had the output.

404. 1:06:11

     Oh, yeah.

405. 1:06:12

     So on line three there, where it said, "I'm still here"-

406. 1:06:15

     Yeah

407. 1:06:15

     ... is that coming from your SDK? Do I have to use the SDK to do that, or is it Realtime API-

408. 1:06:20

     No, it's the Realtime API. It responded because I asked like, "Hey, are you there?"

409. 1:06:25

     Yeah.

410. 1:06:25

     Um, so it, it, it tri-- it realized that, like, it didn't start anything and, like, interrupted and was like, "Hey."

411. 1:06:33

     It was a tool call going on.

412. 1:06:35

     Yeah.

413. 1:06:38

     Cool.

414. 1:06:38

     Um, yeah. So the, the thing is that, like, I didn't render out the tool calls in here, right? So, like, what basically happened between this

415. 1:06:49

     And this was like it started off a tool call. Um, and then that tool call, because I interrupted it then, it stopped that tool call 'cause I interrupted it, it stopped the, the generation.

416. 1:07:03

     It also reset that, like transcript here. It's a good indicator that like the interruption happened. And so when I said this, it remembered it was trying to call a tool, did that tool call, um, and then gave you back the response.

417. 1:07:17

     So that's all the just the, the regular Realtime API.

418. 1:07:23

     Any, any other questions around this? Oop-- Yeah.

419. 1:07:29

     What's the cost per minute?

420. 1:07:30

     What's the cost per minute? We charge per token. There's some translations, I don't know. Anup, do you have the...

421. 1:07:39

     It depends on what you say. [coughing] It is more expensive than TTS and ASR, but it depends on the model you're using. Right. So it, it's more expensive than TTS and, uh, sp- uh, and like speech-to-text- It's chained up with a model ...

422. 1:07:59

     chained up with a model some-- uh, like in most case, but it depends on the use case, um, and sort of like your model choices and stuff.

423. 1:08:06

     So a bit harder to say like what the per minute pricing is because, again, it's by tokens, and it also depends on like if you have like transcription turned on and like how many function calls you have and things like that because it's a mix between audio and text tokens.

424. 1:08:22

     Um, so one of the interesting things, and this is not a thing in the, in the regular API, this is a agents, agents SDK specific thing, is, um, guardrails.

425. 1:08:33

     So like the agents SDK, both in Python and TypeScript, has this concept of guardrails that can either protect your input or your output to make sure that like the agent is not being meddled with or does things that are against policy.

426. 1:08:47

     Um, we took that same pattern and moved it over to the Realtime side, um, where essentially we're running these guardrails that you can define in parallel on t- on top of the transcription at all times.

427. 1:09:01

     You can kind of specify, you can see it at the bottom here, like how often you wanna run them, or if you only wanna run them when the full transcript is available.

428. 1:09:08

     But this is a great way for you to like make sure that the model for-- doesn't violate certain policies. You wanna make sure that these run as efficiently as possible because they're still running in the client.

429. 1:09:20

     Um, but this is a good way to still fix cer-- uh, like stick to certain policies, and if it violates those, it will interrupt it. Now, there is sort of the bit of the cave-- uh, a bit of a caveat where because we're running this on a transcript, um, it

430. 1:09:37

     results in a bit of a timing aspect where if it vi-- if it would violate your guardrail in the first couple of words, chances are it will say those first couple of words and then get interrupted.

431. 1:09:52

     If it is happening at a later point in time, um, then the transcript will be complete-- or like the text output, it's not really a transcript, the text output will be done before the audio is done speaking every-- uh, like, it is, is done saying everything.

432. 1:10:13

     And so in that case, it will just correct itself. So in that ca-- Like, to give you an example of like this is a guardrail that just checks like is there the word Dom in the output.

433. 1:10:26

     In this case, like if I would ask it like, "Hey, please call me Dom," chances are it will call me Dom and then self-correct. If I tell it to tell me a story and only introduce Dom in the second act, then it will, um, catch that it was trying to do that in a mu-- at a much

434. 1:10:46

     earlier point because that transcript is gonna be done before the audio is being comple-- uh, before the audio is being read out to the user. So the user will never hear Dom, but instead the model is gonna be like, "Okay, I'm sorry, I, I couldn't help you with that.

435. 1:11:00

     Like, let's do something else instead." And you can give it, um,

436. 1:11:06

     policy hints essentially on like why it violated this policy or what it should do instead. So you can give it these, this like output info where you can inform the model like why this happened.

437. 1:11:19

     Is, is the transcript still being done with Whisper?

438. 1:11:22

     No. Um, you can choose what, uh, tr-- Uh, so the question was is the transcript still done with Whisper? Uh, you can sw- uh, you can switch the transcript model.

439. 1:11:30

     So we have-- We released in March, uh, two models. One is GPT-4 Mini Transcribe, and...

440. 1:11:41

     We have two, right? Yeah, and GPT-4, um, uh, GPT-4 Transcribe. I was trying to remember like we have only one text-to-speech model, but we have two transcribe models, four transcribe models.

441. 1:11:55

     Awesome. Um, this was the main part of what I wanted to walk with you all through. So, um, one, I'm gonna, um,

442. 1:12:06

     post all of the links and the slides in the Slack channel, which let me go back to...

443. 1:12:15

     Where was that slide? There we go. Um, so in that Slack, I'm gonna post all of the resources, so if you wanna check them out afterwards. I'll also-- Like I've already put a bunch of the resources that I talked about into the bottom of that starter repository, so you should have access there as well.

444. 1:12:35

     And, uh, I'm happy to hang around, answer any questions. Yes.

445. 1:12:41

     How does model caching work with voice, uh, models? Um, my understanding is context-[object clattering]

446. 1:13:01

     Yeah. [object clattering] [people chattering] Yeah. Um- [people chattering] [object clattering] Yeah, the, the question was around how prompt caching works, um, and sort of whether we-- like for prompt caching that like we guide

447. 1:13:26

     the request to the same, um, same system again to run and whether, like, there's any control with that with real-time because latency obviously matters. Um, I don't think there's any controls about that.

448. 1:13:41

     Um, no. Um, I'm getting a no from there. So, like, I don't think there's any, any controls around that right now. Yes.

449. 1:13:51

     Hey. So we all know that having, uh, natural conversations is-- involves more than just spoken words. It involves detecting emotion and adjusting tone. Um, it also involves a cadence and

450. 1:14:08

     even humming to let the other person know that you are listening.

451. 1:14:11

     Mm-hmm.

452. 1:14:12

     I wonder if the current speech-to-speech model is capable of having that kind of natural conversation.

453. 1:14:19

     Um, part of this is like a, a prompting challenge. So, like, it, it definitely can have pretty natural sounding conversations. And like, I think this is sort of the, the part where I highly recommend to check out the openai.fm page.

454. 1:14:36

     Um, 'cause this is like-- it's, it's pretty interesting to see sort of, uh, if we go to, um... Let's see if we find the,

455. 1:14:47

     um... Actually, I didn't show one neat feature

456. 1:14:54

     that I normally call out on the playground. If you're just getting started with real-time and you don't even wanna write any line of code, um, this is a great way to just have conversations and try things out.

457. 1:15:07

     But one of the things it has is this, um, it has a couple of s-system prompts. One of my favorite ones to show sort of this is the bored teenager.

458. 1:15:17

     Um, so if we start this. Hey there. Um, so I'm at AI Engineer World's Fair, and everyone is super stoked about w-voice agents. Can you show me some excitement of this whole thing launching today?

459. 1:15:36

     Let's see.

460. 1:15:38

     I guess it's cool, whatever. [laughing] There's always new stuff launching. People get excited, but, you know, it's just voice agents. Not really my thing to get all hyped up about it. [laughing]

461. 1:15:53

     So you can see in this case, like, it, it put its own pauses in there and stuff. Like, this wasn't a pause because the model was waiting, right? Like, um, it can deal with a lot of that sort of adjusting tone and voice, and it can do similar things like reacting to, like, someone talking in and stuff.

462. 1:16:12

     Okay, thanks.

463. 1:16:13

     You're welcome. Yes.

464. 1:16:15

     Sorry, again, on the new, uh, API that's been released, what else has changed? Semantics, has this improved? Is there anything else that's improved in terms of turn detection or voice?

465. 1:16:24

     Um, we have not released any new VAD models. Um, we primarily released like a new base model for, uh, or like a new model for the, um, GPT-4 Realtime model that is just better at function calling and has been overall well-received from our alpha testers.

466. 1:16:44

     Yes.

467. 1:16:45

     Can you inject, uh, can you inject audio as a, as a background audio, like, uh, ambience audio, like typing audio and so on?

468. 1:16:54

     Um, you ca-- Like, basic-

469. 1:16:58

     To make like you're in a real office.

470. 1:17:00

     Right. Yeah. Um, you can just intercept the audio that is coming back from the model and then, like, overlay your own audio.

471. 1:17:10

     Oh, yeah.

472. 1:17:12

     Hi. Um, I was wondering what your support is for, like, multiple speakers. Uh, if there's more than one person in a conversation, can it detect who's talking and, and do pauses that way?

473. 1:17:20

     There's no, no current, like, speaker detection-

474. 1:17:23

     Okay

475. 1:17:23

     ... on the model. Um, so it might struggle with that.

476. 1:17:27

     Yeah.

477. 1:17:32

     Um, I just wanted to ask, um, about like custom voices. Is it limited to the preset voices you have, or can I upload my voice as a sample, for instance?

478. 1:17:41

     It's currently limited to the voices that we have. We keep adding new voices, though.

479. 1:17:47

     Is there gonna be support to add custom voices any time in the future?

480. 1:17:50

     Um, at any time in the future? Uh-

481. 1:17:53

     Uh, uh, not, not like, not like years. [chuckles]

482. 1:17:56

     Uh, [laughs] what-- I mean, what I can say is, like, we're trying to make sure we're finding, like, the safest approach. Like, we have an article on-online that talks about sort of like the responsible take we're trying to take on this, on making sure that, like, the voi-the voice-- like, if we're providing custom voices, that it comes with

483. 1:18:14

     the right guardrails in place and stuff to avoid abuse.

484. 1:18:18

     All right. Thank you.

485. 1:18:19

     You're welcome. Yes.

486. 1:18:21

     Uh, is there still a thirty-minute timeout on the session? And if so, what is the recommendation for longer sessions?

487. 1:18:29

     I don't think that has changed, um, to my knowledge.

488. 1:18:34

     My personal recommendation would be that, like, one of the things that you can do, and this goes back to, like, for example, in the demo that I was showing, um, if you're keeping track of the transcript and stuff, you can re-- Like, when you're starting a new session, you can populate that context by creating new ite-items using

489. 1:18:56

     the API. So, like, one of the things that you could do is starting a new session, if you know what the previous context was because you kept track of it, you can then, um, basically inject that as additional context.

490. 1:19:08

     Is there a timeout event in there?

491. 1:19:11

     Um, what type of event are you looking for?

492. 1:19:14

     Timeout event.

493. 1:19:15

     Oh, if there's a timeout event. Um, I do not know right now.

494. 1:19:21

     But all of our events are documented in the API reference.

495. 1:19:26

     Yeah.

496. 1:19:26

     So when you say, uh, Realtime API can call tools and function calls, uh, do we include like system file reading and writing in those capabilities?

497. 1:19:38

     Can you-- Sorry, can you repeat that question once more?

498. 1:19:41

     Oh, I was wondering the Realtime API can each function call, uh, functions such as system file writing or reading?

499. 1:19:49

     Oh, uh, whether the function calls can do things like system file reading and stuff. I would say it depends on where you're running that AP-- uh, where you're running that, um, realtime session.

500. 1:19:59

     So if you're running it on the server, you can do anything you can do on the server. If it's, if it's running in the browser, then you're limited to whatever things are available in the browser.

501. 1:20:09

     I see. So but-- So locally, like the chat?

502. 1:20:13

     Um, no, you should be able to like-- you could create a like WebSocket-based like, um, voice agent that runs on, on your device. I mean, like it's gonna use the Realtime API for the model, but then like, because the actual tool calls will all get executed on your system, you should have access to whatever aspect of your

503. 1:20:34

     system your program has access to. Cool? Yes.

504. 1:20:41

     So even before we get to voice agents, we all need bigger, better, and more diverse evaluation sets, especially for anything around function calling and parameterization. Uh, do you have any best practices or suggestions for how we now take evaluation into the voice world?

505. 1:20:56

     Should we keep things in text and then turn them-- you know, use text-to-speech to have voice versions of it? Just if you have any suggestions for how we evaluate the, the full range of inputs that we would expect users to bring to this.

506. 1:21:09

     I mean, one of my suggestions would be if you can go to the leadership track, go to Anup's talk, um, which is tomorrow, I think. Yeah. Um [laughs] he's gonna talk a lot more about like additional best practices of what we've learned, um, in, uh, building voice agents.

507. 1:21:24

     I would say like if you can hold on to the audio, like it, it's helpful. Um, obviously transcriptions definitely, but like it, it's sort of like the audio is still the thing that is like the most powerful thing, especially for speech-to-speech models where you have the model act on the speech, not on the text, right?

508. 1:21:45

     Mm-hmm.

509. 1:21:46

     And like this is one of the, one of the few things where like the sp-- uh, the chained approach obviously makes some of this much more approachable because if you have-- if, if your, if your agent is running on text anyways and you can just store the text and rerun it, that makes that part of evals a

510. 1:22:03

     bit easier.

511. 1:22:05

     That makes sense. And then also for those of us who might be thinking about launching a new voice agent, how would you suggest evaluating it before we get to that stage that we'd have, uh, customer interactions to work with?

512. 1:22:16

     Um, I would start with like having like, um-- I think it goes back to like, um...

513. 1:22:25

     Like some of this is like-- Uh, where is it?

514. 1:22:30

     There. Um, like human review is an excellent solution for this, right? So like have it-- like have a system-- Like, like one of the big things with like things like Lemonade and stuff is like they're able to go through all of these calls and like get an idea, but they also have their own like predetermined set of

515. 1:22:49

     examples that they might wanna test as they're developing the agent. So like that's a great first way. I would scope clearly the problem you're trying to solve as well.

516. 1:22:58

     Like if you're, if you're trying to sort of boil the ocean, it makes a lot of this significantly harder as opposed to like well scoping what the agent should be able to do and what it shouldn't be able to do.

517. 1:23:10

     Makes sense. Thank you.

518. 1:23:10

     You're welcome.

519. 1:23:12

     Get-getting on the same path of our friend over there, uh, actually when I'm testing my agents, my text agents conversational, uh, I use Promptful.

520. 1:23:22

     Mm-hmm.

521. 1:23:22

     It's a platform for do all the prompt testing. Could I put another agent to talk with this, this agent to do all the evaluation?

522. 1:23:34

     I think you can try it. Um, like I don't think it shouldn't work. Uh [laughing] um, like-

523. 1:23:42

     So I put another voice agent talking to that agent to try to execute all the prompts, and then I could-

524. 1:23:48

     Yeah

525. 1:23:49

     ... get like the transcription-

526. 1:23:51

     Yeah, I mean-

527. 1:23:52

     ... accuracy?

528. 1:23:52

     I mean, I know we have, we have use cases where customers also use our models to like prompt humans, right? So like [laughs] um, where it's like for, for like training use cases or other things, for example.

529. 1:24:05

     Awesome.

530. 1:24:06

     Um, so should work out. But I don't know if anyone uses that kind of approach in-- Lemonade does?

531. 1:24:13

     Oh, cool. So the second, the second picture is, is exactly that.

532. 1:24:17

     Awesome. Thank you.

533. 1:24:19

     You're welcome. Any, any other questions? Yeah, go ahead.

534. 1:24:26

     Slightly related. Uh, do you have something around wake word detection on the Realtime API roadmap or-

535. 1:24:32

     Sorry

536. 1:24:33

     ... patterns for wake words?

537. 1:24:34

     Oh, reports? Um-

538. 1:24:35

     No, no. Wake words, so that's like-

539. 1:24:37

     Oh, wake words

540. 1:24:37

     ... activating the-

541. 1:24:38

     Oh

542. 1:24:39

     ... yeah, the KCD.

543. 1:24:39

     Um, no, we don't have any wake words built in or anything.

544. 1:24:44

     No patterns either? Like any patterns to avoid costs?

545. 1:24:48

     Um, no.

546. 1:24:48

     Just one device, I guess.

547. 1:24:49

     No.

548. 1:24:50

     No. Okay.

549. 1:24:51

     You could basically like-- What you can do is you can build your-- like you can turn off our v- uh, voice activity detection and then build your own. Um, and then basically use that.

550. 1:25:01

     So like you could use a model that has like a, like a VAD, uh, voice activity detection model that has wake words in it and then like do it that way and basically commit all of that audio to our API and then send like a commit, like a commit event.

551. 1:25:19

     Cool.

552. 1:25:20

     Thanks.

553. 1:25:23

     Well, awesome. Thank you so much for taking the time and spending the afternoon with me. [clapping] [outro music]