Function Calling is All You Need

Talk transcript

1. 0:00

   Cool. Okay, let's get on with it. So hi, everyone. My name is Ilan. Uh, I'm on the developer experience team at OpenAI. Um, unfortunately, I can't be there in person, as much as I would love to.

2. 0:12

   I'm in a wedding in Costa Rica, um, which is happening later today, so I just wanted to take this opportunity to just talk through, um, one of my favorite concepts in maybe all of, like, AI and language models.

3. 0:25

   So, uh, title of this talk is Function Calling is All You Need. It's a talk workshop. There's gonna be, um, a lot of coding. Please save your questions... No, I'm kidding.

4. 0:34

   Like, just interrupt at any point. Uh, we have a Slack, send them here. Um, yeah, just send them at any point. Uh, and if you wanna, like, unmute yourself and, or raise your hand, I'll call you off, like, as we go.

5. 0:46

   The idea is to keep this super, super dynamic, um, since we have a bit of time. Um, I'll be fielding a lot of, like, questions and requests and then trying to be coding as much as possible.

6. 0:57

   So yeah, this talk is gonna be some lecturing, uh, a lot of coding from scratch, and then some debugging, hopefully not a lot. Uh, so this is a little bit of what the, uh, workshop is gonna look like.

7. 1:09

   We're gonna go over a little brief history of the toolformers, of, of, I'm sorry, of function calling, um, then do a little crash course on function calling, um, talk about just agents, how they're just loops, how RAG workflows and more are just function calls, delegation, and asynchrony, couple random things I've found, and then we are gonna do

8. 1:28

   a Q&A. Um, if you just wanna see, like, the meat of it, this is pretty much it. We're gonna do, like, this is everything I wanna talk about, uh, and everything that we're gonna implement, so this is, uh, just a little bit upfront.

9. 1:45

   Great. So a little history. Um, if we look at the abstractions and sort of patterns that we managed to do with language models, it started as text completion, right?

10. 1:55

    Like, the original GPT, uh, and the GPT-2 and the GPT-3 were all just base models, where you gave them some, some input text, um, and then they would just continue the sentence.

11. 2:05

    Um, this was, at the time, really, really, like, interesting. This was the first time we were able to do, like, very, like, uh, English-sounding, like, real-sounding language. Um, but getting it to follow instructions was pretty hard.

12. 2:16

    So if any of you were testing this back then, you might remember how setting up a chatbot was, um, non-trivial, right? You had to, like, get it to answer questions, but if you just say, like, you know, "What is the best way to, uh, get to the, like, park?"

13. 2:31

    or something, it would continue, like, like, "What is the best way to get to the park? Um, that is what Sally said yesterday," right? And you wouldn't actually get a, a response.

14. 2:41

    Um, so you had to, like, structure it in a way where you would say, like, "This is a question. This is the answer." Question, answer, few shot, and then give it.

15. 2:48

    Um, then they introduced, uh, I think this was actually us, we introduced, uh, function ca- um, instruction following with, uh, InstructGPT. Now you could give it some input, uh, and it would actually do what you're asking as opposed to just completing.

16. 3:01

    Um, finally, we started to introduce this notion of, like, users and assistants and roles, um, and this was all, all done through, through post-training, where you actually gain these personas.

17. 3:11

    Um, and then finally, uh, we, we eventually landed on this, like, you can give it additional tools, um, in order to do, like, external, um, any, like, interact with external states.

18. 3:22

    So this is what the previous playground used to look like. But as you can see, there's no chat. This is just, like, a window, and it'll complete. Now, looking over at, like, the original papers, which was pretty interesting, um, one of the first times that we actually started to do this, like, function calling was through this WebGPT,

19. 3:41

    um, which was this, uh, version of GPT-3 that we trained to be able to-

20. 3:45

    Okay, um, Ilan, I have to cut in here a little bit. We got the Zoom working. Yay. [laughs]

21. 3:51

    Sweet.

22. 3:52

    Um, all right. We're gonna cut over in the audio, so can everyone hear Ilan when he speaks? Uh, Ilan, say something.

23. 3:58

    Hello. Hi, everyone. Hello, hello. How's it going?

24. 4:05

    Okay, so, so we're gonna cut over a little bit in terms of, like, um, people having their own personal audio situation going on. So if you mute your machine, you should be able to hear it on the room, uh, audio, whatever you wanna do.

25. 4:19

    But you can also obviously connect. But, uh, yeah, now we have him on the big screen.

26. 4:24

    Sweet.

27. 4:24

    So okay, yeah. Uh, let's, let's keep going for a bit. I'll, I'll cut in again if there's audio issues. [laughs]

28. 4:29

    Sounds great.

29. 4:29

    Or maybe you wanna say some test words. [laughs]

30. 4:32

    Uh, you're all great.

31. 4:33

    Okay.

32. 4:33

    Thank you for coming.

33. 4:35

    Can we bump up the audio? Where is the guy? [laughs]

34. 4:43

    I think it's not a conference if you don't have AV issues.

35. 4:46

    Okay. Uh, it's, it's still too soft. We can't, we can't hear you. Um, I'm gonna try and bump up the audio. I'm so sorry. [laughs]

36. 4:53

    I'll put up a more interesting slide in the meantime. You guys can look at this while we figure this out.

37. 4:58

    Where's, where's the fucking video guys? Hey, um, Ilan.

38. 5:22

    So anybody got, uh, good jokes? Maybe.

39. 5:30

    I always ask OpenAI for my jokes.

40. 5:33

    It's always the same one. [laughs] I think it's, uh, what was it? Why did the...

41. 5:41

    No, I had it, I had it here. Wait, wait, wait, wait, wait, wait. There we go.

42. 5:45

    I don't trust-

43. 5:45

    Why can't you trust an atom?

44. 5:47

    Yes.

45. 5:47

    Yeah, because they make up everything. [laughs] These are all real, by the way. And for, for those who, like, have a keen eye, the first one is, like, actually from the GPT-2 paper.

46. 5:57

    Um, we, like, gave the model- This, like, description of, like, unicorns in the Andes Mountains, and that was, like, the big, like, first time that it was, like, doing multi-paragraph completion, like continuations that referenced earlier parts of the conversation.

47. 6:16

    It's cool stuff. It's cool history. Can you guys hear me? [laughs]

48. 6:22

    I feel like some people are having a great time.

49. 6:24

    Yes, we can.

50. 6:25

    Some people don't know what's going on. Can, like ... I don't know. Am I good to k- keep going, or should I keep waiting?

51. 6:31

    Uh, I think, I think you're good.

52. 6:33

    Okay. Great. Great, great. I do think this is the vibe of the whole talk, by the way. Th- there's n- it doesn't get more structured from here. Um, okay, so as I was saying, uh, we did, like, this WebGPT paper.

53. 6:46

    Um, essentially we trained a GPT-3 version of the model, uh, or, like a GPT-3 model to be able to use the, like, this very specific set of functions, um, to do web search, and this was, like, back in 2021, so really we had, like, WebGPT a long time ago.

54. 7:01

    Um, but this is, like, one of the first times or maybe the first time that, like, we were having, like ... It's not just generating text, but it's generating actions, and then we're parsing those actions and then introducing it back into context so it can use the responses itself.

55. 7:14

    Um, and how we trained it, there, um, there, there's, like, these, these, like, you know, clever ways of, like, we, we essentially gave people, uh, an interface and let them do the searching.

56. 7:25

    Um, and then I think we took it, like, Reddit, um, yeah, explain like I'm [REDACTED:age], um, and then just, like, had people complete tasks, and they could use these commands.

57. 7:35

    And so we taught the model, and this was GPT-3, right, um, how to essentially imitate users' behavior and then produce responses that were preferred. Um, and, and this was pretty cool.

58. 7:45

    Like, this is how you, you, we, like, start to saw, um, to see, like, this, uh, this use of, like, structured, like, actions essentially. So but this was very specific, right?

59. 7:55

    We were training, like, very specific, um, tools. So then you might be familiar with this paper. This was from Meta at the time, um, where they essentially had a way to teach the models how to use any tools.

60. 8:09

    Um, and they taught, uh, like, they, they used a few tools. I think it was, like, QA, calculator, um, what was this? Like, translation. Um, couple other, couple other tools.

61. 8:20

    But, um, it was actually a pretty clever way where they, like, looked at the log probs at each spot to, like, see where it was, like, best suited to, like, retroactively put in a function call given some, like, completion.

62. 8:34

    Um, so here we can see a few examples. Like, um, essentially it's like if you have a calculator call, if you insert a calculator call that, like, you know, it'll, it'll insert the, um, the actual call to the calculator, which then, you know, you're pretty familiar at this point, will get the answer.

63. 8:51

    If you insert it in the right spot in the sentence, it actually reduces the perplexity of the sentence. Um, and so they didn't actually have a lot of human labeled examples, or I think it was just a few, but it was, uh, really cool because it was this way of, like, um, it could learn to use, uh,

64. 9:08

    any of these tools through this, like, crazy, like, log probs technique. Um, and it was, uh, I was pretty excited when I saw this paper. Now, this is, like, how it learns to use any of these tools.

65. 9:21

    Um, but then finally, uh, in June of 2023, O- uh, OpenAI launched just general function calling, where we essentially, like, pre-trained it to be able to use these tools, or act- actually post-trained it to be able to use, uh, tools.

66. 9:32

    So now you don't actually have to, like, give it ... Like, you can give it examples, but, uh, we just showed it, like, this, like, syntax in with functions that we still use today, and it's just able to call functions.

67. 9:43

    So this is a brief history of, of function calling. Um, and my, I guess, my argument is this is really most of what you need for all the exciting stuff that's happening today.

68. 9:55

    Um, there's, there's obviously, like, additional, like, systems you can use and, like, um, more post-training you can do, but fundamentally, like, functions are so, so, so powerful, and we're gonna look at a few cases today.

69. 10:07

    Um, I'm gonna try to keep an eye on questions, but, um,

70. 10:13

    yeah. Okay.

71. 10:15

    I think you're good so far.

72. 10:17

    Cool, cool. So let's do a super quick crash course on function calling. Uh, two main purposes, and I'm ripping a lot of this from the docs. So, um, they're the fetching data, right?

73. 10:28

    Reading APIs, retrieval, memory, or taking action. Any APIs you can use to write. Managing application state, which is actually pretty overloaded. That can be, like, UI, front end, back end, whatever you want.

74. 10:39

    And then workflow actions, which is, um, any, like, multi-step processes or even, like, meta actions, like switching its own prompt, or, like, loading in different tools, or, like, handing off a conversation, right?

75. 10:53

    Um, so this is a- also a diagram straight from the docs, but, um, I'll quickly brush past this. I'm gonna assume, like, most people have at least seen this, but here it is.

76. 11:02

    It's, uh, you, you essentially tell the model which functions you, you want it to be able to use. Um, and you also provide, like, whatever the user input is.

77. 11:11

    Um, the fu- the model tells you what it wants to do with that function, but it doesn't actually do it. This is one of the, like, a big sticking point of function calling.

78. 11:18

    It doesn't actually use the function itself. It tells you, like, it tells you the intent of what it wants to do with the function. You are then responsible for parsing that, executing the code, doing whatever you want with it, and then providing the result back to the model.

79. 11:32

    And then the model can use that respo- like, result in a, in the generation. Uh, take a look at a quick question here.

80. 11:40

    Oh, no. It's just, it's just winks. Okay.

81. 11:44

    Cool. Um, these are just a few, uh, best practices. This is all taken from the docs as well. Um, you wanna write clear functions. An important one is you gotta apply software engineering best practices when you write these functions.

82. 11:56

    So, um, you know what? Maybe I'll pull up the docs for this.

83. 12:00

    Um- So this is a lot of big text, but essentially, um,

84. 12:11

    th- this is a lot of value here. I tried to pack as much, like, useful information here as, as I could, so I'm gonna quickly go over it, right?

85. 12:17

    Um, you gotta explain the purpose of each parameter, use a system prompt, and include examples. That's, you know, pretty, pretty, like, n- non-controversial. Um, software engineering best practices is a little bit, uh, more interesting, right?

86. 12:31

    You gotta make functions obvious and intuitive, and they gotta follow the function of least principle. Um, like if you give this to a person and they don't know how to use it, then the model might not either, right?

87. 12:42

    You know, models are getting smarter than us, but still you gotta make it, uh, gotta make it easy. Um, also you gotta use enums and obj- object structure to make sure that, like, you are not letting the model make invalid calls, right?

88. 12:55

    Um, like here you, you have this like toggle, like, uh, um, function that takes in like two, two Boolean params and like, obviously, like this is pretty wrong, but, um, there's actually many, many more like subtle cases where, um, like the-- you can, like you're letting it represent invalid states.

89. 13:17

    Um, [laughs] okay. Still no questions. It sucks. Okay. Um, great. And then this is really, uh-

90. 13:27

    There actually... Sorry, there are a couple of questions in the Slack.

91. 13:30

    Okay.

92. 13:30

    If I saw it rather than

93. 13:32

    I see. I see. Um, here, maybe let me see if I can pull up the Slack.

94. 13:41

    Okie dokie. Da, da, da, da, da, da, da.

95. 13:45

    Between functions and tools in my opinion. Hey, Sam. Yes. Um, I, I think we've all been kind of like gravitating toward functions and tools as like the two main ways.

96. 13:57

    Originally it was just functions. Tools was like later we renamed it. Um, I think now, uh, and the way that I tried to specify it in the docs is functions are like the, like raw function calling, right?

97. 14:10

    Like you provide an interface and you are responsible for executing the code. Um, tools, and this is sort of how we treat it in our API, is a super set of functions.

98. 14:19

    Tools include functions, but it also includes things like, um, code interpreter or file search or any of these like, um, like hosted solutions.

99. 14:32

    I'd say this is not be like end all be all definition, but this is a definition that we've adopted, right? There's like tools which are like hosted tools, uh, including functions and then functions is a subset.

100. 14:42

     Cool. Maybe we can just jump into it. Um, if people have questions on this, happy to, happy to field them as well, but I kind of wanna get coding.

101. 14:50

     Okay. Uh, when you start to approach dozens or hundreds of functions, what technically should we apply in order to effectively tool call?

102. 14:58

     Permissions is one technique. Okay. Interesting. And then question two, when you require one tool to provide inputs to another, I have seen tools become layered. How should a reasoner hard code?

103. 15:08

     Great. Okay. These are actually great questions and, um, I think to, to answer them, I might, I might like do a little hack and like, uh, use some existing, uh, code.

104. 15:20

     I'm gonna use Swarm for a little bit of this, um, because it does some nice function calling. Actually, no. You know what? I'm gonna get to these func- uh, in a second.

105. 15:28

     Um, let's just go straight into the, uh, and as always, we should start from the docs.

106. 15:38

     Uh, cool. So-

107. 15:40

     Uh, so this is where I always step in. Can you zoom in on your screen?

108. 15:45

     Yeah.

109. 15:45

     Every single screen page.

110. 15:47

     Yeah.

111. 15:47

     Bigger.

112. 15:49

     Cool.

113. 15:51

     Yeah. There. Yep.

114. 15:54

     Sorry.

115. 15:54

     This is the thing I always do.

116. 15:56

     Yeah, no, this, this is a good call out. Uh, and then I think the

117. 16:02

     terminal here should be good. Okay. Sweet. Uh, we have, we have a function. Great. Um, and then ...

118. 16:17

     And just run it. And we have, uh, the function call right. Now we're not handling it yet. So this is where like I might skip ahead a little bit, um, and start doing some of the like agentic stuff.

119. 16:34

     But first, uh, first off, uh, we, we gotta have a loop, right? Like, you know, you, you, I can sort of hear myself. Um, I think Sohum. There we go.

120. 16:45

     Perfect. Okay. So, um, I think the idea here is like,

121. 16:51

     let's make a very, very, very simple like, uh, input loop. Like let's do, uh, you know, while-

122. 16:58

     Can you also switch to light mode?

123. 17:01

     Light mode?

124. 17:02

     Yeah.

125. 17:04

     Jesus. Okay.

126. 17:05

     Yeah.

127. 17:06

     Sure.

128. 17:07

     I'm, I'm just very experienced with this. There we go.

129. 17:12

     What's the least painful? This is good, right? This works?

130. 17:18

     Yeah.

131. 17:20

     Ugh, my poor eyes. Um, actually maybe I, I, I can't quite see the room, but show of hands, like who has used or implemented function calling in the past?

132. 17:31

     And someone's gonna have to gauge this for me.

133. 17:37

     Okay. It's everyone. Great. I'm gonna skip ahead a little. Um-

134. 17:41

     There's like 10, 20 people, 20% that hasn't done it.

135. 17:46

     Okay. 10, 20%. Um, the important part that you have to know is you can define the function, um, schema, right? Then the tool will specify what you want, and then in this case, you know, if I have like a

136. 18:01

     get weather tool-- Actually, I'm gonna grab this from the docs as well. When in doubt, just go to the docs, you know? It's always good. So-

137. 18:12

     Step one. No, this is all node. Yeah, and this will be useful for later too. Um,

138. 18:20

     step one, call the function. Step two, execute your code,

139. 18:27

     right? So here what we're doing is we're taking, we're parsing out what the, what the function told us. We're parsing out the args, and then we're calling this get weather function, which we don't have yet.

140. 18:36

     Um, but conveniently it's up here, right? So have it up here,

141. 18:42

     get weather requests. And then the last step is we provide the result back to the model. Um,

142. 18:53

     uh, provide the result back to the model and then ask for our completion, right? So just in order, specify the tools, call it the first time, get the tool calls, parse them out, call the function, append, and do that, right?

143. 19:05

     So if we do that, and we just add maybe a, like, add... Nah, I'm gonna do this by hand.

144. 19:14

     Why not? Yeah, so I'll print the completion here, sort it by hand, and then print the last completion.

145. 19:24

     Then what we can see is we'll get the first one that includes a tool call. It'll call the actual temperature, uh, call the weather API, and then we'll get a response.

146. 19:38

     And then it says the current temperature in Paris is something. I can't see it because of Zoom. Um, cool. So this is like the very, very basic setup for a function calling.

147. 19:49

     Let's take a step forward. So this is an agent, a very, very, very basic implementation of an agent, um, that I'm gonna go through really, really quickly because we're gonna start using it.

148. 19:58

     This is very familiar to what you'll see in Swarm or any of the o- other, like, like, basic frameworks. Um, but the idea here is in-- uh, as you can see, like, in the original one, essentially, like, when it had a tool call, I wanted to provide it back.

149. 20:15

     So what I do here is while, like, just keep looping, and this is, like, the very famous, like, agents are a loop. This is, this is that loop. Um,

150. 20:25

     specify the tools, call the model, get the message, print it out, handle the tool calls, append it. And once we have no more tool b- calls, break. This is the whole loop.

151. 20:39

     I called it run full turn in my head. One turn is just like, you let the model do everything. Um, and then we have this, like, execute tool call, yada, yada, whatever.

152. 20:49

     Um, so now we can use this. So

153. 20:59

     agents.py. Did I export it? There we go.

154. 21:25

     I love it. Okay, so now we're just gonna specify one.

155. 21:31

     Um, you know what? Yeah. We w- we, um-- And we'll, we'll do it, we'll do it with a, with a simple loop.

156. 21:42

     So we have this, that, and we have this. Okay. So now we can just do agents.run_full_turn. The one other thing that I'm adding here that I didn't show is, um, this simple utils,

157. 21:56

     uh, that defines this very, very, very useful function. So functions to schema essentially takes in a raw Python object function and then provides it into the, uh, like, correct schema, um, so that you can just define functions directly.

158. 22:12

     And this is the same thing that we have in Swarm, and there's in a few other frameworks now as well. Um, so as an example, we can do like, you know, get weather, and I'm just going to like, you know, return twenty.

159. 22:26

     Like, degrees Celsius. Great. I'll do that here.

160. 22:40

     And then I'll have my messages, imprint messages.

161. 22:48

     Is this emergency reference? I think it is.

162. 22:58

     What did I call it? Oh, oh, oh, oh, oh.

163. 23:07

     I see. This is not part of the agent class.

164. 23:12

     Yeah. Boom. So we called the weather, printed it out, uh, gave us a nice little completion.

165. 23:31

     Um, this is essentially what we wanna see. If I say, um... And then I can just keep, keep going, right? So we, we, we have this basic, um, basic loop.

166. 23:42

     Um, now let's get back to the presentation real quick. That was like a very, very immediate crash course. Now let's, let's get interesting. Um, and, and just for convenience, I'm gonna like pivot to the Swarm implementation.

167. 23:56

     Um, the main reason being it's pretty much the same, um, except we have like some convenient, uh, like looping tools. So, uh,

168. 24:06

     Swarm, Swarm imports agents, and then There we go.

169. 24:23

     So now we can do a simple agent and run a demo loop, and let's see this work.

170. 24:44

     Cool. Um, we have this very basic setup. Let's do everything now. So, um,

171. 24:51

     another show of hands, how-- who here has implemented, uh, RAG?

172. 24:57

     Okay. What about memory? Okay, fewer. And then what about like, you know, multi, multi-step things and workflows?

173. 25:07

     Cool. Um, how about this? There's a lot of stuff we could talk about, but I wanna keep this valuable to you all. Out of this list, can you like just type in the Slack what you wanna, what you wanna see?

174. 25:20

     And I can just change the order in which we'll cover this, um, because they're pretty, they're pretty interchangeable. We can, we can build up. Um, but essentially there's, there's more interesting things

175. 25:30

     later on, but I wanna make sure we, we can build up to them. So just, uh, if you can pull up the Slack and just dump in, you know, what you're interested in seeing.

176. 25:48

     Okay. Lots of memory I see. Um, function generation from the docs, delegation async. Okay.

177. 26:00

     Good old delegation. [laughs] And random cool stuff. I will get to the random cool stuff. Um,

178. 26:10

     okey-dokey. Back to light mode. Um, yeah, let's start with like a very, very basic form of memory, right? Um,

179. 26:20

     how would we-- how do we do this? Let's see.

180. 26:24

     Honestly, like we can have, um, just a list, right? Memory can just be this list, and I'm gonna implement this similar to how it's done in ChatGPT.

181. 26:35

     Um, but ju-just to show like I think the whole point of this talk is like-- or th-this workshop is like doing things from first, first principles and just really removing the complexity.

182. 26:45

     I think there's like a lot of like, not fake complexity, but like added complexity on things that like doesn't really need to be there a lot of the time.

183. 26:51

     Like concepts are a lot more simple, and it's all about function calling. So, uh, we can do, you know, add like add to memory, [laughs] append it, and then get memory.

184. 27:05

     Uh, th- like this is super, super simple. Um,

185. 27:11

     is this, is this good enough? Let's see. Maybe. Maybe. And so what we can do is

186. 27:19

     give it the tools. Let's see. And then when, when would we wanna use them? Let's say you like-- I can just say, so this comment is gonna be used as the, uh, string, uh, as the, as the description in the function.

187. 27:32

     So I can say like, you know, like when the user tells you something like factual

188. 27:42

     about themselves, their life or... Man, I can't see anything. Okay. Or their

189. 27:56

     references, call this function. Um, memory. Um, and we'll add a couple more cool things like expiration, which is one that I've kind of wanted to add for a while.

190. 28:15

     So, you know, for now, let's say false.

191. 28:21

     Um, memory.append, uh, bank. And then we have, I guess what we'd call it, uh, you know, memory text.

192. 28:41

     And we can say, you know, keep the memory text short size. Great. Cursor knows what I want.

193. 28:48

     And then we can just return maybe like...

194. 28:57

     Like this is a super, super, super naive implementation. Um, but now, uh, when we start off, I guess we could even start off with a...

195. 29:13

     I mean, this is gonna be kind of hacky, but I can just like, um, in your first turn, always call get memory. This is not, uh, great. It's just because of the demo loop, but maybe I'll break, I'll break the demo loop out so we can actually do this by hand.

196. 29:30

     But so how, how could we prove this? Um,

197. 29:39

     maybe let's say like, you know, write this

198. 29:43

     memory bank or like keep this memory bank in a local file, JSON file. Read it in at

199. 29:56

     beginning and write it out at every case.

200. 30:08

     Cool, cool, cool, cool. Okay, I trust this. Shall we test it out? So, [echoing]

201. 30:15

     wait, someone mute themselves. Uh, da, da, da, da, da. Okay, we got the memory. Memory. Can any-- Can anyone see any bugs? [laughs]

202. 30:26

     'Cause we're about to test this out. So we got the loop. It's gonna call it- Um, and maybe like let's--

203. 30:34

     memory. Okay. Sure. Yeah, let's try it. Let's try it out, see what happens.

204. 30:44

     So hi. [laughs] Did I not give it the functions?

205. 31:00

     Is it hallucinating this? Oh, yes, because I think we called it functions.

206. 31:10

     Hi. There we go. Okay, so called get memory. There's nothing there. I can just say like, um, I am [REDACTED:physical_attribute] despite what people think.

207. 31:26

     Uh, cool. Uh, so now let's just check, right? It should have written this out. There we go. So now we have it in the memory bank. So now I can actually, uh, end this, right?

208. 31:36

     And then be like, you know, uh, how tall am I?

209. 31:43

     Ta-da, we've implemented memory, right? There's a-- Yes, I'll take a clap. I saw L-L-Luis, you're like my proxy for the-- Luis, uh, Luis Costa. You're my proxy for the audience.

210. 31:54

     You're like the only person I can really see. So please don't, uh, turn off your camera.

211. 31:59

     Um, amazing. Now we can do more interesting things, right? We can, uh, if we want, do a little bit of like smart querying, uh, where instead of just like loading in all of the memory, um, we can like do a little bit of like retrieval, uh, to load in the right ones.

212. 32:17

     Um, and use like semantic similarity or use some kind of search. Um,

213. 32:23

     I, I could try to implement that. Um, that might take a little bit longer, but not that long. But I do wanna pause here. So like given this, and like this is gonna be the style of things that we do, like what, um, what do we wanna see next?

214. 32:36

     I could just keep going with this example. I can pivot. Uh, it could be fun just to keep building on this, see how far we can get.

215. 32:43

     Let's see. Uh, delegation async, have it chat and work in the background. We'll get there. We'll get there. I got that working this morning on Python because I f-- didn't wanna switch to, to Node yet.

216. 33:00

     Uh, delegation async. Okay. Let's get into delegation then. So there's a few different ways we can do this. I'm actually gonna leave the memory, and we're just gonna keep building th-on this, on this agent.

217. 33:14

     Uh, and I am using the Swarm agent just because I didn't wanna debug the one that I implemented. But like if we actually look at what this is, um, like

218. 33:23

     it is very, very, very simple. And then the like run demo loop itself is like, uh, just printing out messages.

219. 33:34

     Um, and what it's doing is like appending client.run, whatever. And like if we look at this client.run, um, if it doesn't stream, it essentially does exactly what we did before.

220. 33:44

     Like keeps looping, get the completion, throw the messages in,

221. 33:50

     uh, append them, handle responses, et cetera. There's a couple more things around context and handoffs that we like don't, don't really have to look at today. Um, but we can.

222. 34:01

     So cool, we have memory. Let's do delegation. Um,

223. 34:08

     so there's a couple ways we can do this, right? Um, there's--

224. 34:12

     If you think of like functions and agents and, and everything, you can, um-- Maybe I'll skip to this slide.

225. 34:20

     Skip, skip, skip, skip, skip. These are like a few of the forms of like agents and delegation that people might be familiar with. We have handoffs, which is like the Swarm style.

226. 34:28

     You take a conversation and fully swap it to a different agent. Um, and what that means is just like replacing the system prompt, replacing the tools. Um, you can have nested calls, which are the easiest to implement and like often somewhat overlooked.

227. 34:42

     Um, and then you can have manager tasks, that's more async. Uh, we will get to that today. So let's do a very basic one, right? Let's say I wanna do like, um,

228. 34:58

     maybe let's give it a chance to like call a bigger model to do a harder task, right? So we can say like, you know, uh, it can delegate to smarter model.

229. 35:12

     Uh, and I'm gonna give like the task description.

230. 35:21

     Um, [laughs] so I'm laughing because this is saying like, it's just gonna make it smarter by telling it to be smarter. That's not-- Despite how well that would work, usually we're not gonna do that.

231. 35:34

     So we're-- we can just make an API request directly. Actually, let's just do that. Let's just do that. So, um, let's do from openai client. And here we could do client.ChatCompletion.create.

232. 35:50

     Let's call o1. Um, I won't provide a system message. I think that's okay. Content is the description. Look at that. And we did it. So now, um...

233. 36:10

     Man, I love Cursor. Okay. Did everyone catch all of that, by the way? All we did was like implement this, this function that calls OpenAI API, and then here.

234. 36:20

     So now I can say like, uh, so now I can add a bit of a description here. I'm like, like, uh, if, you know, if the user asks you to like, um, I don't know, do something

235. 36:38

     Like, that seems difficult or says it's hard, use this instead of

236. 36:47

     trying, trying it yourself. Uh, and it infers, you know, how to use this based on, like, the fact that I called it test description. It's, you know. So let's give this a shot.

237. 36:58

     So hi. Um, let's see. Uh, you know, give me a poem about,

238. 37:08

     I don't know, dogs and the Earth where each

239. 37:15

     other word starts with the next letter the alphabet. It might just try it, because I know GPT-4 can do a variation of this, but, um... Oh, it's giving it a shot.

240. 37:27

     Okay. Let's see if it gets it right.

241. 37:34

     Okay, I don't have, uh, I don't have the patience. Let's say

242. 37:46

     answer briefly, one sentence max. Okay, great. And then

243. 37:57

     to, you know, uh, write a haiku. There we go. It's shorts. Where each other word starts with the next letter of the alphabet, and ends with the previous letter of the alphabet.

244. 38:18

     I don't know. You guys want to try this while this does it?

245. 38:27

     Is it around? Did I lose, did I lose something? Let's see.

246. 38:33

     Hi. Okay. You there? Cool. Um, did I not copy it? Oh, no.

247. 38:48

     Okay. Uh, give me a haiku. Now make sure each word starts with

248. 39:03

     letter of the alphabet starting with A, and each word ends with the previous letter. But this is hard. [laughs]

249. 39:22

     Okay, so it should be making the function call.

250. 39:28

     Oh, I see. I see. I see. Okay, so I'm only printing the function call when it returns, so that's probably why we're waiting so long here. Um, but this is actually a really good, uh, example of like, okay, we are doing, um, this like task delegation technically, and like it is happening in the background and I'm gonna

251. 39:44

     give this a, a sec to figure this out. Um, but like this is obviously a bad experience, right? Like you don't wanna be waiting here. You essentially wanna keep doing other stuff.

252. 39:55

     So... Wow, it's still not, still not back. We'll, we'll, we'll let, we'll let it figure it out. Um,

253. 40:03

     so maybe let's skip straight to async. Um, we're actually got like both more and less time than I thought, so [laughs]

254. 40:12

     I'm gonna let this keep churning away. Um, now let's think about this for a sec, right? If we wanna do something async, it means that like what do we wanna happen?

255. 40:21

     There we go. So call the model, yada, yada, yada. Buzz breezy whispered. Is this correct? I don't know.

256. 40:32

     Wow. Yeah, this is sort of correct. What did it say? Note. [laughs]

257. 40:42

     Great. Great. So it, it did something, right? Um, and it did something that 4o probably could not have, which is great. That's delegation, but we were just sitting there waiting for it.

258. 40:52

     Um, so let's try doing this async. Now

259. 40:58

     I, I, I, I actually want everyone to like kind of just stop and think like how would you implement this, um, in terms of like what behavior do you want?

260. 41:06

     And maybe I, I wanna see people like drop this in the Slack. Just like take a couple minutes, um, and like drop in the Slack like a proposal for how you would wanna do this.

261. 41:19

     And then I might just pick one or we'll like talk about them.

262. 41:32

     Let's call o3. Yeah, not, not yet. Not yet. Yes, import asyncio definitely feels like, uh, important. But I guess the questions are like, you know, when we delegate something,

263. 41:45

     uh, obviously we wanna have this happening in the background. Um, do we want-- like when it finishes, do we want to be... Like do we want it to be injected into the conversation?

264. 41:55

     Do we want it to give us a response? Um, how many-- like do we wanna be able to like

265. 42:01

     interact with tasks that are running? Um, like do we wanna be able to batch stuff?

266. 42:08

     I am not setting up a Kafka cluster. [laughs]

267. 42:16

     I'm gonna give people maybe a couple more, couple more minutes to just like dump some ideas here. Um, batch calls. Uh, yeah, I guess ba- but how, how would, can batch calls work?

268. 42:28

     Maybe Stephanie, if you wanna add some like detail there. Just keep working until it generates a stop word.

269. 42:34

     Right. Okay. Yeah, this is, this is a good idea. JS and setTimeout. Yeah, switching to JS is always a good option because you already have the event loop implemented.

270. 42:46

     Smaller model, smaller model. Okay, you guys are all suggesting some good ideas. Um,

271. 42:52

     but we can actually go simpler, right? Like, like, essentially what we would want, and I can implement like the basic interface of this. Where is my code? Um, is like instead of actually doing this, right, I can like

272. 43:10

     return delegated, right? Response pending. Um, and then later I can say like, you know, check tasks or something.

273. 43:31

     So this, this pattern is actually one I, I quite like a lot, where you, um, you call a model. It's a, a, a function. It's non-blocking, and then later you can, you can check up on them.

274. 43:42

     Now, the thing is, Python i- is like single-threaded, and, uh, async is real. Uh, it's just a little bit tricky. Um, but let's, uh, let's give it a shot.

275. 43:54

     So I do have some async emergency reference. Now, here's the, here's the, um, thing to notice. When something-- like, for this to work correctly,

276. 44:06

     we have this loop that asks for our input and blocks on our input, um, and like is displaying that on the screen, and we don't wanna have that be displayed while it's also, like, injecting messages.

277. 44:18

     Um, so we essentially wanna separate out, like, where you give user input and where actually interesting stuff happens. So that's what I've done in this, in the basics of this async folder, so let's take a look at that.

278. 44:35

     Cool. Um, it looks complicated. It's not that-- really that bad. So, um, I'm using WebSockets. Essentially what I'm doing-- or Socket, sorry. Essentially what I'm doing is, um,

279. 44:49

     the-- I have a server, and all it does is, like,

280. 44:53

     has a handle user input function, uh, and a start message processor. Um, th-these aren't super important. We can take a look at them in a second. But essentially it just, like, waits, uh, on a specific, like, uh, port and, and host.

281. 45:07

     Um, and then I have a client that connects to that, and it just does while true loops and let me, lets me enter in. Right? So if we look at what that looks like,

282. 45:17

     I have it here. Um, so let's go to async. Let me zoom in here a little bit.

283. 45:32

     Also, um, while I do this, is there-- are there any questions that I've, that I've missed? Um, and if there are, just feel free to shout them out.

284. 45:47

     No? Great. Cool. I've covered everything. Everyone is, uh, perfectly up to date with everything said. Okay, um,

285. 45:59

     server dot py. Okay. So I have this, and now I can say like, "Hi."

286. 46:14

     Let me show you the, the rest of this. I guess that's probably important. Uh, da, da, da, da, da. So we have the server. We have the client. We also have the-- a very, very, very basic agent imple-implementation.

287. 46:26

     It looks just like the old one, with the main exception that I'm using Async OpenAI instead. Um,

288. 46:33

     handling tool calls is happening in parallel. So for each tool call, grab it, create the task, um,

289. 46:43

     await for them, and then just, like, await for them all at the same time. Um, and this just lets it, like, create a bunch of tasks that are all gonna run in parallel, and once they're done, like, if they, if they yield back at any point, we can keep doing other stuff.

290. 46:56

     Um, by the way, if these are just, like, heavy functions that are doing heavy processing, it's not gonna matter. The fact that they're in parallel, like, in sync, it means that it's still gonna happen one after the other.

291. 47:04

     But if they're like network calls, so to other models, then it's perfect. Um, the run full turn is the same one. You, like, call a model, check to see if there's any function calls.

292. 47:14

     If not, break, return the response. Um, the only difference here is we're awaiting, uh, the chat completion, right? This is the only difference. And then if we take a look at our agent handler, we've declared, we've declared our agent like normal, right?

293. 47:28

     Right now it has no instructions. It looks very familiar. And this loop is also pretty familiar. It's like,

294. 47:34

     get the messages. Um, here, here's the only difference, right? Um, and you'll see why this matters in a sec. I have a message queue. Uh, and this will be useful because we don't want to process multiple messages in the same conversation at the same time.

295. 47:51

     Like, we want work to happen at the same time, but we don't want, um, like, multiple generations to happen with the same history because then you'll get conflicting. Like, like, if two messages or if two functions return and they both need to be, like, handled, uh, you essentially, while you want them to happen in parallel, the results

296. 48:09

     should, should only come in one after the other. So we have a queue for that, uh, and it treats user messages as well like that. So this is the handle user input that is being called from the server.

297. 48:19

     Essentially just throws in a user input message into the message queue. Uh, and all we're doing here is, like, pulling off a queue, run it, like, you know, uh, put, put the messages back in the, in the message array, uh, and then, like, sleep.

298. 48:37

     Right? This is, like, just a very simple, uh, loop, but it does it with async. So this is what we saw, and, and it feels pretty normal, right? I can say like, you know, "My name is Ilan," and you can say, you know, "What is my name?"

299. 48:53

     See none. Great. Um, now this still doesn't exactly answer how we're gonna do things asynchronously, but it does give us the space to play with it because now things are happening async, and we can do a few more delegations that, that are actually async.

300. 49:09

     So let's start with a, you know, simple blocking get weather function. I think I required all functions to be async here.

301. 49:25

     Um... Yeah, sixty-seven and sunny. Uh, and then

302. 49:31

     let's give it that, and let's test it out.

303. 49:34

     The only annoying part is we have to, like, restart two things now. So hi, uh, what's the weather?

304. 49:46

     Great. So we can see it called a function, got the response. So far everything's normal. Now let's do some delegation stuff. So let's say,

305. 49:59

     let's say we want to call this function,

306. 50:03

     uh, three times for different places. So we have location. Um,

307. 50:12

     yeah. And let's say, you know, like pick a random

308. 50:16

     number from fifty to eighty to return. Um,

309. 50:26

     so if we do it in the previous case, do I still have the round? Yeah. Maybe, maybe not.

310. 50:42

     Okay. Uh, so if we look at... Okay, this is the non-async one. Do we have weather? We don't have weather. Let's give it weather.

311. 50:52

     Let's give it the same weather function. Cool.

312. 50:59

     Um, we only need random. We don't need this, and it's not async.

313. 51:12

     Okay, so now we have this weather function. Let's test th- this out in the non-async case.

314. 51:18

     Um, I'm gonna get rid of the rest for now.

315. 51:22

     Um, so if I say, you know, uh, weather in SF.

316. 51:40

     Cool. Calls it, fantastic. And now it's like weather in SF, New York, uh, you know, and the [REDACTED:age] other cities.

317. 51:53

     It's, um, it's still gonna do the parallel function calling, and here it's fine because they all return immediately, right? But now let's add like an artificial weight. So let's say time.sleep one, right?

318. 52:18

     Um, cool. So now what's gonna happen is we're gonna ask for that again, but it's gonna take so long.

319. 52:26

     Weather in [REDACTED:age] random cities of your choosing.

320. 52:34

     So it's gonna take a while. And the reason is each of these is having to run, um, in like o-one after the other. There we go. It took like over [REDACTED:age] seconds.

321. 52:47

     Now let's do the same thing here. So the equivalent, uh, is, um, there's a, a, asyncio sleep.

322. 53:00

     Someone let me know if I'm doing this wrong, but, um, this should emulate like a very, very similar behavior.

323. 53:07

     So if we instead run it here. Clear that, and that's-- I can say, you know, give me the weather

324. 53:22

     of [REDACTED:age] random cities of your... Um, we should see it return. Okay, so it called all of them,

325. 53:32

     and it got back because they all happened in parallel. Um, this is the magic of asyncio, right? Um, anything that can be parallelized or, I guess, scheduled in a way where, where it's non-blocking, uh, like sleeps, these sleeps are non-blocking.

326. 53:49

     Um, essentially, we can, we can do. And so you can imagine switching this sleep for like an actual API call. Um, so,

327. 53:56

     uh, we can actually do that right now with o1, right? Like we could call o1 multiple times, and they would all run in parallel. Now, the main problem is that we're still gonna be waiting back, like waiting to, to get all of them together, right?

328. 54:12

     Like, like the fact that it-- we can run them in parallel means that like we can have [REDACTED:age], ten-second tasks running in parallel, so it'll take ten seconds, but it's still gonna take ten seconds where I can't talk to the model.

329. 54:24

     So instead, let's have this notion of tasks.

330. 54:28

     There's many ways to do this. Um, I'm going to do a pattern that I quite like. So let's do this. Um,

331. 54:41

     let's define a, you know, create task. So I have a create task function that makes a task.

332. 54:57

     Um, no, and I wanna create like a task ID.

333. 55:05

     And I wanna keep it short. Okay. So now what I've done is I have this function that makes a random task ID, sets it, creates it, and then calls like, let's say, get weather, something.

334. 55:26

     Um, and then it's, you know, it's suggesting this check tasks, which is the next thing that I wanna do. So the next one is check tasks. And so

335. 55:36

     maybe I can just say, like, check task,

336. 55:42

     you know, do for... Amazing. Um, okay. So let's take a moment to, to look at what just happened. We now create a task with a random ID. We give back the ID to the model, and then later we can call check task, get that task, see what the status is, and see if it's done.

337. 56:06

     Um, so let's like add, I don't know, [REDACTED:age]-second delay here.

338. 56:11

     And right now let's just get weather. So description would be maybe create task for, you know...

339. 56:22

     Let's see. Again, this is all like sort of live coding, so we'll see if it works. But I can say like, you know, create a task,

340. 56:34

     um, and set the description to the, to be

341. 56:42

     San Francisco or SF. Let's see what happens.

342. 56:51

     Oh, I didn't give it th- these functions.

343. 56:55

     There we go. Um, let me just check. Cool, cool, cool, cool. Okay, let's do it again.

344. 57:10

     Say, you know, create a task where the location-- where the description is just this.

345. 57:21

     So now it creates the task, and I can say, "Hi."

346. 57:26

     And look at that, it... That's not working correctly.

347. 57:31

     Maybe it was. Maybe it just took a while to respond. Um, here, let's add a longer delay to know for sure.

348. 57:38

     Ten seconds, let's run it back. So create a task

349. 57:51

     with description SF. Uh, cool. So I can say, "Hello there."

350. 58:00

     Cool. So it can still respond to me, right? Um, check the task.

351. 58:06

     So I can still interact with it, and it, it can check to see if it's not done. Um, I guess we need to keep checking to see if it'll work.

352. 58:14

     So, you know, how about or, you know, tell me a joke. [laughs]

353. 58:22

     Who would've thought? Who would've thought? Um, cool. So now, like, check them again.

354. 58:31

     So it called the task. It is done, and we have seventy-eight and sunny. I will take a clap for that. I wanna see everyone clap. Please clap. Thank you.

355. 58:42

     Thank you. Thank you. Thank you. So what you just saw [laughs]

356. 58:47

     was, uh, live asynchronous, uh, programming. Uh, it's very impressive. The models can also, you know, do pretty well. Anyway, here's why this is interesting. We now have a system where we can give it tasks, and it'll queue them up, uh, and then we can check on their progress.

357. 59:05

     So, so already we have the basis for like a really, really interesting thing, right? Like this, this thing here right, right now is just get weather. But if we say just like, you know, uh, run...

358. 59:20

     What is it called? Like, um, you know, call model,

359. 59:25

     and then... No, but we want AI ports, Async OpenAI, get the clients. Um, I guess we could-- Actually, we can just leverage the agents.

360. 59:41

     They're already async. Let's not deal with that again. Um, or whatever. Let's do, let's do it this way.

361. 59:48

     I did it all. I did it all. Fantastic. Okay, so now we can switch get weather for call model,

362. 59:55

     and let's try this again. So let's give it the same task as before,

363. 1:00:01

     which is, you know, "Hi." Just make sure it's running. Um, you know, write me a,

364. 1:00:09

     uh, haiku about, [laughs] I don't know, uh, a coup where the theme is so incredible it makes me cry. Make this a task.

365. 1:00:27

     Okay, so it's created the task. And now I can actually say, you know,

366. 1:00:34

     now make a second one, but pick a theme yourself.

367. 1:00:41

     So now I have this interface where I can essentially keep chatting with this, and it can essentially spin off additional tasks. Um, and you check,

368. 1:00:51

     check all tasks. Um, let's see what the progress is like.

369. 1:01:00

     Look at that. So we have, we have both. Oh, was this o1? That's pretty fast. Um-

370. 1:01:10

     But yeah, so now we have this system that you can actually just call, call multiple ones. So I'm gonna pause here, just open it up for questions for a little bit, conversation, like other directions we can take this, other ways we could have done this.

371. 1:01:20

     Um, but what I wanna look at next is like how can we do this in a way where we don't actually have to check, um, because right now, like we don't need these two terminals.

372. 1:01:29

     This is just over-complicated. Like, I think the, the, the point of having the second terminal is so that we can push things to the conversation without a user having sent a message first.

373. 1:01:39

     So I'm gonna pause here, I'm gonna field some questions, and then we can get back to it.

374. 1:01:50

     Okay. Um, would you not use generators for nested tool calls?

375. 1:01:59

     Um, Tom, do you wanna... I don't know if there's a mic in the room or like, uh, just get, get it, get it to them.

376. 1:02:07

     Yeah, there is. Give me one second. Also, they can unmute themselves.

377. 1:02:16

     Uh, who's... Yes.

378. 1:02:39

     Uh, sorry. Sohum is the one that was, uh, answering it. Is Sohum in the room?

379. 1:02:48

     Yeah. Can, can you hear him?

380. 1:02:51

     No, I can't hear him.

381. 1:03:04

     Hello. Can you hear me in the Zoom?

382. 1:03:07

     Yeah, I can hear you now. I can't hear you anymore. [laughs]

383. 1:03:21

     I don't know if you're saying a question right now.

384. 1:03:32

     No, I was just wondering if it's possible to treat agents objects as a generator

385. 1:03:44

     in Python so you can nest, like, you can yield from individual agents and then yield those tool calls.

386. 1:03:56

     Does that make sense?

387. 1:03:58

     Yeah. Yeah, that makes sense. This is also very impressive that you have to do this. I think someone needs to mute themselves. I don't know if it's Alon or, uh...

388. 1:04:07

     Like, someone in the room un-mute it and, uh, put, put this man through, through hell. Um-

389. 1:04:12

     No, it's me. It's me.

390. 1:04:14

     Oh. Is your laptop un-muted? I can't hear anyone anymore.

391. 1:04:26

     Maybe just speak into your laptop. Okay, but I'll, I'll answer your original question of like, you know, if you put, um... Can you like, essentially make agents into generators so that you can like yield the results as you go?

392. 1:04:41

     The answer is yes, and this is actually like, um,

393. 1:04:44

     the right way to do this. The way I'm doing it now is I'm only exposing the final response, mostly because, um, implementing e- like, the, the generator gets a little bit tricky and I don't wanna have to like debug that.

394. 1:04:55

     Not too much, but you can essentially just surface each of the steps, each of the function calls, each of the everything. Um, that would, um, that would let you

395. 1:05:07

     essentially keep track of more agents at the same time, and you could maybe have like a bit of a flat structure where you just have multiple agents going, yielding events, and then you can essentially see like which one's coming from where.

396. 1:05:19

     Um, but I guess the thing is like there you have to deal with a bit of the complexity of, um, handling all the like, like essentially listening to like all the different events and like, uh, associating them back to like a specific agent and like maybe there is an agent in the front, maybe there isn't.

397. 1:05:35

     Um, here the idea is just I have one agent that I'm interacting with. I'm always just gonna interact with like that one agent, and it's responsible for spinning off other, other ones a- a- and like dealing with, dealing with that as well.

398. 1:05:46

     But you can absolutely do it the way that you're saying. And like if you wanted to build this out like more fully, you probably would in order to be able to surface progress.

399. 1:05:55

     Um, did that answer your question? I'm now scared to get people's talking in the room, but I'll just read them from Slack. Um,

400. 1:06:05

     is there any, is there any good design patterns to create projects with agents?

401. 1:06:12

     There's like a million. Um, you know, like I like prototyping with Swarm. Um, I know PydanticAI is also like a really nice one. I think Sam's there in the room or around.

402. 1:06:24

     Um, there's... Yeah, I don't wanna like... I don't really use, uh, any myself, mostly because like you saw, it's pretty simple to implement your own loop, so that's what I end up doing.

403. 1:06:38

     I think usually for every, every project I either write my own like... It's like, I don't know how many, how many lines is this? Like,

404. 1:06:48

     like 70 lines. And so like depending what you want, you might want more, you might want less, and I'm sure there's good solutions. Um, but like I just ha- like I just copy this around.

405. 1:06:57

     Um, I don't really like working with too many dependencies, especially for something that's so lightweight. Like I, I want granular control. Um, and like as a for example for Swarm, I ended up having to hop in here and like handling specific kinds of tool calls in certain ways so that I can do handoffs, um, which we can

406. 1:07:13

     actually look at and we can implement this without too much trouble. Um, but yeah, my answer is like- There's many you can choose from. I don't have many I would recommend personally.

407. 1:07:24

     Um, but I, I am a, I am a fan of, of PydanticAI. It's pretty cool. I haven't used it too much, but, like, the interface looks nice. Um, it reminds me a lot of Swarm.

408. 1:07:34

     The one I use the most for prototyping is Swarm. I think it's just because it, it's the one that, like, I'm most familiar with.

409. 1:07:41

     Um, when you have to start to approach dozens or hundreds of functions, what techniques should we apply in order to effectively tool call? There's, there's a few answers there, right?

410. 1:07:52

     Um, you can have multiple agents and essentially, like, split up the responsibilities or, like, the, the groupings of the functions, um, and so into, into, yeah, like clusters where you have, like, a set of related functions that are needed for specific tasks, and then you can invoke the correct agent.

411. 1:08:16

     Uh, and this is where, like, multi-agent patterns start to make sense is, like, specifically when you have tons and tons of functions, like, how do you, uh, go to the right ones?

412. 1:08:26

     If you, for some reason, need them all at the same time, uh, you could try fine-tuning. Um, in, in projects for OpenAI, I've ended up fine-tuning up to, like, hundreds of func-- like, it was like 120 functions.

413. 1:08:38

     This was with, uh, GPT 3.5. So the fact that that works gives me, like, pretty high confidence. Like, you can fine-tune, uh, smaller models with a lot of functions and get them to work pretty well.

414. 1:08:48

     Um, the last one is, like, some kind of dynamic function loading where based on the input or based on the conversation, you load into memory or, like, you load into context, um, the most likely relevant functions.

415. 1:09:05

     Um, and there's a few different ways to do this. You can do this with embeddings. You can do this with, like, um,

416. 1:09:13

     having, like, a two-step function call. At that point, you're essentially having agents. Like, if, if you call a function to then load more functions, that's what a handoff is essentially.

417. 1:09:21

     So a lot of these start to look very similar. Um, it's just like how are you loading multiple different ones? Um, okay. For reasoning models, are there tools being called within the thought text?

418. 1:09:33

     Um, so because we don't expose the thoughts, like the, the, the chain of thought, um, that's a little bit hard to answer. For, for, for o1 right now in the API, the answer is no.

419. 1:09:47

     Um, I'm trying to see how much I can say here. Um, it is something that is technically possible, right? Like, you can do anything you want with, with post-training.

420. 1:10:01

     Um, we do not currently allow you to call functions within the chain of thought. Um, so yeah. Right now, the, the function calls happen at the very end. Uh, do you have any good code examples for router patterns in these lots of functions cases?

421. 1:10:19

     Yeah. So, so my, like, my answer there is the-- some-- one of these. Like, if you really just wanna route, um, then, like, the, the idea of having, like, multiple agents and handing off to one of them is actually really nice.

422. 1:10:31

     Um, like, you can just define multiple agents, each of them with, like, multiple functions, um, and then have the first one have like a... We can actually do this quickly.

423. 1:10:41

     Like, we-- I'll, I'll do this in Swarm, but like I said, you can implement this yourself. Um, I don't actually know who else supports handoffs the same way Swarm does.

424. 1:10:48

     But, um, essentially here, let's start a new file. Like, uh,

425. 1:10:57

     routing. So no. Let's see. Swarm repl demo loop. I can say like, you know, triage

426. 1:11:14

     equals one, and then I can have my other two. It's like, you know, maybe I have like some collection of like, um, you can call them an agent, but I can also just call them, yeah, like, uh, what would it be?

427. 1:11:25

     Like, you know, uh, email functions agent. And you can have like, you know, like, send email.

428. 1:11:40

     Check email. I don't know. What else does Cursor want me to write?

429. 1:11:45

     Not get weather. Um, cool. We have th-these two, and then we can do like maybe we have the emails, and then we have maybe a calendar. I don't know.

430. 1:11:54

     So make, like, create event. Can do like calendar. Yeah. I'm just gonna say like, you know, finish what I'm doing.

431. 1:12:10

     Update the prompts and tools, because I'm lazy. So let's see what, uh, let's see what Cursor thinks. Great.

432. 1:12:20

     So now we have an email functions agent and a calendar agent. Call them whatever you want. Um,

433. 1:12:28

     now, like, let's pretend that instead of just having three functions, we have 30, and each one of these has 10 or 15. Um, then, like, if you maybe give all 30 to one agent, first of all, try it.

434. 1:12:42

     Like, if that works, amazing, right? You don't have to deal with the extra complexity. Um, so you don't really wanna do handoffs and stuff until you really, really need to through, through like evals.

435. 1:12:52

     Um, but, um, yeah. So, so and then the special, like, handler functions here is like, you know, uh, transfer to

436. 1:13:05

     email agent, and I'll return email agent, and then cool. So now I have my two transfer functions.

437. 1:13:22

     Oh. Oh, it did it. Okay, cool. So I think this should just work, right? So n- I've defined, like, the actual functions and agents that in your case would be many, many more functions.

438. 1:13:34

     Um, I've defined the transfer functions, and I've given them to the triage agent. So now if I run this.

439. 1:13:45

     Say, "Hi." Um, you know, "I want to send an email."

440. 1:13:56

     Cool. So now I'm talking to the email agent. Now, if you wanna do, like, more, you know, ch- like, transfers.

441. 1:14:05

     Uh, oop, this is not delegation. Okay, so trail assistant. Um, maybe we can tell the assistant, like,

442. 1:14:15

     you know, if you already know what the user is asking,

443. 1:14:22

     just call that function, right? And this is if you wanna have, like, a, a case where, you know, it still routes you, but it's like a faster two-hop. So maybe I can say, like, what are the functions here?

444. 1:14:36

     Um, what are the parameters? Send email to subject body.

445. 1:14:43

     So I can say, like, you know, "Send an email to [REDACTED:email_address] about taxes."

446. 1:14:55

     What was the other one? Body. Um, saying, "Yo, do your taxes."

447. 1:15:04

     So it should transfer me to the email one, and then there we go. It immediately sends the email. So this felt like an immediate function call, but there was a transfer in the way.

448. 1:15:14

     So this is kind of an example of, like, triaging does work. It's really convenient to model it with agents and handoffs. Um, I'd say the primary use case for agents and handoffs is just a glorified triage through, uh, multiple functions.

449. 1:15:28

     Um, yeah. Let's see. Let's go back to questions.

450. 1:15:37

     Uh-huh. Consider having something like Streamlit Gradio until you-- Oh. Are you considering having something like Streamlit Gradio util that allows porting all interaction functionality with one command to app?

451. 1:15:52

     Uh, I don't think so. I don't know. Uh, what did people answer? There we go. [laughs] Sam's in with, uh, PydanticAI, so check that out. Um, how many tool calls can you get in one iteration?

452. 1:16:06

     You-- Like, parallel function calls, I don't think we have hard limits on either the number of functions or the number of parallel function calls. How big a tool library will the models perform well with?

453. 1:16:18

     Super, super general rule of thumb is, like, ten to twenty you shouldn't really pass over. Um, but like I said, I've gotten-- Like, this was a very specific case where, like, it was extremely latency sensitive, and so, like, we had to have flat, like, flat function calling, and we did a hundred and twenty functions with GPT-3.5.

454. 1:16:39

     So you can go pretty far, um, with fine-tuning, but, like, I'd say reliably without, without very extensive prompting, yeah, probably, like, ten to twenty. Like, pa-past that point, you really ask yourself, like, what are you trying to do?

455. 1:16:54

     Right? Like, why are you putting so many functions? Is it super latency sensitive? Can you split it up? Like, yeah. Um, was there a follow-up here?

456. 1:17:05

     Okay. Um, rather than tool calls, I feel like we're moving to generated code agents. Will I soon be able to supply my tools functions? Ah, okay, that's a good idea.

457. 1:17:14

     We should try that. Um, so essentially have something write its own function and then use it. Uh, yeah. I feel like it-- we can probably... How would we do that?

458. 1:17:24

     Yeah, we can try that. We can, we can find a way to do that. Uh, I'll, I'll do that. Actually, let's do it right now. Why not? Let's do it right now.

459. 1:17:30

     Uh, I've never done this before, but I feel like it shouldn't be too hard. So, um,

460. 1:17:37

     okay. Uh, you know, what do we call this? Like, bootstraps.

461. 1:17:43

     Okay, so I'll keep using Swarm because I, I will use, um, handoffs.

462. 1:17:50

     From Swarm import agent. Cool. So now-- [laughs] It's always bother. So let's see. We want an agent that writes its own functions. So agent

463. 1:18:02

     was this one. Um, we want it to write its own functions.

464. 1:18:12

     So maybe we can have it ha-handoff to itself. [laughs]

465. 1:18:16

     So we can do, you know, like, refresh, you know, autoBE refresh functions.

466. 1:18:25

     So we can actually return the same agent. Um,

467. 1:18:31

     and then... Okay. Is it not declared? Is it unhappy?

468. 1:18:37

     Uh, yeah, it's not defined yet. Okay, so now we can define it down here.

469. 1:18:46

     Cool. Functions. Okay. Um, bootstrap. Bootstraps, bootstraps. Okay, so we have this, and we want it to write its own functions. So how are we gonna do this?

470. 1:19:02

     We want it to produce Python. Does anyone have any ideas? Uh, if you wanna shout them out, like, make-- just pass around the mic while I code. Just feel free to...

471. 1:19:12

     I can't, I can't read while you're, uh, what, what you write, but we can, we can try this. So, um, let's say we want it to,

472. 1:19:19

     you know, um- You know, add tool. There we go. And then let's call this, you know, Python implementation.

473. 1:19:39

     Okay, I'm gonna do something very unsafe. So, you know, what would it be? It's like function obj equals

474. 1:19:51

     eval of Python. Don't do this. Don't do this, kids. Um,

475. 1:20:01

     and then will this work? Is this how Python works? Sam's there. Uh, okay, so if I have this

476. 1:20:10

     and I eval, like, you know, def a...

477. 1:20:27

     Does that do it? Um, okay. You know what?

478. 1:20:41

     Um, so I guess it's like, uh, I want to write a,

479. 1:20:47

     write a function that takes a string presenting an implementation of a Python function and returns the actual Python function as interpreted.

480. 1:21:06

     I don't know. Let's see what it does.

481. 1:21:10

     Um, so essentially we wanna do that. Once we have the function, then we can just append it to the agent, and then we might need to reload it. Uh, so we can then just return the agent, and this might be it. [laughs]

482. 1:21:26

     This might be it. So add tool. Exec, not eval.

483. 1:21:35

     Why didn't anyone say that? You guys can shout out.

484. 1:21:39

     Okay. Okay, so it's exec. And then what is this, a?

485. 1:21:57

     I don't know. Someone save me. Like, ah.

486. 1:22:08

     Uh, hey, there's a couple implementations in the Slack.

487. 1:22:11

     Sorry?

488. 1:22:12

     Check the Slack.

489. 1:22:18

     Did someone do this? Hey. Okay. [laughs]

490. 1:22:23

     It's almost like someone's

491. 1:22:25

     No, no, no, but this is not-- Okay, no, no, but this is not what I want, right? Like, I don't want it to run it in a subprocess. I want it to, I want it to evalu- like evaluate the function-

492. 1:22:35

     Uh-huh

493. 1:22:36

     ... and then turn it into a ... Is that-- Oh, go up. Oh, Sam, of course. Amazing.

494. 1:22:43

     Uh, [laughs] Jesus Christ. [laughs] Okay. Um, make it so the add tool function, uh,

495. 1:23:04

     evaluates the implementation and adds it to the tools, similar to now.

496. 1:23:15

     Here's a great reference for how to. That's-- And then we wanna grab-- Was this a good idea? This was a terrible idea. I hope people are enjoying this.

497. 1:23:30

     Um ... Okay, let's see. Um, and let me just read this, 'cause it might not be that complicated to add in. So

498. 1:23:49

     I can't-- Okay, it got in the way. No, not Zoom. Where's Slack? [sighs]

499. 1:24:17

     Was this it? Is this it? [laughs] Okay, I'm gonna, I'm gonna put this on hold for now. It sort of ignored your implementation, Sam. I'm sorry.

500. 1:24:33

     Um, let's see. So if you have parse function, can I just

501. 1:24:40

     do this? Does this work? Uh, okay, let's see. I've-- There's a first time for everything. Uh, wrong one. Sub

502. 1:24:53

     bootstraps. Hi. Um, add a tool that prints hello when called.

503. 1:25:11

     Call it. Oh, but no, no, no, no, no. Make it print it, not just return it.

504. 1:25:27

     My heart's pounding. Okay. Um, so say hello tool, call it.

505. 1:25:35

     Look at that! Look at that. Okay, so now we have-- Yes, I will take a clap for that. We did this together, guys. Um, so this is actually a lot less code than I was expecting, but now we have a system, look at that, it's tiny, um, that can write its own tools.

506. 1:25:51

     Um, and so, like, maybe we can do something like, you know, um, what is, you know, like, make yourself a little calculator.

507. 1:26:05

     I can't believe this just worked. Uh, you know, what is

508. 1:26:10

     two times three times... Can someone check this? Uh, wait, wait, wait. [laughs]

509. 1:26:23

     Three, four, six, nine, eights. Three, four, six, nine, eights. This is, this is crazy. This is, this is so fun. Uh, yeah, I hadn't done this before, and this is a lot less code than I thought.

510. 1:26:34

     Look at this. Look, this is all you need. I mean, it's... This is super dangerous code. Like, this, [laughs]

511. 1:26:41

     this is not good. Don't do this. But it's fun. Uh, I would put this squarely in the fun things. We can, we can now transition to looking at other fun things.

512. 1:26:51

     Um, they're definitely not as fun as this one, I think. Um, they're just a couple of, like, random things that I've found related to real-time, since I did sort of put it in this, uh, title.

513. 1:27:02

     So whatever. Okay. These are the two main tricks that I, that I kind of thought were pretty cool, and you might have already seen them. Um, one is if you've ever dealt with, like, the Realtime API,

514. 1:27:15

     um, like jumping in, uh, before you're done with an idea or, like, done talking or something, um, like, what I was thinking is, like, if, if really what you want is you want it to, like, use the model's own intelligence to decide if you're done talking or not, um, and you can treat our VAD, like our voice

515. 1:27:35

     detection, as like a trigger-happy version of that. It's like a, it'll always tell you when maybe the user is ready to stop talking. But if you want the model to check, you can have a stay silent function or something else that essentially handles the other side of that, where it's like, okay, we're definitely gonna let you know

516. 1:27:53

     whenever the user might be done talking, but then you can actually verify with a function call. And so this implementation is, like, super, super simple. You literally just give-- you give the, the function itself

517. 1:28:05

     to Realtime API and tell it to call it when the user is not quite done talking. And I'm not gonna try to demo this live, like realtime demos are tough, audio is tough, all that fun s- fun stuff's tough.

518. 1:28:18

     But, um, it works, like, surprisingly well. Uh, if you just describe, uh, uh, I can... You can probably find this tweet. It has the full prompt. But essentially it's like, you can, you can like pause, you can say like, "You know, I've been thinking about," and then like stop, and like it'll get triggered, call stay silent, and

519. 1:28:36

     then you can keep talking. Um, so pretty, pretty cool, uh, useful thing. Um, and then the other one is also related to Realtime API. And if, again, if you, if you, uh, are following me, you might have already seen this.

520. 1:28:49

     But, um, someone at Dev Day just, like, came up to me and asked me like, "Hey, um, is there a way to like make the Realtime API talk in like a specific way?"

521. 1:29:00

     Uh, and I was like, [laughs] "I don't know. Let's try it." And so like we had a demo booth, and I sort of pushed someone out of the demo booth and grabbed the laptop, and I was like, "Let's just like try, try this stuff."

522. 1:29:10

     And apparently, if you ask the model to just-- you give it like a, a script, and you're just like, "Hey, read this out according to these XML tags," um,

523. 1:29:19

     you can have it follow them. Here, let me, let me find, um,

524. 1:29:24

     the original, the original one. Yada, yada, yada.

525. 1:29:30

     Sorry, this is-- I'm not trying to show you the whole timeline. Where is it? Okay, this is the stay silent one. Um...

526. 1:29:47

     Where is it? Where is it? There we go.

527. 1:29:52

     Um, uh, it's impossible you'll hear this, right?

528. 1:29:58

     You guys don't hear this. Yes? No?

529. 1:30:03

     Uh, we don't hear it.

530. 1:30:04

     Okay, that's fine.

531. 1:30:05

     But you can share your computer audio.

532. 1:30:08

     Can I?

533. 1:30:10

     It's doing things.

534. 1:30:10

     I'm, I'm, I'm actually-- I'm not gonna try. It's fine. Um,

535. 1:30:15

     wait, can I? Seems... No. No, it's fine. Whatever. Anyway, um, yeah, so if you, if you, if you like give it like

536. 1:30:25

     a script like this, um, we didn't actually train it for this, right? Like this is just a really nice consequence of like behavior. It's technically not function calling, I'm realizing now, but it's like, it's very function calling-esque.

537. 1:30:37

     It's real time. The title of this talk does include real time, so this is, uh, this is it. Um, I will pause here. I have so much more I can go through, but I wanna give you guys a chance to like ask questions, poke around with some of these ideas.

538. 1:30:49

     Uh, Swarm is public, you can just try it out. Um, we can also just try creating other functions. Uh, this, this is one of the-- this is easily one of the coolest, like, little programs I've ever made.

539. 1:30:59

     Um, so yeah. Uh, let, let me see if there's any other questions in the Slack.

540. 1:31:14

     Uh, da, da, da, da, da. It's a room full of people autocompleting. [laughs]

541. 1:31:20

     Make... Yeah, yeah, yeah. Okay, let's not do that.

542. 1:31:25

     Uh, okay, revisiting memory. Let's see. Do you have any suggestions for trying to enforce consistency of stored memories?

543. 1:31:33

     Identifying inconsistencies and figure out how to resolve them. Okay. And then the second part is, what data structures do you suggest for more structured memory, helping enable more meaningful comparison of objects?

544. 1:31:43

     I think-- I mean, this question opens, like, you start to go down a path where you can get as complex as you would like, right? Like, like you can start simple, and you can end up with, like, an entire operating system to manage memory, right?

545. 1:31:57

     There's like a whole range. Um, one way to do... O-off the top of my head, like, one way I can think to do this is, um, when you are about to store memory, do a retrieval, like do search, to find similar memories or like memories that like are semantically similar.

546. 1:32:18

     Um, and then do an explicit check, uh, with a model to see if like, you know, th- one is like updating or contradicting another, right? An example of this is like, you know, what is the latest state of a project, right?

547. 1:32:31

     You know, if someone's like, "Is the project..." Like, at some point you say like, "I'm working on the project. Like, it's not ready yet." And it saves that, and then later you're like, you know, "I, um, ha- like the project's done now."

548. 1:32:44

     Right? Like, those are two contradicting memories. Um,

549. 1:32:49

     what you can do is essentially, uh, have a timestamp for them, but also when you are about to store the second one or any memory, check for similar ones and create like a direct, like, essentially,

550. 1:33:06

     um, like node, uh, pointing from the original memory to the new one. And so I guess, I guess what I'm thinking is like that way if anybody asks

551. 1:33:23

     y- and you surface both, like you can keep both in memory. And when you do like retrieval and semantic similarity, you can surface both. But essentially, you can present like the whole chain of updates.

552. 1:33:34

     And so you can just present the last one if you want, or you can present the whole chain and the model have so- has some idea of like, you know, maybe if you ask it like, um, you know, how long is this project delayed?

553. 1:33:45

     Because that was the last you heard about it. But somebody at some point said, "Oh, it's actually done." Um,

554. 1:33:51

     like if it raises both and has one with a later date, you know, maybe. But if you like, if you've already made this chain explicitly, um, like you can actually represent that, and you can choose to not show the previous ones.

555. 1:34:03

     So that's one idea. Like I said, there's so many. You can like-- there's so many ways to do this. Uh, and, and like, as soon as I stop talking, you guys walk out, someone's gonna like be like, "Oh, there's actually this, like much easier way that like this idiot didn't think of."

556. 1:34:15

     So anything you think of probably works. Um,

557. 1:34:21

     here's a real-time API prompt. Yeah. Cool. Uh, any other, any other questions about anything? Um, it's been like an hour and forty minutes, so happy to, like, keep going with random content.

558. 1:34:37

     Um, there's a couple demos that I have that are like demoing some public repos that we have that you can use. Um,

559. 1:34:47

     yeah, but I wanna g- I, I wanna give anyone a chance to like speak up, say something if you want, ask questions.

560. 1:34:54

     Um, I, I don't see, I don't see a, a lot happening in... Like I can't, I c- you guys are tiny on my screen.

561. 1:35:09

     Uh, cool. Okay. Then I'll pose this to you. Sorry, what?

562. 1:35:15

     Oh, no, go ahead. It sounds like you're wrapping up. [laughs]

563. 1:35:21

     Yeah. The, the last thing we can do, I guess, is like, either I can, um, try and pull up a real-time, uh, demo that shows how to do the o1 stuff, uh, in a, in a slightly easier way.

564. 1:35:35

     Okay, cool. People wanna do that demo. [laughs] Okay, let's do it. So, um, the reason I'm a little iffy about it is because, yeah, it's real-time, uh, and those demos are tough to do publicly, but, um, there is this repo, openai, um, realtime-twiglio-demo,

565. 1:35:58

     um, that essentially sets up your whole, um, like phone calling assistant. Uh, and it's just works right out of the box. It walks you through the steps. Um, I may end up sharing a, a phone publicly.

566. 1:36:12

     Don't call it during this demo and don't share it, and I will delete it. But, um, yeah, I trust you guys. So let's see.

567. 1:36:25

     Okay, so looks like this is already running.

568. 1:36:31

     Oh, I do have it already running. Okay.

569. 1:36:34

     Um, oh, never mind. I, I did, uh, I did make it all dots. But yeah, essentially this, um, this little checklist is like live updates. So as you set up the account, as you set up the phone number, as you set up your local web server, et cetera, it'll like get checked.

570. 1:36:50

     So setting up Twilio and Groq and everything has been one of the most annoying things I've had to do multiple times, um, related to the real-time API. So this hopefully makes that process like a lot, a lot easier.

571. 1:37:00

     You can do most of it directly from here, and it lets you know like when things are done. So you don't actually have to do a lot from the Twilio console at all.

572. 1:37:07

     Um, anyway, so, um, I-- let me show you the back end. So there is a

573. 1:37:18

     handler function. There we go. Function handlers. So, um, you can either implement tools, uh, locally as a JSON schema, and we can, we can try that really quick. So this is the real-time playground, and I can say like, you know, make a-

574. 1:37:36

     Function to, you know, get the answer to answers to everything.

575. 1:37:45

     No params. [laughs] Maybe one param. Okay, so it gave, it gave us this nice little, uh, nice little function. Now we can just paste it in here.

576. 1:38:01

     Save it. And cool. So now we have this get universe answers. And I can save this configuration and call the phone number.

577. 1:38:15

     So let's see. Hello? Hello? Okay, this is why I love them.

578. 1:38:31

     Let me, let me try once more. Okay, let's go. Tool. Paste this. Save changes. Save config. Um.

579. 1:38:53

     Hello? Hello? I don't think it's happy. Is someone else-- No, I didn't show the number. Hello? Okay.

580. 1:39:08

     It's not working. That's fine. Sort of expected it. Um, essentially what you can do, like, when you, when you use this, um, it's cool because these tools you can either specify local ones with schema and they appear here, or you can define them in the back end.

581. 1:39:24

     Um, so if you define them in the back end, you can actually give, like, code to execute. Um, if not, they'll just show up here, uh, and you can enter in, like, what the response will be, uh, like a mock response.

582. 1:39:34

     Um, but you can also, like, set up back-end functions that will actually get handled in the back end. Uh, the sample one is, like, a real implemented get weather one.

583. 1:39:43

     But as you can see, there's like-- it's pretty straightforward to implement, like, the o1 one. Now, the main difference between the Realtime API here and, like, what we were doing before is the Realtime API does actually allow asynchronous functions natively.

584. 1:39:59

     So you can-- the model can call a function, um, get no response, and you can keep talking with the model, um, until a response is back. So that, that is behavior that, like, we specifically had to teach the real-time models, because unlike a chat conversation, you can't really enforce, um, like, you can't really halt the whole conversation

585. 1:40:21

     until the function response comes back. So we had to do that for, for, for launch. Um, because originally when we hadn't done that, you know, it-- we hadn't ever shown it how to do asynchronous functions, so it just couldn't, couldn't handle them correctly.

586. 1:40:35

     Anyway, sadly, uh, demo didn't quite work. Maybe I can try this one more time. We'll see.

587. 1:40:40

     Third time's a charm. Hello? Hello?

588. 1:40:59

     No. Okay, great. Anyway, um, I believe that is most of what I wanted to cover today.

589. 1:41:11

     Um, I'm gonna play with this hyper unsafe function-making agent a bit. But, um, yeah, thank you all for coming. I'll hang around for questions for a bit if anyone has any.

590. 1:41:23

     Otherwise, I think we're ending a little bit early. Um, like I said, yeah, mo-most of you can, can hang out or, or leave as you please. Um, but I'll-- maybe I'll call it wrapped up here.

591. 1:41:36

     Um, maybe what we can do is if you're interested, you can stay and, like, hack on some of the stuff that you saw. I'm happy to hang around, uh, and answer questions as well.

592. 1:41:46

     So thank you all for coming. Uh, hope you, hope you got something out of this.

593. 1:41:58

     Um, Anup is also here, uh, to help you folks.

594. 1:42:03

     Uh, otherwise, Ilan, I guess you can share your screen a little bit in case people have questions.

595. 1:42:09

     Mm-hmm. I'm actually gonna dump this, but this may be a really bad idea. But, um,

596. 1:42:15

     here you go. Here's the code for the super unsafe self, like, tool writing agent. [laughs]

597. 1:42:34

     And I'm probably gonna stop sharing my screen.

598. 1:42:38

     Um, yeah, I can, I can, I can share the repo and the slides. There's not a lot on the slides, but yeah, I'm happy to, happy to share them after.