How to add secure code interpreting in your AI app

Talk transcript

1. 0:00

   [upbeat music] Hi, everyone.

2. 0:16

   Uh, thanks for coming, thanks for coming to my, uh, to my workshop. Uh, my name is Vasek and, uh, today, uh, I will show you, and you can code along, um, how to add AI code interpreting into your AI app, uh, or into your app.

3. 0:34

   So, uh, what, what we will be building, uh, will be very similar to, uh, Claude's new AI Artifacts, uh, app that they j- or Anthropic's AI Arti- uh, Artifacts app that they just released about two days ago.

4. 0:53

   Uh, so if you are not familiar with it, the way it looks like is that, uh, you have a chat on the left, and on the right you have this preview.

5. 1:04

   And what this preview does is that, uh, Claude can write code. Um, in, in Claude's, uh, case, it's supported just for HTML, JS, and CSS code, if I'm not mistaken.

6. 1:18

   And then it actually renders and run the code, uh, right next to the, to the chat.

7. 1:25

   Uh, we'll build something, something very similar, uh, with a few caveats. And first is that, uh, it, uh, we will be able to run AI-generated Python code, uh, not just HTML, CSS, J- JavaScript.

8. 1:41

   I will also show you at, in the end how to run, um, other, uh, other runtimes, uh, and other languages. And, uh, we will be, uh, using E2E, which is a company that I co-founded, and, uh, for secure code interpreting.

9. 2:01

   Uh, and, uh, you can code along, uh, while, uh, we are building it. So here's a, here's a, like, quick demo, uh, what we are, what we are going to build.

10. 2:12

    On the left, we'll be chatting with Sonnet 3.5, and under, uh, every time it generates code, it's actually implemented through t- function calling. Uh, we will run that code in a secure sandbox that can run pretty much anything that a, a Linux machine, uh, or Ubuntu machine can run.

11. 2:29

    And, uh, uh, you can spawn many, many of these sandboxes for secure code interpreting.

12. 2:37

    Uh, it's, uh, the workshop is made for, uh, code along, but of course you can just watch. Uh, but, uh, if you are interested in actually following me, uh, you can, uh, go to, uh, our repository and, uh, clone the cookbook, and, uh, then check out through workshop start.

13. 3:01

    And, uh, you also will need, uh, two API keys. Uh, one of them is E2B, uh, API key, uh, which... Oh, sorry, which you can get on our documentation.

14. 3:14

    So E2B, E2B, e2b.dev/docs. Then you go to, uh, API key. You sign up, and we give you your API key. And also you will need an, uh, you will need Anthropic's API key.

15. 3:29

    In case you don't have that or you don't want to get it, I, uh, created one for this workshop. And so if you go to... And you are free to-- Well, I kind of trust you that you will not misuse it.

16. 3:45

    Um, if you go to dab.sh/e2b-workshop, you can get the API key for free. I will delete it after the workshop.

17. 3:55

    Um, so I know everyone had a lunch, so you're probably kind of tired. I will try to keep it, uh, sort of short. Uh, so it most likely will not take two hours.

18. 4:05

    Uh, we will get the result, uh, much faster.

19. 4:10

    Uh, so, uh, I will, I will give you like a few quick minute, a few, a few seconds, uh, or tens of seconds to set up everything. Um,

20. 4:22

    but just to get a better idea, how many of you want to follow along?

21. 4:28

    Okay, cool.

22. 4:29

    It's taking quite a while to download the repo.

23. 4:34

    Sorry?

24. 4:35

    It's, yeah, the repo seems quite big. It's like giving me like one hundred and forty-two percent.

25. 4:40

    Oh, shoot. Um, oh, that's probably because, like, we have, uh, images there that you can generate with the... Oh, sorry, I didn't realize that.

26. 4:48

    That's cool. I'll just shut it down like this.

27. 4:51

    Yeah. Uh, that's a good point. By the way, if anyone has any questions, just, like, feel free to ask during the workshop. If anything isn't clear, I will repeat the question and, uh, and try to give you a answer.

28. 5:19

    Okay. Yes.

29. 5:21

    Is the code interpreter running on our machines or not?

30. 5:24

    Mm-hmm. The question is if the, uh, is code interpreter running on your machine or somewhere else. The code interpreter isn't running on your machine. Uh, it's, uh, using, uh,

31. 5:37

    a tool called E2B, uh, which I'm a co-founder of, and, uh, we're basically building a runtime for open source runtime for AI agents that, uh, you can, and well, very soon will be to, uh, self-host on AWS, GCP, Azure, uh, and other big cloud providers.

32. 5:57

    Um, I can go a little bit more into detail, uh, what's the whole Problem and challenge with code interpreting around security. Um, I was thinking that maybe first we implement the app so you can kinda get a, a idea of how it works, and then, um, during Q&A, uh, or after that I can, uh, get into the

33. 6:19

    security and what are the challenges if you want to build this in-house.

34. 6:24

    Cool. So how, how's the, how's the repo cloning, uh, look like?

35. 6:31

    It's slow.

36. 6:33

    Slow?

37. 6:33

    It's resetting my 22.

38. 6:33

    Oh, shoot. Um, okay.

39. 6:40

    I had to, uh, go to the GitHub and download the examples

40. 6:40

    Sorry.

41. 6:41

    Could you get the clone, but from the GitHub open download instead?

42. 6:45

    Oh, yeah. Uh, that's a good point. Uh, you know what? Let me ... I probably can create, uh, very quickly just the repo without other examples. Um

43. 6:58

    ... And I will just put, uh, the example that we are interested, uh, in there.

44. 7:11

    Which should be much smaller.

45. 7:15

    Excuse me. Is there a link to your slides? Sorry to ask that.

46. 7:18

    I don't have slides. Uh-

47. 7:19

    Is there a link to what you're showing us?

48. 7:21

    Yeah, so, uh, if you go, uh, if you go to github/[REDACTED:url]/e2b-cookbook.

49. 7:33

    Uh, you go to examples.

50. 7:34

    Oh, okay.

51. 7:37

    And there's this Anthropic, uh, it's called Power Artifacts, and we will be building like a open source version of Anthropic's, uh, new Artifacts UI that has Sora code interpreting, uh, in it.

52. 7:51

    Uh, okay.

53. 8:31

    Okay. Uh, let me just quickly set up the new repository. Uh ...

54. 8:55

    Okay, this one should be much smaller. If you go to, uh, this URL, uh, I will show it, uh, on a big screen. Uh

55. 9:06

    ... If you go to this URL, which is my personal GitHub, um, and go to s- uh, /workshop-repo, uh, you can ... You should be able to clone it,

56. 9:21

    oh, much faster. Actually, sorry.

57. 9:33

    That link is in the Slack now in the general.

58. 9:36

    Thank you. Perfect. Um ... Okay, can you ... Uh, you should be able to clone it now. I didn't have, I have only README there. Now all the files are there.

59. 9:52

    And, oh, I don't have branches there, I just realized.

60. 9:58

    Actually this, this should be good. It should be good. Um,

61. 10:05

    yeah, yeah. Good. Perfect. So, uh, sorry for the confusion. Uh, let me, uh, do it again properly.

62. 10:18

    Um ... Okay. So, uh, if you want to clone, if you want to, uh, code along, please, uh, go to this, uh, GitHub repository and clone it.

63. 10:42

    This should be much faster. Um, is it m- is it faster?

64. 10:45

    Yeah. Yeah.

65. 10:46

    Perfect. And do you already have it cloned?

66. 10:49

    Yes.

67. 10:49

    Perfect. Okay, so now we can start. Mm.

68. 10:56

    Uh, when, uh, after you clone it, uh, just install dependencies, and, uh, you will also need, uh, those API, uh s- yeah, API keys. So if you create a file .env.local, uh, which will need E2B API key and Anthropic API key.

69. 11:19

    You can get E2B API key on, uh, this website. So if you go to e2b.dev, click Get Started,

70. 11:29

    and go to API key. Uh, if you quickly sign up, uh, you can get your API key. If you, uh, just want to follow along, uh,

71. 11:44

    yeah, if, if, if you, if you just want to look, you don't need an API key. And you will also ha- you will also need Anthropic's API key, and in case you don't want to spend your money on that, I created, uh, I created, uh, a Anthropic API key just for this workshop.

72. 12:02

    So if you do go to this link, it should open a 1Password, uh, re- Share, share model or share, share website. And you can get your, uh, Anthropic A- API key there, and I will cancel it after, after the workshop.

73. 12:19

    All good?

74. 12:20

    Yeah.

75. 12:20

    Perfect. So, uh, once you have your API key set up, uh, install dependencies, and then just run, uh, NPM run dev. All right.

76. 12:32

    And you should see what I'm seeing here. Uh, the app should start on port 3000. So what we are doing just now is we started an, an Next.js app, and which is like a starting point, and we will now, uh, follow, uh, you can now follow and code along, and I will show you how to add the

77. 12:49

    code interpreting there. So if you open the, the app,

78. 12:55

    um, it... Oops. It just shows an input, uh, which is like this is a scaffold of the app where you can ask Claude something. So let's say we want to, uh, generate random chart.

79. 13:11

    And actually, now it won't work because we need to implement everything. So, uh, I just hit an Enter, but, uh, I'm g- I should be getting, uh, or, or you should see in your terminal four oh five.

80. 13:28

    So, um, now we, that we have everything set up, um, in, in case I'm too fast, just, like, scream at me, uh, and I will, I will slow down or just wait for you.

81. 13:39

    Um, uh, now we have everything set up. Uh, I want to just quickly show around the code, around the repository, and what we will be doing. So there are two, uh, important parts of this project.

82. 13:50

    First is, uh, what we are going to use is, uh, Vercel's AI SDK. And this, uh, in the JavaScript world is essentially, like, easy and nice way how you can connect to different models and stream them to your, uh, to your front end from, uh, your, uh, back end.

83. 14:11

    Uh, so, uh, it's already installed, uh, once, uh, when you, when you ran NPM install. In case you want to check out the docs, uh, after, after the workshop or during, uh, go through this link.

84. 14:24

    And the second thing that we will be using is, uh, the Code Interpreter SDK. So that's, uh, that's something that we built at E2B. And essentially what it does, uh, it's open source, and what it does is that, uh, you can create your own custom code interpreter, and, uh, that you can predefine the whole environment.

85. 14:48

    That code interpreter runs in a sandbox, which is like small VM, um, when you, when you are using our cloud. It runs on our cloud. Soon you will be able to self-deploy it.

86. 15:00

    And, uh, in specifically for this SDK, inside the sand-sandbox is a Jupyter server, uh, running, uh, w- to which you can just send Python, JavaScript, uh, R code, uh, and Java code, uh, we have in beta.

87. 15:20

    And you get back standard output, error output, but we also get back charts. Um, so the reason we are using a Jupyter, uh, server there is because, uh, that's what we noticed has been, um, one of the most frequent use case, uh, from our users and our c- uh, customers.

88. 15:41

    And, uh, generally, uh, at least, uh, during the time of GPT-4 and for, uh, Turbo, uh, the code that you usually got from the LLM was, uh, was for Python, like data science, Jupyter Notebook.

89. 15:59

    And this is really great for visualization, which is, like, big, uh, use case for what we are seeing with, with, uh, with users.

90. 16:07

    Um, just a, just a quick note. Uh, this whole Code Interpreter SDK is built on, like, a general sandbox. This is small VM. Uh, we start those VMs pretty quickly, around few hundred milliseconds.

91. 16:19

    Every time you create an instance of code interpreter, we actually create that sandbox. Um, so this is kinda like a wrapper around this, our sandbox SDK. That's also open source, uh, which is, uh, which is here.

92. 16:34

    Uh, and so if you want to build something pretty, pretty custom, uh, you can do that. So code interpreter is like a one specific use case of how you can use the sandbox.

93. 16:42

    You can look at the sandbox as a general Ubuntu machine that's r- made, uh, with security in mind for running, like, AI-generated code.

94. 16:52

    Um, so, uh, if, uh, as you noticed when we were, uh, running this app, when I wrote something, uh, I just hit the Enter and nothing happened. So first we need to, uh, we need to implement an endpoint to, uh, call the, uh, to call the VM.

95. 17:11

    Uh, sorry, to call the LLM. Uh, and for that we are using a Vercel's, uh, serverless function, and there is a predefined file called, uh, route.ts. And here, uh, is, uh, commented out a function that will handle our POST requests, uh, that we need to implement.

96. 17:32

    And here we will be using the Vercel's AI SDK.

97. 17:37

    So, uh, now we finally start coding, and now we will do just like a simple, uh, simple, uh, implementation of messaging with, with Claude. First, uh,

98. 17:52

    we need to parse, uh, those messages coming, uh, from the request.

99. 18:00

    And, uh, then, uh, we actually use the AI SDK, uh, Vercel's AI SDK. And from that we will be using the stream text method.

100. 18:17

     And so the model we are using is Anthropic,

101. 18:21

     uh, which is the Sonnet one. And, uh, we want to pass the messages. Um, oh yeah, there's an await I was missing. And

102. 18:34

     we also, uh, want to include a system prompt. And so w- I already predefined a prompt in a file. Uh, it should be in a lib directory. There is a prompt file.

103. 18:47

     And so the prompt is really s- simple, it's a simple prompt, uh, uh, that tells the, the s- tells the LLM Sonnet that, uh, it's a skilled Python developer that can do a bunch of, uh, visualizations.

104. 19:02

     Uh, so let's import that variable prompt. Okay. And

105. 19:20

     add it here. And now, uh, that should be pretty much it for the, for the basic implementation. Now we just need to return stream from our, uh, post,

106. 19:34

     uh, endpoint. So result.doaiStreamResponse. All these are methods on the Vercel AI SDK. Uh, so, uh,

107. 19:46

     if you are more curious, you can just, uh, jump into it there and into the recommendation. But, uh, if we go back to our application. Sorry. Uh, now we should see that if I type hello, first our message is there, and we get a response from, uh, from Claude.

108. 20:07

     Am I too fast or are you, uh, following?

109. 20:10

     Yeah.

110. 20:10

     Too fast?

111. 20:11

     How did you define messages?

112. 20:13

     Sorry?

113. 20:13

     How did you define messages?

114. 20:13

     Yeah. So, uh, the messages, um... So this is just the back, ba- backend part. Oh, yeah.

115. 20:22

     And it's coming in the, in the request.

116. 20:28

     What's the prompt that, the prompt-

117. 20:30

     Mm-hmm. The prompt is... Sorry, I will show it here. Uh,

118. 20:36

     in a, in a file.prompt dot, uh, in a file called prompt.ts. And it basically just says that, uh, it's a skilled Python developer that's capable of, uh, running like a v- visualization on data science code.

119. 20:54

     It would work for, uh, different use cases as well. Now, I just wanted to show something where we, at the end, get a chart, uh, that you can render on the front end.

120. 21:08

     Oh, one, one more thing, uh, that I wanted to mention, uh, is that in case you are not follow, uh, you, you, you are, like, lost of what just happened.

121. 21:19

     In the original repository, uh, in the examples, uh, repo, uh, there are branches called Workshop 1, Workshop 2, Workshop 3, Workshop 5, uh, 4 and 5, and those are like the stages that we are implementing here.

122. 21:33

     So in case you are wondering what is happening, it should be pretty much the same. And if you just want to get to the final, uh, point, you can check out the worksh- workshop-final, and you should get, uh, everything there.

123. 21:57

     On the time series data set, is it-

124. 21:59

     Sorry?

125. 22:01

     The prompt of the, the series create a new file.

126. 22:04

     Uh, that file should already be there. Uh, it's in lib directory. Lib-prompt. Yeah.

127. 22:24

     So while we have, uh, the backend, main backend code, uh, on the left, I just also wanted to show you how it looks like on the front end really quickly.

128. 22:33

     Um, s- it's a ba-- it's a, it's a basic Next.js app, Next.js 13 app. So, uh, there is a

129. 22:42

     layout file, um, uh, that just, uh, defines a basic layout of the whole app and, but there's a one important or main important file which is called Page, which is our main page.

130. 22:55

     And there, uh, we are using, uh, one hook from the Vercel AI SDK, which is called useChat. And this is how we are streaming the messages, uh, and s- uh, from the server and sending the messages to the server.

131. 23:09

     So, uh, the useChat here, uh, it's already implemented there. You don't need to write it. It's hitting the endpoint API, uh, uh, /api/chat, which is what we implemented on the left.

132. 23:23

     And, uh, it takes care of all the streaming, uh, and, uh, sending the messages on the, on the backend. And we just then pass the messages to our chat component, which does those, uh, chat bubbles.

133. 23:38

     So, uh, how do you get your... What, what kind of code did you generate to this last, last

134. 23:44

     Um, now, now, uh, the, the goal was just to get a communication with Claude, um, uh, with Sonnet. And, uh,

135. 23:53

     later, like, we will implement Python.

136. 23:55

     Right.

137. 23:55

     Yeah. We will add a tool for running Python. So we will do it through, uh, function, uh, calling or tool usage.

138. 24:03

     If anyone needs it, there's a channel now for the workshop, so code can get thrown in there and-

139. 24:08

     Thank you. Okay. Uh, can you, uh... So, uh, h- how many of you, uh, has, uh, have managed to get to this stage?

140. 24:24

     Just a few people. Okay. How many of you are still working?

141. 24:29

     Okay.

142. 24:33

     What's the name of the Slack channel? Uh, workshop secure code in your AI. It's listed in general.

143. 24:40

     So the name of the Slack channel is workshop secure code in a- your AI, right? Yeah.

144. 24:54

     Yep. What's the name of the branch again? Sorry.

145. 24:58

     Sorry? The name of the branch. So name of the branch. So, uh, the repository we are working with is, uh, ch-ch-ch, this repository. Worksh-- uh, this is my personal repository.

146. 25:10

     I just created it quickly now because, uh, the original cookbook repository wasn't working. Uh, if that's the... Is that the question? Of, of course, the original one. Oh, the original one.

147. 25:20

     Uh, yeah. So, uh, that is, uh, workshop-start.

148. 25:28

     And- [unintelligible] Yeah, that's a, that's a original repo I wanted to use, but it's too big, and it took, uh, uh, too much time for everyone to download it. But if you want to check out the final code for, for this, like, first step, you can go to, uh, original repository under workshop-one,

149. 25:48

     and that should be, uh, the implementation, um, of this step, pretty much.

150. 26:08

     Okay. Um, can I get to the next step?

151. 26:14

     Cool. So, uh, now we just are able to chat, uh, with, uh, Sonnet. There really isn't any, uh, code execution going on. And the way we get to the code execution is, uh, in this case, through tool usage.

152. 26:29

     So you could do it in, in different ways. Uh, the easiest out of the box, uh, is, uh, create a tool called something like Run Python, in our case, for this specific use case.

153. 26:42

     And, uh, then inside the tool, we will actually implement a secure code execution inside a sandbox. Uh, you could have other tools like Run JavaScript or Run, uh, Bash Command, create a file inside the sandbox, or you could do it in a completely different way.

154. 27:00

     So we have seen users, uh, just ditching tool usage because it was pain for the streaming, and, uh, they didn't like how we managed the whole tools. And just, they just asked the model to, uh, define, uh, sorry, to, to generate everything in markdown, and they just parse it themselves.

155. 27:17

     So it's really depending on your use case and what you are looking for. But, like, for this demo, tool is the, uh, easiest one. So, uh, now we are getting to the, uh, part two.

156. 27:29

     Uh, so if you want to eventually get the final code, you can go to workshop-uh, two. Uh, and first thing that we want to define is a new, uh, new f- new tool on our, uh, uh, on our backend, uh, inside the, uh, Verless AI SDK.

157. 27:49

     So, uh, for that, we have a helpful helper method called tool. And, uh,

158. 27:57

     we want to create Run Python. Uh, we will call our tool Run Python. You can name it in whatever way you want. Uh,

159. 28:06

     and there we will use the tool method from Verless AI SDK,

160. 28:13

     and just describe, uh, what it should do to the LLM. So what-- in JavaScript world, uh, usually you would be using something like Zod for that, that describes the schema of the tool.

161. 28:28

     Uh, in Python, you might be using g-- you might know Instructor or something like that. Uh, so first we will provide like a simple description, uh, that just runs Python code.

162. 28:42

     And then, uh, we, uh, want to define the parameters for the tool. And we really have one main parameter, uh, parameter, which is code. That's the code we want to run.

163. 28:54

     Uh, and for that we will be-- now we will be using... Oh, sorry. Uh, I should have, uh, used Zod here. Uh,

164. 29:04

     uh, so Zod object. Uh, we are basically describing the schema, uh, to the, to the LLM and forcing it to always generate, uh, or return this type of structure.

165. 29:19

     So, uh, sorry. First we defined, uh, code, which should be string, and we can also pass description, uh, which is the, the code to run.

166. 29:34

     Uh, this is the main, uh, thing we want to pass. Uh, and just for... Oh, I'm missing a s-- a colon here. Just for the better UI that we can then build, uh, we are-- we will also pass two more, uh, parameters or ask LLM to pass, uh, give us two more parame- parameters.

167. 29:53

     First is title, which is like a, a short title of,

168. 30:00

     uh, uh, des- that describes the code, and we just wanted to u-- uh, show it on the front end. And then a description

169. 30:12

     which is pretty much similar, but, uh, just a little bit longer,

170. 30:22

     uh, of, uh, description of what the, what the code, uh, what the code does.

171. 30:27

     So just by doing this, uh, we are telling, or Vers- Vercel AI SDK is doing it for us. Uh, it's, it's telling the LLM that it has a tool, uh, called runPython that it can call.

172. 30:40

     But we are not calling it, uh, just yet.

173. 30:47

     Is everything clear so far? Or let me ask differently. Is something not clear?

174. 30:54

     There's some typos short.

175. 30:56

     Oh, thank you. So, uh, mm-hmm.

176. 31:06

     What's Vercel AI SDK mean?

177. 31:08

     Sorry.

178. 31:09

     The SDK you're talking about.

179. 31:11

     The Vers- the AI SDK? Yeah, so that's a, that's a SDK that we are using. Uh, it's literally called just AI SDK. Uh, it's from Vercel. Uh, you can learn more about it if you go to, uh, sdk.versa.ai/docs.

180. 31:32

     And it's basically, um, like JavaScript, uh, very easy way to stream output from LLMs and to LLMs in JavaScript world, and especially if you are building like Next.js apps.

181. 31:52

     So inside that tool that we are using, runPython, we need to implement one more function, which is called, um, exe- uh, execute. It will be async function. Uh,

182. 32:05

     and this function is then automatically called when the LLM decides to call our runPython tool. And the parameters, uh, that we just defined here will get passed to the, uh, uh, to the execute, uh, uh, function.

183. 32:24

     So what we will get here-- Mainly what we are, what we really care about here, uh, or really the only thing we care about, is the code.

184. 32:35

     And currently, like, nothing is going to really happen. Uh, but, uh, in- inside this body of this function, we will be calling the Code Interpreter SDK that I showed you previously, uh, and the Code Interpreter sandbox, where we execute the code, and we get a result from that.

185. 33:11

     Okay. So, um, there's a new, one more new concept that we are going to introduce, and which is the concept of the sandbox. Uh, we have a predefined file called sandbox.

186. 33:23

     It's in the lib directory. So if you go to lib, you go to sandbox. I will open it here on the right side. Uh, you will see there's a import, uh, that we are using, which is Code Interpreter.

187. 33:38

     Uh, and we are importing Code Interpreter from the Code Interpreter SDK, uh, which is this SDK I was talking about, uh, a few minutes ago. So what this SDK does, um, is that it will run AI-generated code, in this case Python code, uh, in a secure environment.

188. 33:59

     Uh, and it will, uh, it will be actually running inside a sandbox, which is a small VM. It will run on our cloud. Uh, soon you will be able to, uh, run it on your, uh, on your own, uh, cloud.

189. 34:12

     We have a-- We have two methods here that we need to implement.

190. 34:18

     First is, um, that we need to acquire the sandbox, or we need to create a sandbox. And once we have a sandbox, uh, we then, uh, can run the Python code inside the sandbox, inside the Code Interpreter sandbox.

191. 34:33

     And there we have predefined methods on the Code Interpreter sandbox for that.

192. 34:39

     Sorry, how is it this runs on the device or in the cloud?

193. 34:41

     In the cloud. In the cloud, yeah. Um, so, uh, after-- Once, once we implement this, I can go into more details about the security and, uh, problems with actually running it on anything else but a Linux machine.

194. 34:56

     Uh, but, uh, yeah, f- uh, if, if you have any questions, like, feel free to ask after that. Yeah, go ahead.

195. 35:03

     Do you know, slightly unrelated question on... I know Next.js a little bit, but you've got the runPython on line eight on your left-hand side there.

196. 35:12

     Uh-

197. 35:13

     It really looks like it's, uh, a, a key.

198. 35:17

     Uh-

199. 35:17

     But it also refers to the, the function.

200. 35:20

     Oh, this one?

201. 35:21

     Yeah. So the-

202. 35:22

     Oh, it's just a, it's, it's a, it's a... Oh, I see, I see the confusion. Sorry. Uh, we are not referring to, to the... You mean this function? Referring to this function?

203. 35:32

     Yeah. Is that a key or a function?

204. 35:33

     It's a key. Uh, so on the line twenty-seven where my cursor is, it's a key of the object.

205. 35:39

     Yeah.

206. 35:39

     And, uh, sorry, I didn't realize that. Uh, we will be very soon calling this runPython function that we are importing from the file on the right.

207. 35:48

     Yeah, okay.

208. 35:49

     That-

209. 35:49

     That, that naming is just coincidence. It's not-

210. 35:52

     Yeah, yeah, yeah. Uh, that's a good question. I didn't realize that. Thanks. Um...

211. 35:59

     Can you repeat the question?

212. 36:00

     Uh, yeah. So, so the question was, uh, it was confusion about the naming. So here we have, uh, a key called, uh, or field called runPython. And there we also are importing function called runPython on the line fifteen here.

213. 36:16

     And, uh, it's just a con-- It's, it's not-- The line twenty-seven is not actually referring to a, uh, function from the line fifteen. It's just a coincidence of the naming.

214. 36:27

     Is that function being-

215. 36:30

     The, the one on the line 15, we are about to use it very, very soon. Uh, so we are just going to implement it. So the, uh, definition of the function or the, the, the, um,

216. 36:41

     the function is defined on the, in the, inside the file sandbox that we have on the right. It's inside a lib directory. And this is where we will be using the code interpreter SDK.

217. 36:55

     So, uh, first we need to, first we need to actually get the sandbox, uh, the code interpreter sandbox. And for that, uh, we can, uh, do it very, very quickly, uh, simply, uh, and that would be that we create a new

218. 37:17

     sandbox instance. Uh, so we imported code interpreter here on line seven from the code interpreter SDK.

219. 37:26

     And we can just, uh, call this, and we get-- This actually starts, uh, the full, like, small VM inside, inside our cloud that you can control through the SDK.

220. 37:36

     Um, it takes about currently eight hundred, nine hundred milliseconds,

221. 37:42

     uh, to start it. And you can do it many, many times, uh, uh, advance. So usually what we see is that every user inside your app, every user session would be a separate sandbox.

222. 37:55

     And once we have the sandbox, uh... Oh, sorry. Uh, oops. Uh, actually, we need to... Um, sorry, I-- We need to call code interpreter.create. Uh, I didn't realize. And we need to evade that.

223. 38:09

     Uh, that was Python syntax what I just wrote. Um, uh, and this creates what I, what I, what I just said. There's one problem with this, is that now every time, uh, if I would call this, uh,

224. 38:27

     endpoint, post endpoint in, on my backend, and, uh, I would just call run Python function and then create a new sandbox, there will be no concept of context. So it would create a new sandbox every time, and you would be just running the new code in a completely new sandbox every time you don't have any reference to

225. 38:45

     past code snippets that are generated by the, by the LLM.

226. 38:50

     So, uh, for that, uh, you can, uh, actually have a sandbox running and reconnect to it a little bit later, uh, by, uh, calling reconnect. So what we will change a little bit in our create or connect function is that we first check if the sandbox exists for a given user ID, uh, which would be something like

227. 39:13

     a user session, uh, that you would implement o- once you have authentication. And if it has, we will connect to the sandbox. If it, uh, uh, if it doesn't exist, uh, if it exists, we connect to the sandbox.

228. 39:25

     If it doesn't exist, we create a new sandbox.

229. 39:30

     Uh, so we can first list all the sandboxes,

230. 39:40

     which is code interpreter.list. Uh, you need to add the evade there. And once we have all the sandboxes, uh, we can check if the sandbox has attached a user ID onto its metadata.

231. 39:56

     So what you can do when you are creating a new sandbox is that you can add metadata to the sandbox.

232. 40:03

     Uh, and if the sandbox with our user ID inside its metadata exists, we want to connect to the sandbox, so we keep our old session and have the whole context there.

233. 40:14

     Uh, so first we want to find the sandbox if it exists.

234. 40:22

     And so we call find on the all sandboxes, which is a list, uh, of information of all, all our sandboxes that are running. And, um,

235. 40:34

     and then, uh, we call, uh, metadata. And on that metadata, uh, which is like optional, optional object, uh, or dictionary, uh, it has, it should have user ID key that should be equal to the parameter that we'll be passing to this create or connect method.

236. 40:57

     And we want to evade this whole full co- this code. Oops.

237. 41:10

     Um, okay. So, uh, is everything clear? And once we have, uh, our sandbox info, uh,

238. 41:25

     we check it actually exists. And, uh, if it doesn't exist, that's the moment we want to create a new sandbox.

239. 41:35

     Uh, so we will be actually here returning a new sandbox. And as I said, you can add, um, metadata, uh, to the sandbox. So we will just use this metadata to make a note that...

240. 41:54

     Sorry. Uh, to make a note that, uh, y- this sandbox belongs to this user. Uh, in our case, it will be just one user, but in your application, you would have many, many users.

241. 42:09

     So that's a case when the sandbox didn't exist, and we need to create one, and this line will create a new sandbox in the cloud.

242. 42:19

     And in case, uh, we have a sandbox with a user ID inside its metadata, we want to just connect to the sandbox. And for that, we have-

243. 42:32

     A function called, um, reconnect on the code interpreter object.

244. 42:39

     And there you just, uh, pass the sandbox ID, and that we can get from the sandbox we found on the line 11.

245. 42:49

     Uh, sandboxInfo.sandboxId. And let's just return it. So what this, uh, does

246. 43:00

     is that at first, we ... Okay, I don't. Uh, we check, uh, all our running sandboxes because we probably have

247. 43:10

     separate sandbox for every user or users, every user session, most likely. Um, and then once we have all the running sandboxes, we check the, uh, metadata of each sandbox and check if a user ID, if the sandbox has the same user ID for which we are calling, uh, this createOrConnect method.

248. 43:32

     If it doesn't, if such sandbox doesn't exist, we will just create a new one. We attach user ID to this sandbox metadata, uh, um, and return this whole, uh, promise and object from that promise.

249. 43:48

     Uh, if it actually exists, uh, this me- uh, sandbox for this user, we will just reconnect to the sandbox.

250. 44:07

     Any question, or is everything clear? Cool. So, uh, there's, uh, one, uh ...

251. 44:25

     Actually, now we c- finally can get to, uh, implementing the right run Python me- method. So, uh, once we have our code, uh, this await does ... Oh, sorry.

252. 44:37

     This await doesn't need to be here. Uh, once we have our code, uh, for creating or connecting to our existing sandbox, we can just call it. Um, so createOrConnect to the sandbox.

253. 44:53

     Uh, inside this run Python function, we, uh, pass our user ID.

254. 45:01

     And, uh, once we have this sandbox, we can finally, uh, run the code. Uh, so let's also add console log to make sure that we know what kind of code we're running, we are running.

255. 45:18

     And on s- in s- I, I mentioned that inside the sandbox or the coding interpreter sandbox in this case, uh, we have a Jupyter server running. And we give you programmatic access to the Jupyter server.

256. 45:30

     Uh, so you can call notebook on a sandbox instance and call execCell. And by default, this method will execute Python code. Um, it can be, it can be AI generated, human written, predefined, whatever you want.

257. 45:48

     And we just pass, uh, our code there.

258. 45:54

     And we get results, result from it. Uh, and just need to await that and return the result.

259. 46:08

     Uh, probably we want to log it as well.

260. 46:12

     If you, um, if you make multiple calls, uh, is the, is the, is it like a Jupyter Notebook where you, you have the cells and, and the session is consistent?

261. 46:24

     Yes. Yes. Uh, so it really be- it, it's really just like legit Jupyter Notebook. We actually have a PR where you can connect to that notebook, um, that's hosted inside a sandbox.

262. 46:37

     Um, it's probably at the moment the easiest way how to implement like a consistence, uh, sorry, uh, co- uh, implement, uh, persistence, uh, when you have multiple different code snippets where a one code snippet can referent, reference, uh, a variable from a different code snippet or function.

263. 46:59

     And so y- during the single user session, you just want to keep usually what we have seen, a single context, but it really depends always on your specific use case.

264. 47:15

     Is anyone struggling with the run Python implementation? Yeah.

265. 47:20

     So, uh, this is the, the execCell capture standard out, but you also mentioned standard error from these.

266. 47:27

     Yeah.

267. 47:28

     How do you capture the other-

268. 47:28

     The, the result i- ... So the question is, um, if execCell, uh, returns, uh, it seems like it just returns standard output, uh, right? Yeah. Uh, so what it returns is, uh, our custom object or execution, and there, uh, you have access to standard output, error output.

269. 47:47

     Any runtime errors that are nicely parsed, uh, with, uh, uh, traceback that you can feed back into our LLM so to fix itself, and also any rich output at like PDFs, charts, PNG, JPEG files.

270. 48:03

     Uh, and we'll be just sending this to the front end where we can use it and present it to the user.

271. 48:13

     So this run Python function, it's the function that we, uh,

272. 48:19

     are importing on the left in our, uh, endpoint when we are calling the LLM, where we implemented the stream text. And we can finally, uh, we can, uh, we can finally call the run, uh, run Python function inside our execute method.

273. 48:38

     So, uh, it-- there's really nothing like s- too special about, about it. Uh, one thing that we need to do though is that, uh, we need to use that user ID.

274. 48:50

     So we also, uh, need to get user IDs from somewhere.

275. 48:55

     Uh, we will be just sending this from the, from the front end. Uh, so let's say in a real application, we probably, um, have like a authenticated session with the user ID.

276. 49:07

     Uh, in, in this case, we'll just send it from the front end. I will add it in, in a minute. And that user ID, uh, we pass through the run Python method.

277. 49:20

     And once we have results, we want to return the results from the execute function. So when we call re- return here, and we are returning an object, that's what we'll se- we will get sent to the front end, uh, with, uh, LLM's answer.

278. 49:40

     So that's like the result of the function call, um, or tool call.

279. 49:46

     Okay, so just to make sure I don't forget anything. Um,

280. 49:55

     yeah. So, uh, is, is that part clear how we are importing the run Python, uh, function to our post, uh, the route file, and just we want to call that with the AI-generated code and then get the results to the front end.

281. 50:14

     So now we are getting to more interesting part, and once we implement this, uh, we will have like, uh, actual running it inside our app, and we can, uh, try the app again.

282. 50:24

     Uh, and we get to what you, you, you just ask, uh, and we can, uh, get different types of output from the results. So, uh, first is we want standard output.

283. 50:39

     And the result object from our exec cell, we are just returning it here. There's, uh, contains logs. So we get standard output.

284. 50:50

     It contains, um, error logs. So, uh, oops. Error output.

285. 50:59

     And it also contains two more things. Uh, any runtime error. So that's a error that you would get when AI gen- probably generated, uh, wrong code or code that doesn't work.

286. 51:14

     Uh, we can catch that, and, uh, we will return, uh, that in a error field, uh, which is nicely structured that you can use to, uh, fix itself. And the last part is that we, uh, we get, uh, something-- Oh, I, I have a, a little bit unfortunate naming here, uh, called cell results, essentially.

287. 51:38

     So let's, let's, let's name it cell results. And those are like, uh, evaluated-- It's, it's evaluated notebook cell that's inside a sandbox.

288. 51:51

     Uh, and those can be those charts, images, PDF files, HTML, JSON. It can be just like text. So it's usually the last line of a Jupyter cell, um, uh, in a Jupyter cell-- uh, in Jupyter Notebook.

289. 52:09

     And also, anytime you call, uh, like a display data, like you want to display a chart or anything like that, those-- that will be in a cell results, which is like a array of results.

290. 52:20

     Uh, a- and, and based on the type of a result, you can then parse it.

291. 52:29

     And all these four objects, we just, uh, return it, uh, from the execute function, which will get us to-- which will get it to the front end.

292. 52:43

     Any questions for, for, for this part? Mm-hmm. Go ahead.

293. 52:47

     Can you speak to like Python version packages and all that stuff?

294. 52:51

     Yeah. So the question is about Python version packages and everything probably inside the sandbox, uh, and the environment.

295. 52:59

     Uh, for a code interpreter SDK, um, so I, I mentioned that the code interpreter SDK is wrapping a sandbox, and that sandbox can be completely predefined by you. Uh, in our case, when we created a code interpreter SDK, we installed bunch of packages that you would probably usually use, uh, in a, like, a data science use case

296. 53:22

     or AI data analysis use case, like NumPy, uh, Pandas, uh, Seaborn is installed there. And so, so that's like something that gets you going out of the box. And there's also Python three point ten installed in the, in the environment.

297. 53:37

     You can also create a completely custom sandbox if you want to, if you just give us a Docker file. And, uh, inside the Docker file, you can install whatever package you want.

298. 53:50

     But you just-- you, you will be just using a little bit lower level API that we have. Uh, it's not scary. It's just like, uh, you need to set up everything yourself.

299. 53:58

     So usually you start with the code interpreter SDK, and then once you find out, "Hey, I don't need these packages," or, "I need different," uh, you can install whatever, whatever you want.

300. 54:07

     Uh, question there?

301. 54:08

     Uh, I was curious on your, uh, choice of the Jupyter framework, notebook framework for the code interpreter. Was there-- You mentioned that it was your client's, uh, preferred tool to use.

302. 54:18

     Like, was there any other rationale to go with interpreter you might be able to share with us?

303. 54:24

     Uh, so here we really went just with, uh, an optimi-- So, sorry. Uh, the question was how we went about choosing, uh, the Jupyter framework inside the sandbox. Uh, if there's any more to it, uh, than what, uh, what I mentioned.

304. 54:42

     So the ma- main decision was, uh- How we should set up the sandbox that out of the box it's working for you when you plug it to an LLM and you don't get much, like, errors.

305. 54:55

     Because, uh, that's what users come to us, especially if they are a little bit less experienced with LLMs, and they are wondering why it's not working. So we were really optimizing on how most likely will AI-generated Python code or code will look like.

306. 55:10

     And the answer is, it's gonna be Python code, and it's gonna be like a Jupyter Ser- uh, Jupyter Notebook code with pre-installed packages. Uh, it's essentially like what you, GPT-4 would expect, uh, and that means what their code interpreter tool would, uh, have installed.

307. 55:29

     Yeah.

308. 55:30

     Are you able to expose the, uh, Jupyter environment as a notebook to use that?

309. 55:35

     So the question is if, uh, you are able to expose the Jupyter Notebook, uh, f- inside a sandbox. Yes, you can. Uh, it's, uh... So, oh, sorry. Um,

310. 55:49

     uh, so that's not, like, documented anywhere, but you, uh, actually can set up, uh, exactly this because we have few customers, uh, inter- in ter- in, um, uh, that wants to implement something like this.

311. 56:02

     Uh, the answer is just text us on Discord or just send us an email, and we will give you a guide. Uh, we have, like, few experimental PRs like that.

312. 56:11

     For example, we have a sandbox with full working graphical interface where you can start apps, uh, uh, like game. Uh, so it's everything is Linux, but you can pretty much start anything you want there.

313. 56:23

     A lot of people has been using it for or have been using it for, uh, evals, uh, when you want, uh, the LLM control something more graphical. Or if you want to build something that's more like a tandem human in the loop, uh, you can do that as well.

314. 56:42

     Uh, so, uh, just to sum it up, we implemented the POST request, uh, and the execute, uh, sorry, the run Python tool. That's finally now we should be able to run the code on the front end.

315. 56:54

     So, uh, one last thing we need to do is that we need to go to our page do- a page component, page.tsx.

316. 57:05

     And, uh, there, uh, we need to create this, uh, like, a dummy, uh, user

317. 57:15

     ID. Uh, this would be-- Because we are now passing it on the front end, y- in your real app, you actually had, you would actually had a user ID.

318. 57:24

     And in the useChat hook, um, that's coming from the AI SDK, uh, we just pass the user ID to body.

319. 57:38

     Um, let's add a console log here to the

320. 57:44

     messages, so we can check if everything is working correctly.

321. 57:56

     And if I go back to my, uh, app,

322. 58:02

     open my, uh, Chrome DevTools, uh, and I say something like simple, like, print hello world.

323. 58:19

     I can see I'm getting the messages. There's a new UI, uh, that wasn't there before. And if I check the messages, the reason there's so many messages is because, like, it's constantly streaming them, so it's updating the message, uh, messages object.

324. 58:32

     And so one message, first message is my, uh, prompt, print hello world. And, uh, actually, can I zoom this in? Yeah. First message is print hello world, and the second message is from the assistant, which has the answer.

325. 58:49

     And there is also something interesting called tool invocations, uh, which is an array of all the tool invocations that the assistant or LLM decided to do. And one of them, uh, the only one there is the run_python that we defined.

326. 59:05

     Uh, it has two important, uh, fields, args and result.

327. 59:11

     So args, uh, that's what we described, uh, when we were defining that, uh, tool, that run_python tool. So this is the code. That's what we asked it. And the result, that's what we returned from that execute method, uh, inside that POST function or execute function.

328. 59:31

     And there we have cell results, like, an empty array, because really-- So cell results doesn't capture any standard output. It just evaluates the, the Python code, and there was nothing to evaluate.

329. 59:43

     Uh, like, as a side effect of print, it's just nothing. Uh, no error output, but in the standard output, we have hello world. And that's what, uh, we wanted.

330. 59:54

     So now if we ask it to actually for, for actually something more complex like, uh, calculate by using Monte Carlo

331. 1:00:07

     method and visualize it, it will take a little bit time. Now it's writing the code, uh, which unfortunately, uh, isn't streamed at the moment because the tool streaming isn't implemented there in the AI SDK.

332. 1:00:24

     But if you check it out, we have a tool invocation here, which is our run Python. Uh, it has the whole code here.

333. 1:00:36

     Uh, it, it's, uh, taking one thousand or ten thous- ten thousand, a hundred thousand iterations. So it will actually most likely take a little bit of time to, to, to finish this.

334. 1:00:47

     And if we, yeah, and we gotten two new messages because- Uh, we get a result from the, from the, um, run Python code. And if you check it again, there's a tool invocations.

335. 1:01:01

     We have result and, yeah, we have standard output. But the interesting part is that in cell results, we now have a, um, one of the results which has PNG and text fields.

336. 1:01:16

     So PNG is a base64 image from the cell, and that's the visualization of the simula- Monte Carlo simulation. And the text is just like a human-friendly description of, of the chart.

337. 1:01:32

     Uh, now what we will work now on in the next step is, uh, displaying this, uh, on the front end, on the right side as in the artifact.

338. 1:01:41

     It's still processing.

339. 1:01:43

     No, it's, it's, uh, here. It's already finished. If you check out the cell results, um,

340. 1:01:50

     you see the PNG, and that's what we asked it. Um, the reason we don't see it anywhere here, uh, on-- inside the app is because we haven't implemented the front end yet.

341. 1:02:00

     That's what we are going to do now. But we already have results from the code interpreter sandbox, uh, in our messages, uh, variable.

342. 1:02:08

     Where did you get the messages from?

343. 1:02:11

     So the question is, where did I get the messages from? Uh,

344. 1:02:15

     this is code on the front end. If you go to page.tsx,

345. 1:02:21

     this line, uh, the useChat line, is, uh, returning all the messages from the LLM, including, uh, to- tool calls.

346. 1:02:32

     Thank you.

347. 1:02:32

     Yep. Any questions? Uh, so I would say we are, uh, pretty near the, uh, the end now.

348. 1:02:50

     We need to just display it. Uh, and for that, we need to parse the tool invocation from the, from the message, uh, latest message with the-- We need to get the latest message with the tool invocation.

349. 1:03:03

     So I will, like, for the sake of simplicity, uh, I will just do something, um, a little bit naive here, and that is, uh, I will just care about the latest message we get from the LLM.

350. 1:03:16

     And from the latest message, I will just parse the tool invocations that was there.

351. 1:03:22

     So first, we get a latest message s-

352. 1:03:30

     that has tool invocation and, uh, this is a little bit annoying. The-- But first, we need to reverse the array

353. 1:03:44

     in JavaScript. Um, and then we search in this for message

354. 1:03:50

     that had toolInvocation field, and that field wasn't empty. Uh, so, uh, that array wasn't empty. So, uh,

355. 1:04:02

     the length will be bigger than zero. So this can or m- might or might not exist, uh, this latest message, uh, with toolInvocation. It will not exist, uh, in the first few seconds once we, uh, give an LLM a prompt to actually do something and generate code.

356. 1:04:24

     And now we just want to extract the toolInvocation field, uh, from our, uh,

357. 1:04:35

     from our, um, l- message with toolInvocation. And it, again, it can or might or might not exist. So both, uh, latest message with toolInvocation can be message or undefined, and the toolInvocation can be a toolInvocation or undefined.

358. 1:05:05

     So we, so far, we just added these, these two, uh, or like two, two, uh, function calls or tool calls.

359. 1:05:16

     And we are working with messages that were sent from the back end, and that's the s- part that we implemented, uh, up until now.

360. 1:05:28

     The reason we are doing this is that we will be sending the toolInvocation to the component, uh, that, uh, I will add in a sec. And that component will be showing the chart that, uh, and, and any output from the running the AI-generated code.

361. 1:05:53

     Any questions? Okay. Once we have a toolInvocation, um, we want to, uh, display it.

362. 1:06:13

     And for that, I have, uh, a pre-made component called SideView,

363. 1:06:20

     uh, that exists in components and sideview file. And we-- So, so we can just, uh, import it from there.

364. 1:06:36

     And we want to render it right next to the

365. 1:06:42

     chat, um, component. So-- And this SideView takes the toolInvocation. That's why we did what we did, uh, with those messages here.

366. 1:06:54

     So I imported the side view here. Uh, it's a component that's, uh, already exists inside a, inside a project, and I'm now rendering the side view, uh, next to the chat, uh, component that we already have there.

367. 1:07:22

     And if you check out the app, uh,

368. 1:07:26

     when you, when we ask, when we ask the Claude to print something... Oops. Yeah, it's working now. Um,

369. 1:07:39

     it should... Oops, it's not displaying. It should display the side view, but for some reason, it's not, uh, showing now. And that might be I forgot something to implement inside the side view.

370. 1:07:53

     Um, do you need more time to add the side view, uh, to this file, or I can go to side view? Okay.

371. 1:08:06

     Oh, yeah. Uh, the side view has, uh,

372. 1:08:11

     uh, one thing that I forgot, and that it also expects, um, a field called data. And what that, uh, field does is that we want something that shows us that, um, the tool-- that the LLM is using the tool and we are waiting for it, and we want to know about it on the front end.

373. 1:08:32

     So we want to send some additional data with the LLM's response from our server, uh, from, from our endpoint.

374. 1:08:41

     Um, and unless the way side view is implemented, unless there is not this kind of data, this can be any data, um, it will return null. So the, the idea is just simple.

375. 1:08:53

     We want to show some kind of loader and, uh, uh, I want to show you how you can, uh, know about what is happening with the tool execution more than just like waiting for a new message.

376. 1:09:07

     So for that, uh, if we go to, back to route file,

377. 1:09:15

     and there we have imported a object called stream data. And stream data is a helper object from the AI SDK, and that will help us to stream any, uh, arbitrary data back to the front end alongside our messages, uh, from the LLM.

378. 1:09:36

     So we will just, uh, here, uh, below, uh, parsing the request, we will just create a, uh, data objects called stream data. Oops.

379. 1:10:00

     And it's a stream, and we can append, uh, objects into that stream. So what we want to do is when we are calling execute,

380. 1:10:10

     uh, or when, when the funct-- uh, the tool gets called, we want to first say, "Hey, the,

381. 1:10:18

     uh, f- tool run Python is running." Oops, it's not update, it's append.

382. 1:10:31

     And w- once we get what we need, we also want to update the stream with,

383. 1:10:39

     uh, basically the same, just the status will be, uh, complete.

384. 1:10:46

     And that's what I arbitrarily picked. You can pick whatever you really want, and it's just like the, how the side view, uh, component is implemented, uh, uh, in- inside our front end.

385. 1:10:58

     But the idea is that, like, now you can use the stream data to send, uh, arbitrary data from the back end through the front end and stream it, uh, which is an important part.

386. 1:11:11

     What is just missing now that we need to include that stream data object to our response from our, uh, API endpoint.

387. 1:11:21

     And for that, we need to just, uh, do two things. First, we need to close that stream data s- um, stream, uh, because it's a stream, so it's currently open when we created it.

388. 1:11:33

     And we also need to change the r- uh, res-- uh, return here to include the data that we, uh, the, the stream, additional stream. So we kinda need to create two streams that we are, uh, in a, in a s-- combine them in a single stream and send it to the front end.

389. 1:11:51

     And, uh, for that, what we are going to do is, uh, first the result

390. 1:11:59

     from the LLM call, uh, which is this result from the stream text, can be converted to the stream, to a stream.

391. 1:12:08

     Uh, so result to AI stream. And there, uh, we can do one thing, and that's,

392. 1:12:20

     uh, we can pass it a callback called, uh, on final, which is once everything is done, we want to close our, uh, data stream.

393. 1:12:42

     Okay, um, so this, uh, what this does is that it converts, uh, a result from the LLM that we are using with, uh, w- we get from stream text to a stream.

394. 1:12:56

     And, uh, once- LLM is finished with generating all the f- uh, all the responses. Um, that's what this callback on fail tells us. We want to close our data stream that we created, uh, a few moments ago.

395. 1:13:11

     And the last missing part is just now we want to, uh, we want to return both the stream and the data.

396. 1:13:20

     So, uh, for that, uh, there's a object called StreamingTextResponse from the AI SDK. It's already imported.

397. 1:13:34

     And if we call it, uh, you, you can pass it three, uh, should pass it three parameters. First is the stream, which is the response from the AI. Second is the initial data.

398. 1:13:46

     We don't want any. And the last is any, uh, uh, arbitrary data that you want to send along the stream.

399. 1:13:54

     Okay. Any questions?

400. 1:14:17

     What checkpoint are you on?

401. 1:14:19

     Uh, this should be stage, uh, uh, workshop dash, uh, four.

402. 1:14:36

     Can you go back to page.tsx and-

403. 1:14:38

     Mm-hmm

404. 1:14:39

     ... make changes?

405. 1:14:51

     So actually, on the page.tsx, what we need to do is we need to, uh, get the data object from the useChat hook. Uh, so that's what we are going to add now.

406. 1:15:16

     So we haven't modified anything on the side view, right?

407. 1:15:18

     Oh, sorry.

408. 1:15:19

     Side view loading. Side view frame, we haven't modified anything.

409. 1:15:24

     Uh, the question is if we have modified side view, uh, component. We haven't yet. Uh, we just, uh, are rendering side view component here, but it needs additional data about the information if the, uh, code is, uh, still running.

410. 1:15:40

     Uh, uh, the, the AI generated code. And that's what we are adding now. And once we add that, it should, it should display.

411. 1:15:51

     Uh, so if you, uh... One last part for this data stream, uh, is that we, uh, need to extract the data from the, that we are sending from the back end, uh, and get it on the front end.

412. 1:16:06

     So useChat, again, from the AI SDK, is pretty handy here because, uh, we can just add-- it, it just returns another field called data, which is exactly what we are sending.

413. 1:16:18

     Uh, we can even print it. And, uh, all we need to do is once we get the data variable, we just pass it as a prop to our side view.

414. 1:16:36

     And that should be, uh, it. And now if you go to the app again and ask it anything that would result in generating code,

415. 1:16:55

     uh, it's a little bit, uh, Sonnet is a little bit slow.

416. 1:17:00

     Yeah. It's... And now we have the side view rendered.

417. 1:17:06

     And it's currently just showing code. Uh, but what we will add as the last part of this workshop is the preview, and there we'll be rendering, uh, any charts returned from the code interpreter and any standard output and, uh, error output.

418. 1:17:25

     Can I ask one quick question? What's the special about Anthropic's

419. 1:17:41

     Claude here?

420. 1:17:41

     Uh, the question is, what's special about Anthropic's, uh, Claude, uh, model here? Um,

421. 1:17:49

     you could use any model you want. Uh, it's just the new Sonnet model is really good with code generation. And so it usually is capable of one shot, same, like, relatively, I would say, like, advanced examples.

422. 1:18:05

     Not like full projects, but it can give you interesting results.

423. 1:18:21

     Yeah. Question?

424. 1:18:22

     You mentioned that all your runtimes are Linux at the moment, and I wondered, uh, I think you hinted that there were some security reasons that that was the case.

425. 1:18:30

     And I wonder if you could expand on that a little bit.

426. 1:18:32

     Yeah. The question is, uh, that all the sandbox runtimes are Linux, uh, and what are the reasons for that, uh, and probably if, if those are security reasons. So yeah, that's correct.

427. 1:18:45

     All those runtimes are, are Linux. And so there are two parts, uh, of the, of that question and the answer. So, um, the sandbox is a regular VM. It's not a, it's not a container.

428. 1:18:58

     Um- And that VM is running on our server, on a host machine. And so both host machine and the sandbox operating system is Linux. Uh, the reason for that is that we are using something called Firecracker, which is an open source, uh, VM from AWS, and they are using it for running, uh, AWS Lambdas.

429. 1:19:21

     So it's really battle tested with running, um, untrusted code. And Firecracker is very specific in a way that it requires, uh, Linux with specific kernel for security reasons. And inside the Firecracker, inside the VM, you really can't run anything else at the moment than, than Linux.

430. 1:19:51

     Yeah. Sorry. Uh, go ahead.

431. 1:19:54

     So you mentioned the dependencies are keeping the default in sandbox, right? So how hard would it be, I'm just curious, to read the imports and customize the sandbox?

432. 1:20:06

     Y- it's basically just your custom Docker file. Uh, so the question is, um, how hard would it be to customize the sandbox, right?

433. 1:20:13

     Yes.

434. 1:20:15

     Uh, with install, uh, custom dependencies.

435. 1:20:18

     Yeah.

436. 1:20:18

     Uh, you would just give, uh, give us, uh, your Docker file, and inside the Docker file you would have something like pip install whatever you want, or you can have even NPM.

437. 1:20:30

     Uh, if you want to build a little bit more communication on top, you can-- We have users running, uh, sorry, uh, Fortran. So, uh, you can really run anything you want.

438. 1:20:42

     Uh, depends on how much, uh, customizability you want. Uh, I can show it after, after a call, but if you go... Oh, sorry, after the, after the, uh, workshop.

439. 1:20:53

     If you go to our documentation, e2b.dev/docs, we have a guide there for customization, uh, specif-specifically for the code interpreter SDK.

440. 1:21:08

     Yeah.

441. 1:21:11

     You mentioned Firecracker and there was also a Docker container component. So, uh, is it like a two level

442. 1:21:18

     Yeah. So the question is that, uh, if we are using just Firecracker or Firecracker in containers together, uh, I, I think like the confusion comes from the Docker file I mentioned.

443. 1:21:29

     Yeah. So, uh, we are using just Firecracker. Uh, the containers and the Docker file, uh, w- we are using that only for like a simple way to define your Firecracker environment, essentially.

444. 1:21:45

     So what we do on the background, uh, once you give us your Docker file to customize the sandbox, is, uh, we start it as a container, extract the file system, and then convert this to, to Firecracker VM.

445. 1:22:00

     And, um, then when you start your sandbox, you are essentially starting your custom container, uh, programmatically.

446. 1:22:09

     Yeah. Question there.

447. 1:22:11

     What's like the starting time when you guys custom containers?

448. 1:22:14

     Uh, was the static, uh, was the timing?

449. 1:22:17

     Yeah, like after you start the sandbox.

450. 1:22:19

     Oh, yeah. Um, so how much time does it take to start the sandb- uh, custom sandbox? It takes the same time as a pre-made sandbox. Uh, so at the moment, it should be around nine hundred milliseconds, uh, every time you call that, uh, sandbox.create that I wrote there, uh, in SDK.

451. 1:22:39

     And, uh, soon it should be around four hundred milliseconds.

452. 1:22:46

     Okay, now we are getting to the last part, uh, of this workshop, and that's in case you want to just, uh, check out, uh, git checkout the last part.

453. 1:22:56

     It's, uh, workflow dash five. And we will go into the side view, and what we want to do, we want to implement this preview, which, uh, that's, uh... So that's essentially the result of the code execution that we, uh, get from the, from the LLM.

454. 1:23:15

     Um, let me close a few files first.

455. 1:23:20

     So, uh, we really care about the side view component. Uh, uh, the o- only thing we care, uh, for now. And if we open the side view component,

456. 1:23:34

     uh, it's like scaffold and predefined. Uh, the, uh, thing that's missing there is that we have, uh, artifa-- uh, sorry, we have a artifact view, uh, component that's also already exists.

457. 1:23:50

     That's currently c- commented out, so that we need to uncomment inside the side view.

458. 1:23:56

     And inside, uh, let me turn that on left, uh, put it on left. And inside the

459. 1:24:06

     artifact view that's on the right now, um,

460. 1:24:14

     if you go to artifact view function, which is the component, we have a bunch of to-dos. So let's, uh, focus now on one main, uh, to-do and, uh, item, and that's render image.

461. 1:24:27

     And so let's just optimize now for the use case where we want the LLM to render a chart or an image, uh, an image that we, uh, that we want.

462. 1:24:36

     And we want to display that image here. So what artifact view gets through props, uh, propagation is, uh, the result of our code execution. That's the result here. Uh, that's what we send from the, from the back end.

463. 1:24:54

     Uh, that's this file. Um- And we want to access the cell results because cell results, uh, has, uh, this object has all the PNG files, JPEG files, HTML, PDF, everything around that.

464. 1:25:12

     Um, so, uh, you can see that we are already doing the parsing here, uh, when we are extracting the cell results, standard output, error output, and runtime error from a result object.

465. 1:25:27

     And if there are any cell results, uh, we just expect there will be PNG. Uh, in, in a real app, you probably would need to check that, uh, a little bit more thoroughly.

466. 1:25:40

     And if we have a PNG file, we will just, uh, render image, and that's basically it. So,

467. 1:25:49

     uh, I will just add a f- a little bit more styling here. Um,

468. 1:25:55

     we will use Next.js image component, and because the image, uh, the image is PNG image is in base64, uh,

469. 1:26:06

     that's exactly what we are going to use here.

470. 1:26:22

     And width and the size will be like 600, 400. It's really arbitrary for now.

471. 1:26:31

     And, uh, we just want to put this image in a div container

472. 1:26:46

     so it's centered. And make it a little bit nicer with padding.

473. 1:26:58

     And render our logs there. The logs output is something that's already prepared here, uh, in this file.

474. 1:27:06

     Uh, so, uh, once we add this, um, when we, we should see an image, uh, inside our app if we ask for something like that Monte Carlo sim or just like, uh, generate or create a random chart or like 3D chart.

475. 1:27:34

     And here it is. So this came straight from the sandbox, uh, from the, uh,

476. 1:27:42

     Jupyter server running inside the sandbox. Uh, the next step would be that we won't be able to cover in this, uh, in this workshop. Uh, but next step would be how to make these charts interactive, uh, because now it's just an image, uh, generated from the, from the sandbox.

477. 1:28:00

     One of the ways to do that, uh, I'm gonna go back to code. Um, uh, one of the ways to do that would be, uh, use the sandbox not for generating images, but to operate on top of charts.

478. 1:28:12

     Uh, sorry, on top of CSV files on your data set. So what you can do is that you can upload files to the sandbox. You can connect a cloud storage to a sandbox that you already, already have bunch of files.

479. 1:28:26

     And, uh, you can just let AI know, uh, the LLM know about how these files look like, like CSV files, what columns are there, and just ask it to questions about this data set.

480. 1:28:40

     And with usual, like Sonnet 3.5 is very capable of this already. It will start generating, um, code that's a- capable of extracting data from our CSV file, send the data, run that code inside a sandbox on top of our CSV file, send it to a front end, and on the front end, generate, uh, or display this data

481. 1:29:02

     with something like, uh, Chart.js, Plotly, uh, any, any library, library you want. Yeah, the question?

482. 1:29:08

     How table previews does this?

483. 1:29:10

     Uh, uh, sorry, can you repeat that?

484. 1:29:11

     Does code interpreter have table previews? You can display table previews.

485. 1:29:16

     Uh, if you can display table reviews, uh, previews. Uh, so the... Yeah, the question is if we can display table previews. Yes. So anything you could display besides like interactive widgets, widgets, uh, inside a Jupyter s- uh, Notebook, you will get in this result object or cell results object.

486. 1:29:34

     Uh, uh, and it can be data frames, for example.

487. 1:29:38

     Uh, is that an image or is it a interactive widget?

488. 1:29:42

     Uh, the question really is, uh, it depend-- Uh, sorry. The, the, so the question is if the table preview is an image or, uh, something or just like a description of the table.

489. 1:29:56

     Uh, the answer is it depends. It depends on what you tell the LLM to do. So if you ask the LLM to, um, use the right libraries, uh, that when are evaluated return, for example, HTML, you will get HTML on front end, and you can just display it.

490. 1:30:13

     Um, so yeah, it r- depends on how the AI generated code looks like and what kind of libraries you are using and dependencies you are using inside the code interpreter.

491. 1:30:25

     Yeah. Go, a question here.

492. 1:30:27

     Do you want to say anything more about security architecture? I mean, you've given us this... I guess it's really simple, but it's just executing in sandbox. But do you want to say more about it?

493. 1:30:38

     Yeah. So the question is, uh, if I can go a little bit deeper into the security of the sandbox. Um, so, uh, Firecracker really does very heavy lifting here.

494. 1:30:51

     And, uh, the way it works, uh, and our infrastructure, uh, works is that you start a sandbox. We, uh, have a small VM, uh, prepared for you. Uh, and that VM is the Firecracker VM.

495. 1:31:09

     Uh, and the VM is designed in a way that when you try to get outside of the VM, it just restarts. Uh, so we are actually giving you full root access.

496. 1:31:19

     You can run any command you want, and when you ... Unless there's, like, a network, um, hole somewhere where you could just connect to a third-party service, when you try to something like exit or system restore or anything like that, the, the, the machine will just restart, and we'll just start a new sandbox for you.

497. 1:31:39

     Uh, on top of that, that, uh, Firecracker is, uh, wrapped inside a jailer, so which, uh, removes ability to run s- certain sys calls, which, uh, usually you shouldn't really care about.

498. 1:31:55

     Everything just works, uh, as, as you would expect. And, um, the way Firecracker, uh, works for the, like, high level security is that it's using Linux's and, uh, kernels KVM, uh, for virtualization.

499. 1:32:11

     So you are ... There's a l- little bit more overhead instead of containers because it's a full VM. It's not just a process. And so every, like, user session is a separate VM, so, uh, that is isolated and, uh, can't ...

500. 1:32:27

     It really doesn't know about other VMs or sandboxes, uh, inside a network. So it's out-of-the-box multi-tenant environment if you look, uh, at the sandbox as a single tenant.

501. 1:32:37

     Say that again.

502. 1:32:38

     Uh, it's a multi-tenant environment, uh, if you look at a sandbox as, uh, one of the tenants.

503. 1:32:44

     Can I ask a question?

504. 1:32:45

     Yeah. Question.

505. 1:32:46

     Do you save the data that you're ... that you're going to code as well as the result? Do you log that in your system?

506. 1:32:53

     Uh, the question is if we save any data that you send to the, uh, to the code interpreter sandbox or to the sandbox. Uh, the answer is no. Uh, so at the moment when you start a sandbox, um, and you kill the sandbox at some point or it just closes by itself, uh, e- everything is destroyed.

507. 1:33:16

     Uh, and what we log is only that you did certain operations. So you, for example, like, uh, created a file, but we don't save, like, content of the file.

508. 1:33:26

     We just want to, like, know about what is happening inside a, inside a VM. And, um, what you can do for the persistence, uh, is you can connect your cloud storage, uh, to the VM and save the data there.

509. 1:33:40

     One more question.

510. 1:33:42

     Yeah. Ques-

511. 1:33:44

     So these, these VMs, uh, are they being randomly created on the cloud? How do you reconfigure these VMs when you-

512. 1:33:51

     Uh, can, can, can you, sorry, repeat a little bit louder?

513. 1:33:54

     You mentioned about the Docker files that you can reconfigure, like, Python version and respective packages. How do you start on that?

514. 1:34:04

     Yeah. So the question is, uh, did I mention Docker files for customizing the sandboxes, the VMs? Uh, uh, yeah, so, um,

515. 1:34:16

     I actually, uh, I can show it to you here. If you go to e2b.dev/docs, what you can do, uh, there is we have way to customize the code interpreter.

516. 1:34:30

     And the way we do it, um, is through a Docker file. And, uh, it's a regular Docker file where you can use most of the Docker file things, uh, and, and keywords.

517. 1:34:42

     Uh, it just needs to be Ubuntu-based. And basically, what the Decker f- Docker file is, uh, good for is that you can define the, like, a file system of the VM, the environment of the VM.

518. 1:34:54

     And, uh, that can be, like, you can install any packages. You can, uh, define any environment variables, uh, save any files or, uh, pre-save any files. So we have users save, like, a scaffolded Next.js app that just the LLM can send render, um, uh, generated components.

519. 1:35:13

     And, and they sh- then the LL- uh, the Next.js app is just running inside the, inside the VM. And so the Docker file really is just, like, a means of letting us know how you want the environment to look like, um, and then we just convert the Docker file, actually that container, to a, a VM.

520. 1:35:35

     Does, does that answer the question? Yeah. Uh, here's question.

521. 1:35:39

     Is the conversion from Docker file to Firecracker VM, is that a standard process or is it ... I mean, how, how standard a process is that? Like, you're layering one file system from Ubuntu to one VM.

522. 1:35:54

     Is it because they both are same Linux-based, uh, kernels that they just, uh, work or ...

523. 1:36:01

     It's, uh, it's ... Uh, the question is, uh, h- w- uh, how does look like that conversion from a container, uh, to a sandbox, right? Like, that file system.

524. 1:36:12

     Uh, it's actually much simpler than it sounds. So what we really do is just ... You know, like, you know how you have different types, uh, of file systems, um, uh, in your computer?

525. 1:36:26

     Um, I don't think, like, many people touch it, uh, touch it nowadays, but back in the days, if you are booting your machine, you can pick, uh, how your file system would l- look like.

526. 1:36:35

     That's what essentially we do. Uh, so we take, we run the container, and, um, on our infrastructure, there's literally a shell command, uh, that copies the whole file system to an outside file, outside from the container.

527. 1:36:51

     And we just convert that file to that correct format, and that format, uh, is supported as a file sy- uh, as a root file system for the Firecracker. So it's really is just

528. 1:37:02

     a bunch of files together that we extract from the container.

529. 1:37:07

     And, and it's, it's very similar to how the container s- get created. So if you want, you could theoretically, you know, uh, uh, images have this bunch of TAR files and layers.

530. 1:37:18

     You could extract these layers from an image, and you wouldn't probably even most likely didn't need to run the container, uh, and you could just take the image and convert it to the, to the file system for the VM.

531. 1:37:31

     So you said conversion code is part of your code base or is this-

532. 1:37:34

     Yeah, yeah, it's, it's in the code base. So we are fully open source. Uh, uh, we have bunch of repositories here. There's a repository called Infra, which is a little bit less known.

533. 1:37:45

     Um, it's, like, one disclosure is it's not that now very friendly to get it running on your own. That's what next thing we are working on. And, uh, here, uh, if you take a look into, um, special sc- scripts, uh, you will s- find a script to, for conversion of the Docker file to, uh, the sandbox, uh,

534. 1:38:09

     the, the VM file system. Uh, there was one question first.

535. 1:38:13

     Uh, yeah. How do you end up registering your own custom modules inside the sandbox?

536. 1:38:19

     Yeah. The, uh-- So the question is, how do we end up registering, uh, custom modules inside a sandbox? Uh, very similarly, uh, like what, uh, I talked about it, uh, uh, few times.

537. 1:38:31

     When you are creating a-- You, you can create a Docker file that describes how the sandbox will look like. Uh, we convert a Docker file to a custom sandbox, and then, uh, we start a sandbox, and you can start a sandbox using the SDKs.

538. 1:38:47

     Um, because the Docker file creation happens, uh, with your Docker instance, we are not doing it on our cloud, we can use private packages or private images or whatever you really want.

539. 1:39:01

     Uh, yes. Question.

540. 1:39:02

     So I, I was looking at this report earlier, and it says, I think at the bottom, that GCP is kind of the only platform that's supported currently. Do you have plans to support others in the future?

541. 1:39:12

     Uh, the question is if we have, uh, plans for supporting other clouds than GCP. Uh, yeah, strong yes. So AWS is, uh, actually where we are even, uh, migrating, migrating with our cloud, uh, version.

542. 1:39:24

     Uh, we want to support AWS as the first thing. Then, uh, we will have a proper, uh, tutorial for GCP, and the next one is supporting any Linux machine, uh, so custom on-prem clouds.

543. 1:39:40

     And, and just as a follow-up, so is your business model eventually-- So for, you know, instead of firms trying to build this capability themselves today, there aren't really many options for this.

544. 1:39:49

     Do you wanna sort of sell this as open source but support it, uh, as, as well? Like, is that, is that where the revenue would come from?

545. 1:39:56

     So the question is about business model. Uh, yeah. So, um, two parts. And it's, it's pretty, uh, similar or pretty much, uh, any, uh, open core business model. Uh, so most features are open source.

546. 1:40:12

     You can self-- If you can self-host it on your own, uh, and, um, manage it on your own or, or you can use our cloud. There will be a few features more targeted, uh, for enterprises, especially around, like we have-- we got quest- we get questions about how do you load like terabytes of data into a sandbox.

547. 1:40:30

     Like that's not something that you, um, uh, encounter during your weekend hacking or something like that. And then we are also planning a lot of work on top of observability, uh, because like as the LLMs are getting better, uh, you will want to know what is happening inside a sandbox because it will not be just like a

548. 1:40:48

     running a simple Python script. It will be more like a workspace, like a permanent workspace for your agent, um, uh, your AI app. And, uh, you're gonna want to know what is happening inside the sandbox.

549. 1:41:02

     Uh, like are the f- Are any files getting created? Is any network requests, uh, happening? You want to have, uh, programmatic access to this information, and you want to be able to like stop the sandbox before it actually happens if you don't like it.

550. 1:41:17

     So those are like some of the things that we are working on next and might ex- some of them might stay, um, like private or behind a license.

551. 1:41:25

     What's the timeframe for that feature? Or let's say, you know, we love having that in your product.

552. 1:41:29

     Uh, this summer.

553. 1:41:31

     This summer.

554. 1:41:31

     Uh, yes. Question there.

555. 1:41:33

     Is it possible to have somehow GPU containers?

556. 1:41:37

     Uh, yeah.

557. 1:41:37

     You have like your

558. 1:41:44

     So the question is, uh, if we have any GPU sandboxes, correct?

559. 1:41:48

     Yep.

560. 1:41:49

     Yep. Uh, not at the moment. So, uh, that's very, uh, intentional because we kinda don't want to go, um, into like a whole GPU provider, uh, business. So a- and even like now, usually if you have your-- if you are serious about GPU work, you want to offload it to someone who does it really, really well, and

561. 1:42:09

     there's like a lot of players in this space. Um, so, uh, yeah, the short answer is no. The long answer, it makes sense for the future, but not something that we are focused on right now.

562. 1:42:24

     Yeah. Uh, question there.

563. 1:42:26

     Uh, yeah. Can you say a bit more about connecting these cloud data stores to the sandbox? Like, uh, what kinds of limitations or recommendations?

564. 1:42:33

     Uh, so th- this cloud what?

565. 1:42:36

     Uh, you mentioned that you can connect cloud data stores.

566. 1:42:39

     Oh, yeah. So the question is how you can connect something like S3 or Google Cloud Storage.

567. 1:42:44

     Yeah. What kind of-- There's what kinds of, what kinds of stores can you connect?

568. 1:42:46

     Mm-hmm.

569. 1:42:46

     What kinds do you recommend?

570. 1:42:48

     Yeah. So, um, uh, I, I will show you. We have a guide for this in our documentation. So currently you can connect anything that looks like S3, uh, or like has S, S3 API.

571. 1:43:03

     Uh, mainly that's Google Cloud Storage, uh, that's, uh, Amazon S3 obviously or AWS, and, uh, Cloudflare's R2. Um-

572. 1:43:16

     There's a little bit more complicated setup that in the ne- the future version of the SDK will be just like a simple call that you say, "Hey, I want to mount this endpoint."

573. 1:43:25

     And, uh, you need to basically, what we are doing is we are using something called FUSE protocol for that, which allows you to connect your, uh, cloud storage to a file system, and then it sort of looks like basically that it's part of your file system.

574. 1:43:44

     But actually, when e- every time you are ex- uh, reading or writing to a file inside a cloud storage, uh, you are, uh, making a network request. But the nice thing about that, especially if you are, like, more enterprise customer and, uh, you have, um, uh, uh, you care about your users' data, is that the data doesn't

575. 1:44:03

     really leave, uh, uh, your storage. Yeah. Question?

576. 1:44:12

     So I understand you're using VMs with Firecracker. Is there a reason you're not planning on using containers instead? I get that they're doing the heavy lifting for you now.

577. 1:44:23

     Uh, so, uh, the question is if there's any reason or what is the reason we are not using containers. Um, I have a separate presentation on that, but-- and the short answer is security.

578. 1:44:34

     Um, uh, so we, um, uh, when customers come to us, they come to us 90% it's, like, in-house solution that's either serverless function like Lambda function, or they are managing a fleet of containers or something like that, uh, with Kubernetes.

579. 1:44:55

     Um, and, uh, you can sort of make containers secure. It just takes, uh, it just needs more work and, uh, you then want things like,

580. 1:45:09

     like running Docker in Dock- or it's hard to run, for example, Docker inside Docker. Like it's, it's doable, but it's just like more pain. And, uh, so ergonomics and security is like part of the answer, uh, where we eventually figure out, "Hey, like, if you make an ability to get you a VM really, really fast, um, y-

581. 1:45:31

     you will just, like, get f- full computer for your, for, uh, LLM," uh, which is like a nice win. And there's nothing you need to do special. It's just like a Linux machine.

582. 1:45:42

     Uh, the second, the second part of this answer is, uh, something we don't have implemented yet, but we think will be super important, and that's snapshots. Um,

583. 1:45:54

     with Firecracker and few other VM projects out there, uh, you can very easily make a snapshot of the whole VM at its current state, and not only file system, uh, but also memory.

584. 1:46:06

     And you can do it pretty fast. It takes, uh, again, I think like eighty milliseconds or something like that. And we think it will be super important in the future, especially once LLMs get more capable and cheaper.

585. 1:46:19

     You will want to go into sort of this tree search problem where you can have many agents, uh, versions of your like AI app, uh, explore the whole space, and every time, uh, in, like if you imagine like a graph, like a tree, every node would be like a snapshot of the VM, and you can come back

586. 1:46:36

     to it and load it again. Uh, so you can-- you bring sort of this, um, uh, determinis- a little bit of determinism into a non-deterministic system because you can just save it at any point and come back to it.

587. 1:46:53

     And we can also save it and, and prevent your agent from doing anything you don't like.

588. 1:47:02

     And, uh, just one last note is that something like this is sort of possible with Docker con- or containers. It's just like it, uh, the technology isn't really finished.

589. 1:47:12

     Uh, it's like half working, half not working. It's called Criu, C-R-I-U.

590. 1:47:25

     Yeah. If, uh, anyone has, uh, any last question, uh, now is the time. If not, uh, thank you for, for following me, uh, and coding along. And if, uh, you want to ask, uh, any question like one-to-one to me, uh, feel free to, uh, catch me on in the hallway, uh, or, uh, send us a message on

591. 1:47:45

     Discord or just email me, uh, it's, uh, on, on E2B. Uh, it's [REDACTED:email_address]. Thank you. [audience applauding] [upbeat music]