From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

Talk transcript

1. 0:00

   [on-hold music]

2. 0:12

   Welcome everyone. Can you guys hear me okay?

3. 0:14

   Yeah.

4. 0:15

   I've been standing here for fifteen minutes without saying anything, so [laughs] we can start now. My name's Abhishek. I'm on the RL and agent infrastructure team at OpenAI. Um, what that means is we work on the infra for reinforcement learning specifically, and on the product side, we also develop infra that helps run untrusted code, uh, as part of

5. 0:36

   ChatGPT, Codex Web securely and reliably at scale. This talk is called From Fork to Fleet: Designing an Agent Sandbox Cloud. Um, I'll be very clear that there are a lot of words in the title that have OS and infra concepts, but this is a first principles talk.

6. 0:53

   Uh, so we'll cover what sandboxes are and why they are needed from first principles. We will also try to cover design intuitions around designing an agent sandbox, uh, cloud to run sandboxes securely and reliably at scale.

7. 1:07

   So if, if some of these words don't mean anything, don't be worried. We'll explain from first principles and, and go from there.

8. 1:15

   Um, last year I gave a ta- talk called, uh, How to Build an AI Sandbox from Scratch. Uh, if, if you're interested, you can, uh, look at that talk as well.

9. 1:24

   Think of this as a spiritual sequel, uh, to that talk.

10. 1:31

    Okay, so now let's forget about sandboxes or clouds for a second. Let's just go back in time. ChatGPT came out. It's a very large pre-trained model. Uh, people ask all sorts of questions, and it responds really, really well, really, really human-like answers.

11. 1:45

    But when people ask questions like, "What is three plus three?" Or, "What, how many Rs in strawberry?" Sometimes it works and sometimes it doesn't.

12. 1:54

    It answers questions like, "What is three plus three?" quite well because it's trained on the entire Internet, and apparently people have written three plus three equal to six many, many times on the Internet, so it gets it right.

13. 2:07

    Uh, but people haven't asked how many Rs in strawberry enough on the Internet, and so it gets it wrong. And so it's obvious that for anything code or math related, uh, or any problem which has a verifiable reward, which means that it can be tested whether it's true or false, uh, the model needs something more.

14. 2:26

    And the key unlock was that given the model, uh, tool calling capability or a way to execute code, the model gets these verifiable reward questions around code and math correctly.

15. 2:39

    And that's from first principles came that if we give the models the ability to execute code, it can hill climb and be very, very good at, uh, math, code, and other domains that have verifiable rewards.

16. 2:51

    Like cut to twenty twenty-six, and we are seeing the, the consequences of doing that at scale. Uh, so now how can it answer what is three plus three, and how can it answer how many Rs in strawberry?

17. 3:04

    Well, it can write code to do this [laughs]. So if you see the diagram, like, uh, we have a training loop, and the training loop gives it tasks or questions, and then the training and the harness then parses the response of the model, and the response might say, "Hey, execute code on my behalf."

18. 3:21

    Uh, the, the harness is responsible for executing the code, and then a grader judges whether the answer is correct or not, and then the training loop backdrops and changes the weights and that's how we train it to do two things.

19. 3:34

    We train it to call code execution or tools on certain classes of problems, and secondly, we ensure that the code it executes actually solves the problem. So this is why, like, tool calling is important, uh, on the training side.

20. 3:51

    Now let's talk about the product side. In the previous slide, we showed how the models are trained to emit, uh, code in order to solve certain tasks. The agent executes the code, and we verify the reward.

21. 4:04

    Well, all of this is useless if we don't support this model on the product side. So it's basically the same slide as before, but we don't have a training loop, and now the harness is parsing the response and calling, uh, the tools and executing the code somewhere, right?

22. 4:21

    So where is this code or tool being executed, right? This could be your laptop, like with Codex or any agent you want, or it could be a node on the cloud with Codex Web or ChatGPT.

23. 4:33

    While we can expect the model to generate non-malicious code, but good security practices and hygiene mean that we want to protect our environment, whether it's a laptop or the cloud node.

24. 4:44

    Attacks can be, like, intentional or unintentional, and we want to protect against those. And it could be like trying to get root on your system or trying to exploit a kernel vulnerability.

25. 4:55

    The models are getting really, really big, and they might try to help you in an overzealous fashion and try to get root to do so. So we want to try and avoid and make sure, like, it doesn't attack the node or where it's running.

26. 5:07

    Thus, this is where the sandbox come, comes in. We need the sandbox to run this untrusted code and ensure that it can do its work, but it shouldn't be able to exploit any vulnerabilities and get root on your system.

27. 5:19

    In the cloud, other sandboxes with other users' data must, might be running as well, and we don't want it attacking and getting data of other users out. So a sandbox is a place, uh, an environment in which you can run these tool calls and execute code on behalf of the model securely, and it could be on your

28. 5:36

    laptop, or it could be on the cloud.

29. 5:40

    Uh, I, I think that, like, everyone's probably used OpenClau or any agent like that. Um, I think it's a very, very, very big peek into what's to come. A lot of these agents are running locally on your laptops, and I think it's [laughs] it's kind of a slap on the face for twenty years of cloud computing that everyone's

30. 6:01

    running this locally on, on their laptops. And if you see the, the image, I don't know if this is an actual product, but it's very funny because the lid of the laptop is open.

31. 6:10

    It's because you don't want your agents to sleep. Well, we have a whole cloud. There are, like, industries built on this thing, so I think, like, the future is, like- Us running your agents in the cloud, like they're persistent, long-running, and I really, really hope that moving forward, this is not something you see, uh, anywhere.

32. 6:28

    But people rented like VPSs, they ran OpenClaus on Hexner or Mac Minis in the cloud. So I think like OpenClau was a very, very good like peek into like what might be for like sandbox clouds in the future.

33. 6:45

    We've discussed why sandboxes, uh, are important in both research and product, uh, but they have slightly different needs. Um, on research, we want to optimize for throughput. We want to run many, many training loops at scale, uh, and have many, many rollouts.

34. 7:02

    Um, a rollout is one version of a task. So what is three plus three? And you might have five or five answers to it, and one answer is a rollout.

35. 7:10

    Um, we want to take many, many shots on goal in parallel. Uh, and so throughput is very, very important, uh, in research. In product, latency is very important. I think any, any successful product in the last twenty years has been super fast.

36. 7:25

    So if, if you don't start a sandbox in time, and you don't execute code fast enough, people will churn, uh, from your product. Um, reliability is important on both, both sides.

37. 7:36

    Like if you fail constantly, you've wasted like GPU tokens, uh, on both sides, and GPU is like gold right now. So you want to just make sure all the tokens you're getting, uh, are useful.

38. 7:48

    Um, and similarly on like if, if you're not-- if your agents aren't reliable on the product side, like it's game over, right? Like people will churn from your product.

39. 7:56

    Uh, security is also important. Um, on the research side, like, uh, we are training on OpenAI infrastructure, so if a model gets root and it's not aligned, it can try to attack OpenAI infrastructure, it can exfiltrate and release our model weights or whatever.

40. 8:12

    And then on the product side, it can exfiltrate other users' data, attack the infrastructure as well. So security is important for like both, both, uh, research and product.

41. 8:25

    So today we'll focus on these three pillars. There are many, many parts of a sandbox cloud, but we'll specifically focus on runtime. So how can we run a sandbox on one node securely?

42. 8:35

    Secondly, we'll focus on persistence. Uh, I think compute was the first unlock. People realized you give sandboxes a, a Linux computer, and they do crazy things because they're pre-trained on so much Linux data.

43. 8:48

    But I think now if you help them-- if you give them a computer with an actual disk that can be saved, then they become a true knowledge worker. And the last part is orchestration and how to run these at scale for many, many users at ChatGPT, uh, and Codex scale.

44. 9:03

    Yeah. So before we start, this is a first principles talk. So let's discuss like how Linux executes code on your machine, right? So on Linux, a thread is the smallest unit of execution.

45. 9:18

    Um, the kernel is the thing that provides privileged access to a thread via something called system calls or IOCTLS. So whenever, whenever the user space program wants to access some hardware or some privileged resource, it needs to talk to the kernel and call this like operation that switches the hardware context, uh, to a more privileged context.

46. 9:42

    And so let's look at what that looks like.

47. 9:45

    So if you see here, your processor has like different rings of execution. So based on which ring you're executing in, you get different privileges. So the kernel mode is executing in ring zero.

48. 9:56

    It has the highest privilege, and anything in user mode is, uh, running in ring three. Whenever we want to access privileged resources, we call system call instructions that change the context of the CPU.

49. 10:10

    And so there are two attack vectors in a Linux system, right? First is getting root. In getting root, you're still in ring three, but you're the highest privileged user on the system.

50. 10:21

    So you can actually pretty much do anything on the system. You can read your SSH keys, encryption data, et cetera. And then the second version is actually running a, a kernel mode exploit, so running code in ring zero.

51. 10:34

    Uh, this is terrible. You can actually dump processes, memories, and like, like I don't even want to say what all can happen. So you can get root, and you can have kernel exploits, and if you get kernel exploit, it's like it's a, it's a New York Times article waiting to happen.

52. 10:47

    So, so these are the two attack vectors on a Linux, uh, system.

53. 10:52

    Uh, so with that background, let's design the simplest way to execute tools on a Linux system. Uh, for example, we can literally have an API server that your har-harness is calling, and it-- for every tool call, it can fork a process, exec the tool that the model needs, and just have a fork and exec model, right?

54. 11:15

    So now there are a couple of problems, uh, with this, uh, model, right? A, as we discussed the execution model, the forked process can now directly talk to the kernel, and so the model can try to attack the kernel, get root, or try to get a kernel exploit, right?

55. 11:32

    The second thing is imagine [chuckles] you have a while loop and just forking processes in one tool call, right? So now you've kind of become a bad neighbor or a noisy neighbor, and you've kind of brought down the node and no other tool calls can run, right?

56. 11:46

    So fork exec is the simplest thing you can do. It has one thing going for it. It's the most performant solution, uh, because there's-- it's, it's as fast as just forking something.

57. 11:56

    It's native performance. But everything else is very bad about this solution. Yeah.

58. 12:02

    Uh, [clears throat] so like we saw that fork and exec-ing was bad. There was a noisy neighbor issue, and there's a security issue. We now turn to something called containers on Linux.

59. 12:13

    Uh, you must have heard of Docker and other things associated with the containers. But remember, this is the first principles talk. So let's go what-- in deep into what containers are in their very raw form.

60. 12:24

    Containers rely on two concepts on Linux, uh, namely namespaces and cgroups. Uh, namespaces are for resource isolation, and cgroups are for controlling the amount of resources a, a set of processes, uh, can consume, right?

61. 12:39

    So, let's see in depth how they can help our problems.

62. 12:44

    So, in this diagram, you can see that there are different type of namespaces, uh, a set of processes can have. So, on the left, you can see, uh, this container has a PID namespace.

63. 12:56

    So inside, inside that process namespace, it looks like we have a process hierarchy of PIDs one, two, and three. But if you look at the PIDs from outside the container, they're just regular processes with other PIDs, right?

64. 13:10

    So, you've abs-abstracted one resource, which is the process inside this container. Another example is a mount namespace. So, if you see the bottom half of the diagram, you can mount any file system on top of, uh, any mount inside the container.

65. 13:25

    But from outside, the original mount is still visible, right? So, this is a nice way of like isolating different resources in a container, and there are many namespaces like PID, network, mount, et cetera.

66. 13:36

    Uh, so moving on to the second principle of containers, right? Like cgroups. So now we have a way to isolate resources within containers via namespaces. But remember, I gave the example of, uh, someone doing a while with fork(), right?

67. 13:53

    Like, how can we control that? Well, we can have cgroups and control how much CPU and memory a container can, uh, uh, consume, and that way, like, we can, like, uh, make sure that one container doesn't bring down the entire node.

68. 14:07

    So, namespaces and cgroups provide some amount of isolation, uh, and control on the, uh, system resources to not bring it down in case it's malicious or bad. Yeah.

69. 14:19

    Um, if you've seen the previous diagrams, uh, there's-- the, the fundamental problem with containers is that there are still native processes, uh, running on the host. So, even if we abstract resources and two containers can't attack each other because their resources are isolated, but a process in a container can still exploit the kernel boundary and try to

70. 14:40

    get root or a kernel exploit, right? And once they can get root, they can attack other people's data from other sandboxes, exfiltrate data, and try to get control of the node like we discussed before.

71. 14:50

    Yeah. So, security is a spectrum. Like, you can still get a decent amount of protection from containers. Uh, one example to do this is something called seccomp. So, you can actually have a filter on the amount of system calls, uh, your container can call, and you can also control the arguments each system call can take.

72. 15:13

    So, you can say that you're reducing the attack surface of the kernel, uh, for, for this, like, container and, like, kinda have some amount of sanity on how it can attack the kernel.

73. 15:23

    The problem with this is that many times you don't know beforehand what system calls some container might call, right? So, now you are blocking requests for users, and later on, like, you have to, like, change the seccomp filter to allow something or not allow something.

74. 15:38

    So, the feedback loop is, like, pretty bad for a product like OpenClou or any agent that wants to do, like, crazy things, right? Like, you want to make magical experiences happen in the sandbox, and you don't want to restrict them, right?

75. 15:51

    So yeah, th-this is the fight. Like, to sum it up, like, containers interact with the same host kernel, so they do, they do have some protections, but at the end, it's the same host kernel they're trying to attack, right?

76. 16:01

    And they can get root and kernel exploits, et cetera.

77. 16:06

    Yeah. So, we went from, like, forking to containers. Like, raw fork was obviously very, very bad for all the reasons. Uh, containers are better than fork because they still have some protection, and you can still have seccomp for reducing the attack surface.

78. 16:20

    Can we do better? Can we reduce the host kernel exposure, uh, a little bit more?

79. 16:27

    Yeah. So, the second thing we can discuss is, uh, to solve the attack surface problem is something called gVisor. The crux of the security boundary is the kernel API from before.

80. 16:37

    So, gVisor uses this point as its key security story. Um, so gVisor implements a lot of syscalls in user space. You can think of it as an application kernel.

81. 16:49

    Its sentry is a user space kernel written in Go, and the file system is accessed by another daemon called the gopher. So, the sentry, uh, implements the Linux API itself, including process management, workload management.

82. 17:03

    And whenever you call a system call, it's intercepted, and it's, like, serviced by this user space, uh, program. So, then you can argue that you're not implementing an attacking code that's running in ring zero or kernel mode.

83. 17:16

    You're actually running in user space and ring three, and any exploit you have is still in user space, so it's better than you exploiting the kernel directly.

84. 17:26

    But the fundamental problem is that the sentry and the gopher, like, just going back, the sentry and the gopher are still on top of the host kernel. So, if there is an exploit on them, you can still have a chained two-step exploit.

85. 17:41

    So, you first exploit a problem in the sentry or the gopher, and then you exploit from the gopher to the kernel, right? You can still get to the host kernel eventually.

86. 17:50

    And with, with these models, uh, of, like, five point six and other types, like, you can have, like, like, them, like, figuring out bug reports and these things and, like, trying to chain exploits, right?

87. 18:01

    So, you can still get to the host kernel, right?

88. 18:05

    So yeah, the chain is harder here because it's a two-step chain compared to the others, but it's still reachable. Like, can we do better than this? Yeah.

89. 18:18

    Yeah. That's the same slide. It's saying that in all of these, uh, solutions, you can eventually get to the host kernel, whether it's fork, gVisor, or containers. Yeah.

90. 18:29

    So the key question is, can we have a way in which untrusted or malicious code can have exploits, but they can never or find it very, very hard to exploit the host, right?

91. 18:40

    So even if I can get root, even I, if I can exploit the kernel, I still want my host to be protected, right? Like, that's the final goal we have.

92. 18:49

    So do we have something that we can use for this? So if you've used-- if you've, like, known some of my background, you'll know where I'm going with this.

93. 18:57

    Uh, so turns out Linux provides a very, very nice thing called virtualization, uh, which is hardware-powered at the CPU level.

94. 19:07

    And it provides an abstraction at the hardware level, so-- which means that even if you can get, uh, root or ex-execute in, like, ring zero in the guest, your host is still protected.

95. 19:20

    And this happens because the guest kernel runs in ring zero, but in a separate processor context called VMX non-root, while the host kernel and the hypi-hypervisor run in ring zero in something called VMX root mode.

96. 19:33

    So ring zero gives the guest kernel full control inside the guest, but no control on the host. So you can exploit the guest all you want, but the host is still protected.

97. 19:44

    Uh, the processor has to switch between the guest and the host whenever the guest wants to access some privileged, uh, resource. We'll see how that works. But that is a key trade-off here.

98. 19:54

    There's a performance penalty you pay every time the CPU is switching back and forth between these two modes. Let's see with a diagram what I mean, um.

99. 20:03

    So you can see in, in the, in the old diagram, we showed ring zero and ring three for executing users, user space and kernel code. But here, the guest kernel and the guest user space, uh, are running in separate, like, CPU context.

100. 20:17

     So this is guaranteed at a hardware level. And so if you can-- even if you can get to ring zero in the guest, like, your host is, uh, like, protected, uh, from it.

101. 20:31

     Yeah. Now let's talk about, uh, what, what does paravirtualized and hardware-based virtualization mean. This is the key of all the VM sandboxes you might have seen on Hacker News or Reddit.

102. 20:42

     So we'll, we'll go from step by step, right? So on the right-hand side, you see something called the VMM. Um, is-- can any- can everyone see that on the right-hand side?

103. 20:51

     Yeah. It-- The VMM stands for virtual machine monitor.

104. 20:56

     Um, this software is called a virtual machine monitor, and you might have heard of QEMU when you search, search for Linux virtualization. QEMU and other VMMs, their sole job is to talk to devkvm.

105. 21:09

     So you see that arrow going down, like the devkvm is the hypervisor API of the Linux kernel. All QEMU and other VMMs do is set up the kernel root FS, set up some-- allocate some memory for this guest virtual machine, and call, like, into the devkvm API.

106. 21:29

     As discussed inside, when the, when the guest runs on the left-hand side, it's running as a completely different Linux system. It doesn't know what's happening on the host side, but it has these devices like a block device or a net device coming up as regular Linux devices.

107. 21:46

     But when it actually tries to access the devices, they exit out to the host context, and if you see the block back-end and the network back-end, there act-- there's actually code running emulating the devices in the host, uh.

108. 21:59

     So whenever the guest exits out, it's being serviced by the VMM process and these device processes.

109. 22:06

     So what do we mean by paravirtualization? Uh, so paravirtualization means that we want good performance while we want to access hardware in the guest. So the guest drivers, when you t- when the user space talks to them, the guest drivers are aware that they're running in a virtual machine.

110. 22:23

     And so they talk via something called virtio, uh, which is a more efficient way of the guest and the host talking to each other. Uh, and so inside, there are just PCI devices like any other hardware, and when you talk to the PCI devices, you magically exit out onto the host.

111. 22:39

     Uh, so fifteen, twenty years ago, a bunch of Linux wizards [chuckles] made this thing happen and made this performant, and it, it lit-- it truly is magical, like how it works reliably and performantly across different, uh, hardware processors.

112. 22:51

     And so from the host's point of view, when this is running, you just see a blocked thread. Like if you do a PS, you'll see a blocked thread, but that's running on the guest context.

113. 23:02

     And when the thread exits out, the thread finally wakes up. It wakes up in this VMM process, and the process runs, and it's running inside. Yeah.

114. 23:13

     Yeah. So we had this seismic shift in twenty twenty-three where a bunch of new VMMs came. For a long time, it was just QEMU that was used for running virtual machines.

115. 23:25

     But QEMU has a lot of cruft. It supports many, many architectures. It has many devices, and it's written in C. And historically, many, many escape attacks were attacking the devices written in C.

116. 23:37

     So the first w- the first Rust-based VMM was something called CrossVM, which our team at Google wrote, uh, when I was there, to support Linux virtual machines on top of Chromebooks.

117. 23:48

     And the key point here was that we don't need all the cruft of QEMU, QEMU, and we can use Rust to be a memory-safe implementation of this tricky system software.

118. 23:58

     And secondly, like, we have these emulated devices that we can jail. So if you attack the block device, we've only given it permission to at-access block resources. So you can't access network resources.

119. 24:10

     Similarly, if you attack the net device, we've, we've jailed it, and we don't give it access to block resources. So then you can still have a second gate of security.

120. 24:19

     So both Rust-based safety, like being memory-safe, and this jailing at a more granular level at the device, like, provides better safety than just QEMU. Yeah.

121. 24:31

     So you must have heard this word microVMs, like, everywhere and, like, no one really answers, like, uh, why the word micro comes. Uh, so turns out it has nothing to do with what's running inside the guest.

122. 24:43

     It's, it's everything to do with the VMM itself. So all these new age Rust-based VMMs, uh, they have a much smaller memory footprint because they don't support as many devices, and they also boot much faster because they don't have as much cruft.

123. 24:58

     So a mixture of, like, just less bloo- less bloat and just booting up faster is why the industry has called them, uh, microVMs. Yeah.

124. 25:08

     Uh, one second. Uh, yeah, sorry. Uh, and then you must have seen Firecracker and cloud-hypervisor. I-- There's a lot of confusion around what-- who came before. [chuckles] CrossVM was the first, like, Rust-based VMM that came before, and then Firecracker forked CrossVM, and it's used on Amazon for their Lambda and serverless load.

125. 25:28

     And cloud-hypervisor is a more general, like, VMM that many, many companies, like, contribute to. And when you s- historically see microVMs on the internet, like, it's powered by one of these VMMs, basically.

126. 25:40

     Yeah. So it may seem complicated, but at the end, everything is APIs. Uh, microVMs are no different. Um, so here's a small view of how you can actually start a microVM.

127. 25:53

     So your harness actually forks a cloud-hypervisor binary process. When the process starts, it exposes an API over a Unix domain socket. And if you can see in step two, we call the create API and give it, like, the root FS, kernel, CPU, and memory.

128. 26:09

     And then we finally call start. And when we call start, you can see from previous diagrams that the VMM literally calls into dev KVM just like before, and it starts these, uh, like, guest, uh, microVM.

129. 26:24

     And then once the guest is running, the, like, in this case, the agent sandbox is running, it might be that I want to talk to something inside the sandbox to say that, "Hey, save your state," or I'm attaching some devices, or I'm doing some XYZ operation.

130. 26:38

     So generally in, in agent sandboxes, you have a PID1 that exposes an API server, and your harness or something on out- outside is talking to this, uh, uh, API, uh, server.

131. 26:50

     Uh, and then this is the way how you can use, like, microVMs to be, uh, agent sandboxes, uh, that, that can run, like, pr- more securely than the other primitives, uh, that we showed.

132. 27:01

     Yeah. Uh, in this case, we are using vsoc, which is a socket that can help you communicate between the guest and the host, or you can use the IP stack on the node itself.

133. 27:12

     Yeah. And so there are no free lunches in systems. Uh, and so there are some trade-offs. So obviously you get really, really good isolation using microVMs at a hardware level.

134. 27:24

     Uh, you can still attack the host, but it's much harder. You have to attack the KVM stack, and then you have to attack the device. The chain is much, much harder to do, but it has been seen, and I'm sure it will be seen more with these models.

135. 27:38

     Um, you can jail the devices gran- granularly, so you can use seccomp and other security hardening things for the devices. So even if you get compromised on one device, your whole system cannot be brought down.

136. 27:50

     Um, and like I said, there is a performance overhead. Like, we-- like, when you are ex- exiting and, and entering the host and the guest context, it's a very, very heavy, uh, operation and you pay in, like, performance.

137. 28:03

     Um, memory sharing is not as easy. There's something called a balloon driver, and so you have to actually ask the guest to give back memory and reclaim it. So it's always a, like, reactive thing.

138. 28:14

     You can't immediately, like, reclaim and claim memory. Um, and then a lot of sandboxes these days have GPU access, uh, presumably for auto research or some sort of, like, ML research type of agents.

139. 28:27

     This is not as easy with microVMs. Uh, there's something called VirtIO GPU, but that provides high-level graphic library type, uh, access. Um, for metal acc- direct metal access, there's something called VFIO, but it can only be shared by one sandbox at a time.

140. 28:42

     It cannot have multi-tenant things. But given all of this, my view is [chuckles] that security, like system tricks can cover performance issues, but they cannot hide security breaches. And as a company, you can lose trust once and it's, like, very hard to regain.

141. 28:58

     So I always prefer the more secure solution and try to make, make, make, make up with system tricks for performance issues. And in my history of, like, working on sandboxes, I've seen there are like-- I would call it like the seven stages of grief, the seven stages of sandboxing.

142. 29:14

     Like, in the end, everyone always wants a VM because they tried everything. They tried containers, gVisor, v8s, and then they realize, "Oh, I want a whole Linux box because I don't want to, like, end up without XYZ functionality.

143. 29:28

     And if I want a whole Linux box, I want to obviously be secure." So if you're a startup or a founder, like, in this space, like, let me save you the story and two years of grief.

144. 29:38

     Just please use microVMs from the start. And then if it doesn't work, tell me, and then we can talk about other things. [chuckles] So now we've discussed how to run untrusted code on one node, and I think [clears throat] now, like, people have woken up to the fact that these models are very, very good drivers of Linux boxes.

145. 29:56

     Like, so if you give them a computer, they can just pretty much do magical things, as we've seen with OpenClau. They just pre-train on a lot of Linux, right?

146. 30:04

     However, like, imagine if I gave you a computer without a disk. Every time you close the laptop, like, your data and your work goes away, right? Like, that's not a fun world to live in.

147. 30:15

     And, and somehow the agents are some, in the cloud at least, are in this sort of world right now, right? So we want to give them durable storage. And so this, this part of the presentation is specifically working on disk storage, not memory persistence, but disk persistence.

148. 30:30

     So let's see why it's important and how we can do it. Like-

149. 30:36

     So as shown in previous diagrams, your microVMs have disks attached to them. Uh, they might be just files or other disk devices on the node that you pass through to the VM.

150. 30:46

     Um, and so from a product perspective, the task, the tasks that the users are doing in these sandboxes are becoming much more complicated and much more longer horizon. So people are making like presentations and entire GitHub repos are being created inside the sandbox now.

151. 31:03

     Now, imagine if, like, the node dies on the cloud or the model has a flake, and you created this, like, presentation and, like, you just lost it, right? It's, it's A, bad for us because we wasted a bunch of GPU tokens.

152. 31:15

     It's obviously bad for the user because you did a lot of this work and, and you lost it. So not just from a product perspective, uh, but just from like a good experience and like utilization perspective, we need to have some way to save the disk state of the sandbox.

153. 31:33

     And let's go into like three big use cases, uh, on what persistence can unlock, right?

154. 31:40

     So counterintuitively, persistence actually helps reliability and scale. Uh, they might seem like orthogonal concepts, but, but they're very much related. So, for instance, you have a long-running task in a sandbox that has many packages installed and, and you've created like GitHub repos and presentations and things, right?

155. 31:58

     If you keep checkpointing it periodically, and if the node fails or the cluster fails, you can now restore the sandbox in the exact checkpoint state on another node. You can also do it intentionally if you want to upgrade a cluster or do some AB testing on nodes.

156. 32:13

     The persistence has now let you reliably run sandboxes across your fleet, right? So it's a very good way to scale and be reliable. [clears throat]

157. 32:23

     Like I said, like, how many of you have used goal mode in Codex or know what it is? Yeah. Amazing, right? So if you, if you run goal mode, now it's like, like my-- I think three days is my record for running, uh, uh, something.

158. 32:37

     But, like, now people are doing longer and longer tasks, and I think this trend will continue in the cloud as well. So to support this, we obviously, like, need to have checkpointing so that the model can save state, restore it on another node, and, like, you can keep going forward and forward, right?

159. 32:53

     Like, uh, and so you're resilient to any failures, uh, across in the, in the infrastructure.

160. 33:00

     This is the most interesting part, actually, right? Like, so if your harness wants to explore multiple, like, solutions or sample spaces, it can actually checkpoint the sandbox state, and it can, like, do a Monte Carlo, like, tree search and, like, go ahead and, like, backtrack, checkpoint again.

161. 33:17

     So this way, it can actually do rollouts over many, many days and come back with the actual, like, uh, solution, right? And I think my sincere hope is that if, if we make this infrastructure correct, we can help, like, solve diseases, like find new drugs, because the model can just keep going on for longer and longer, right?

162. 33:35

     But it doesn't happen till we have really, really rock-solid primitives to do this. Yeah.

163. 33:41

     And so given the needs and the, the, the

164. 33:45

     pillars we want to support, here are some of the things that this snapshotting solution should support, right? Uh, first, at ChatGPT or Codex scale, we want to do incremental snapshotting.

165. 33:56

     And what that means is like, if you call snapshot twice, I'm just snapshotting the diff between the two snapshots. Otherwise, if I have to save gigabytes of data at every turn, like, like, I'm gonna bankrupt the company and, uh, like, it's just a slow experience regardless, right?

166. 34:12

     Um, the snapshotting ex-- uh, API itself should be very, very cheap and fast, so the model and the harness can keep snapshotting and exploring, like, very fast. Um, and similarly, like, just like creation should be fast for products, like restoring is just nothing but just re-creation from a snapshot.

167. 34:31

     And so restoring should also be very, very fast for a good product experience. And then we have two paradigms we'll discuss. One is always saving, uh, in which the harness doesn't have to explicitly call a save API versus explic-explicit saving, where the harness is calling save, save, save.

168. 34:50

     We'll see how we can implement both. So I'll give a reference solution, um, yeah.

169. 34:57

     Uh, and then there are some more design choices with disk snapshotting, right? So first, like I said, on the left-hand side, do you want incremental or full snapshots? I argue, I think at our scale, we want incremental snapshotting.

170. 35:09

     Secondly, like, do you want to, uh, snapshot the entire root FS, or do you want to have certain folders like workspace or mount something, something that you want to, uh, you want to keep that, like, configurable?

171. 35:22

     And lastly, like, you-- we can see how you can snapshot at a file system level or at a block. So at-- In Linux, disks are nothing but block devices, and each file maps to different blocks.

172. 35:32

     We'll see after this. And so you can do very, very efficient, like, snapshotting by just zipping up the blocks that have changed for a file. Or you can do entire files that have changed, right, for more write amplification.

173. 35:45

     Uh, and the high level flow of snapshotting is basically you figure out what's changed, you zip it up, put it to the cloud, and then when you restore, you pull it down and, and you restore the microVM, right?

174. 35:59

     And so this is a first principles talk, and so I just wanted to go over how Linux storage works from first principles. So Linux represents disks as block devices.

175. 36:09

     So you see on the right, like, it thinks of a block device as having logical blocks from zero to N. Um, a file system maps directories and files into something called an inode data structure, and the inode says, okay, offset zero in file F maps to this block, logical block in disk D.

176. 36:29

     And then on the disk itself, there's firmware running which says, oh, logical block one is like sector hundred or sector five hundred or page XYZ. So the hierarchy is on these logical blocks up to the file system, uh, and we leverage that for block-based snapshotting.

177. 36:45

     And then within a microVM, there are two ways of accessing storage. On the left-hand side, you can think of this as sharing a folder like Google Drive, but it's very, very inefficient because you're-- the VM is doing file system operations, exiting at every file system operation, which is very inefficient.

178. 37:03

     Uh, on the right-hand side, you actually give a disk-like abstraction at a block. So you give a block device to the, uh, microVM, and it is way more efficient because you can use the caches inside, uh, the guest, and you don't have to exit out as much.

179. 37:17

     You only exit out when you truly need to access the block device and the host has to service you.

180. 37:25

     This is the high level, uh, diagram of how you will have always-on persistence, uh, in disk snapshotting. So if you have two microVMs, uh, they have a block device which it-- They will see this dot image as a block device inside.

181. 37:42

     And as they are writing to the block device, we are writing through to the cloud. So there's some sort of like distributed file system that we mount inside that's giving this, uh, always-on persistence.

182. 37:53

     And like I mentioned, for explicit persistence, uh, we have an actual API called the Save API.

183. 38:01

     Calling the Save API from the harness figures out what, figures out what's changed between the last snapshot. Uh, it bundles up this div into an artifact and returns a snapshot ID.

184. 38:14

     Later on, you can give us this snapshot ID. We figure out the lineage of snapshots that make this snapshot ID, and then we download them one by one and apply it on the node, and you get a microVM restored with this thing.

185. 38:27

     Uh, so let's see like how, how we can implement this, right?

186. 38:32

     Give me one second. I wanna see how we are doing on time.

187. 38:41

     I think we have five minutes left, so we'll go faster. Yeah. So, uh, we can do explicit persistence, uh, using something called Copy-on-Write. So Linux has these XFS file systems, uh, XFS-like file systems.

188. 38:54

     So you can do pretty, pretty much have zero latency copies because you don't change any blocks when you copy. But when you actually change the blocks on a file, that's when you pay the penalty.

189. 39:05

     And so in this, in this design, we have a base dot image, which might be the Codex base image or whatever, like the ChatGPT base image. We create a zero copy on top of it, which is a writable layer.

190. 39:17

     And then when you write to it, now you're changing the blocks in this layer. And then when you want to snapshot, I use something called FIE map, which tells me what blocks have changed and what ranges.

191. 39:28

     I zip that up and store it to the cloud.

192. 39:32

     And I can do something very nifty here. I can actually lie to you while I'm uploading to the cloud, so the snapshot can happen, return very fast as I'm uploading in the background.

193. 39:42

     I don't have to wait till I'm back, uh, till I'm uploading. And then on the other side, like I have this diff, I download this like artifact, I figure out what extents have changed, and then I apply it back on top of the base image, and I start the microVM again.

194. 39:57

     So now we have a restored sandbox with the exact same state at a block level, right?

195. 40:03

     Now, how can we do always-on persistence, right? Like, uh, so this is one way of doing this. I know there's NFS also and other like distributed file systems, but NFS, for instance, isn't as performant, and it's not POSIX-compliant.

196. 40:16

     And I think our models are just very good at anything POSIX-compliant and standard. So you can write a s-- you can write a file system actually on top of a, a GCS or S3 on durable block storage, and you can use something called NBD.

197. 40:30

     So i- within the sandbox, you'll actually see a block device, but inside it will have literally a tiered cache of blocks that are persisted first to an in-cluster cache, and then the in-cluster cache is like writing back to the, uh, uh, block storage, the object storage.

198. 40:46

     So you have this like nice global tiered architecture where you're actually like caching things at the block level and finally inside the microVM and get a very performant, uh, like file system inside.

199. 40:59

     Yeah. So that's the persistence part of the presentation. Uh, uh, and so the one takeaway I want you guys to think about is I think storage is the next unlock here.

200. 41:08

     As you're working on sandboxes, think of what, what all you can snapshot and restore fast to give like this new paradigm to harnesses so they can re-recover from failures and explore, do Monte Carlo like searches.

201. 41:21

     Uh, so we've discussed running things on one node, and of course, uh, if our one node dies, we are done. So we want to be able to run across many, many nodes across the world.

202. 41:30

     So I'm not gonna mention about Kubernetes or other like orchestration things. This is a first principles talk, so we'll lightly hint about the challenges here.

203. 41:39

     So [clears throat] ideally, we want multiple machines to support the runtime we discussed. So we can group nodes into clusters and spread the clusters across the regions. A top-level control plane chooses a cluster using region load and other factors, and it's not very different from the orchestrators you are familiar with.

204. 41:58

     Um, these, like it ideally chooses a cluster close to your ChatGPT cluster, so you can like have fast like, uh, access to the harness. Inside the cluster, there's a scheduler also, and the scheduler tells you which node to pick based on the load and other factors, right?

205. 42:12

     So if nodes are dying or failing, it won't choose that. It will intelligently like route the sandbox to the thing. And again, low latency and reliability remain very, very key north stars for this architecture. [clears throat]

206. 42:26

     And so here, like we can use some microVM features to support low latency creation ideas. So a lot of systems cheat for low latency. They pre-- they pre-warm like sandboxes, and they pick one, which is great.

207. 42:38

     Another way to do this is you can actually take a memory snapshot of a microVM and just in time start it, start it in milliseconds as the request comes.

208. 42:47

     And so you can leverage this like nice microVM property that you can save the guest memory and start from that. And the third one is a hybrid solution. So you can have a warm pool, but as it's growing, you can like grow it from the memory snapshot, so you can get best of both worlds.

209. 43:02

     Like the trade-off for a warm pool is that you're, you're consuming CPU and memory, uh, in idle state. Ideally, you don't want to do that, right? So there's a trade-off between, uh, one and three and two, basically.

210. 43:14

     Yeah. And so here is a way where we can use snapshot r-restore for better orchestration. So remember we discussed that a snapshot can have a lineage of many, many layers.

211. 43:26

     So once you want to restore from a snapshot and you find out you have like, let's say four layers that you want to pull down and that makes the lineage, you can actually like smartly route you to a node which has to download the least amount of stuff.

212. 43:39

     So in this diagram you can see node A has some layers, node C has some layers, but node B has all the layers that you need. So the scheduler then routes you and it, it gives it the highest score and routes it because it has all the snapshot layers.

213. 43:52

     So you can use like snapshot with orchestration to just have faster like, uh, uh, creates and even just more reliable, uh, uh, uh, like orchestration. Yeah. This is the talk, and, uh, hopefully it gives you some design intuition around like sandboxes, why they're important, what's the next unlock, and I, I want to see more of you guys

214. 44:13

     using it in a secure way. Thank you. [audience applauding] [outro music]