Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, Co-founder, Trigger.dev

Talk transcript

1. 0:00

   [upbeat music] How's everyone doing?

2. 0:17

   Good. [chuckles]

3. 0:18

   It's a full room. Look at this thing. [laughs]

4. 0:21

   Um, okay, let's get started. Um, okay. So, here is our, you know, agent. You know, it's got the turn loop, it's got the LLM loop. Now, this, uh, little example sort of works well enough running on your own machine, but what if we want to sort of deploy these to production backends and, you know, run them on

5. 0:39

   our servers? So what do we want them to do, right, when they run on our servers? We want them to do, you know, long-running, meaningful work. Uh, should be durable across turns and, and versions of our code, and it should be able to, you know, recover from errors.

6. 0:56

   So I'm Eric, I'm one of the founders of Trigger.dev, and we've been sort of trying to make it easy to deploy these types of agents to production for the last few years.

7. 1:05

   Um, what I like about this, uh, [chuckles] little meme here is which one is the agent and which one is the human? I have to think. [laughs]

8. 1:12

   Um, uh, yeah. So, uh, this talk is sort of about, like, the fundamental shift that agents are, like, posing to backend infrastructure, and some of the ideas for sort of how to achieve these durable agents.

9. 1:25

   So before we go into that, I wanna do a little history lesson here. Um, let's take a step back and see sort of how we got here. So the very first dynamic web backend was CGI back in 1993.

10. 1:36

    Anyone here ever done CGI stuff? [laughs] Cool. Thanks.

11. 1:42

    Uh, so the model was really simple. Uh, a HTTP, HTTP request comes in, the server forks a whole new process, request data goes in, the process does some stuff, and then it writes the response to standard out, and then the process goes away.

12. 1:55

    So it's completely stateless. Um, shortly after that, uh, PHP came out, which sort of turned into the LAMP stack. Um, and... Oops. Um, so sort of, uh, the LAMP stack sort of reused the PHP process, right?

13. 2:12

    Um, but it kept sort of the principle that, like, all you needed to do to create a response was the request, some state from the database, um, and then it would do the request.

14. 2:21

    So the second request would come in, and it would do all the same work again, and it would produce the response. So this is sort of request plus DB equals the response.

15. 2:30

    This sort of became known as the shared nothing architecture, right?

16. 2:35

    So looking at another way, shared nothing sort of means that the compute layer is stateless, right? There's nothing, uh, there's no s- meaningful state, like, in the compute. The state is in the, in the database, right?

17. 2:46

    So this became the dominant backend infrastructure for the last thirty years, right? Everything that followed from this, like, uh, Ruby on Rails, Node.js, serverless, it all follows the same paradigm.

18. 2:58

    Um, as web applications became more, uh, you know, complicated and sophisticated, they started performing these, like, sort of side effects outside of the request in DB life cycle. Um, these side effects are async tasks.

19. 3:11

    So they s- you know, started out simple. Uh, send an email, charge a credit card, you know, resize an image. But soon, they became sort of these, like, multi-step side effects, right?

20. 3:20

    That, like, this process order example here, um, where you sort of do things in sequence, right?

21. 3:27

    Uh, you'd quickly run into a problem, how to handle failures in something like this, right? So if send receipt fails, uh, you can't just retry the whole process order thing again, um, without charging the credit card again twice.

22. 3:40

    That's just bad. So about ten to fifteen years ago, workflow and durable execution engines were sort of adopted to solve this problem, right? So you'd write your code like this now, where you sort of, uh, wrap every single side effect in, like, a step that becomes cached as it's, uh, executed.

23. 3:56

    So now it, you know, solves the problem nicely. When you call process order for the second time, uh, you skip the things you've already done, and then you do the thing that you want to do originally, right?

24. 4:06

    And you don't charge the credit card twice. So this is, uh, I call this model sort of the replay model. Um, so it builds durable execution on top of existing, like, stateless compute architecture, which is, I covered was the, you know, that's how everything works, right?

25. 4:19

    So, you know, you get this nice side effect of you get this, uh, execution history, this audit trail of everything that happened. Um, and also by being able to sort of resume to a specific point in time, you can, yeah, you can re- recover from a failure for that, but you can also, like, wait for something else

26. 4:33

    to happen, right? Um, so you can wait for, like, a human to do something, and then you can resume execution.

27. 4:40

    Um, some of the downsides of this replay system is, like, because now you sort of have to wrap everything in, in these steps, and e- everything outside of steps is de- has to be deterministic.

28. 4:49

    You kinda get this, like, uh, rigid structure. You have to write your code in a certain way or things break. Um, and also, like, replay journaling, uh, re- the replay journal versioning is, is kinda tricky if you deploy a new version.

29. 5:02

    So this is sort of the very simple and truncated history of, like, sort of the state of the world in twenty twenty-three when LLMs came out. Um,

30. 5:11

    at first, they really fit neatly into this paradigm, right? They would just become another step in a workflow, right? Um, they would, uh, they would classify some text or something.

31. 5:19

    But it was still in this old workflow era, right? Uh, not long after that, we sort of got tool calling and tool calling got good, and we were sort of introduced to the agent loop, right?

32. 5:30

    The big difference there is code is sort of no longer orchestrating the LLM. LLM sort of orchestrates the code, right?

33. 5:37

    So we're back at our agent loop, right? And, like, what happens if we, uh... Yeah, you can see that. Um, if you can, if basically we wanna, you know, make this agent loop durable, and c- can we do it with this replay model, right?

34. 5:49

    What does that look like? Um, so what does that look like? Every LLM call, right, becomes a, a step in, in the replay, uh, journal. Uh, every tool call becomes a step.

35. 6:01

    Uh, on resume, you know, the function re-executes on top and sort of replays all that stuff, right? Um, so after a single turn of the LLM not doing too much, you know, this is sort of what the, uh, the replay log looks like, right?

36. 6:16

    And as you sort of keep interacting with the agent, the log grows and grows and grows. At a certain point, you might hit into some sort of like fundamental limit of your replay system.

37. 6:26

    Um, that could be there are like too many actual entries, or it could be like the entries get, grow too large. Um, but yeah, th- this sort of, kind of falls over once you hit that limit.

38. 6:37

    And sort of, uh, there's this measure of like how long agents, uh, can actually do meaningful work, and apparently it's doubling every four to seven months. So right now we're on about like a few hours, but like not too long from now we'll be on like multiple days of length as these agents build to actually do meaningful

39. 6:56

    work. So, you know, replay gave us these like sort of durable transactions. But, you know, an agent isn't like a transaction. It's like a session, right? And it lasts like as long as the user wants it to last.

40. 7:08

    Uh, multi-step workflows are sort of start and end, and sessions keep going for as long as possible.

41. 7:15

    So if we sort of take a step back and think about what an agent needs to be durable like from first principles, um, I think of it as like an agent sort of has these two halves, right?

42. 7:27

    Um, the first half is the context. So this is all, all your system messages, user messages, tool calls, tool results, assistant responses, right? So this is all like the actual context, everything that went in and out of the LLM.

43. 7:39

    Um, so this is extremely valuable. Obviously, you wanna make that durable, right?

44. 7:45

    Um, but you also have this sort of execution layer, and as agents are, like, more complicated, doing more things, they kinda want a machine, right? They wanna be able to do stuff like they could do on your laptop, right?

45. 7:55

    They wanna be able to write files, use memory, like create sub-processes. And so I, I, w- I think of both of these, um, as super valuable pieces of state, but they can be treated separately.

46. 8:07

    So the context is first and the most important. It's, it's just an append-only log of sort of everything that happened, right, like I said.

47. 8:14

    Um, and you can make this log durable using any sort of like primitive that already exists, like a database, object storage, like distributed file system. You know, there's a ton of like technologies that are coming out that, that are specialized in making this sort of thing durable, right?

48. 8:31

    And when that is durable, you've, y- when that context log is saved somewhere, now you can have durability across versions of your code, right? So you upgrade your harness, and you can still use that same context, right?

49. 8:42

    Um, maybe the machine crashes, and you can still-- that is saved somewhere, so you can pick up where you left off, right? Um, and append-only logs scale really well.

50. 8:54

    Uh, but what about making this sort of execution side durable, right? Uh, for these, you know, I was saying the types of agents right now that are doing meaningful work, we-- there's a lot of state that happens in the compute layer that we might wanna save.

51. 9:06

    Maybe you've cloned a GitHub repo, you know, you've installed some packages, you've got some datasets in memory, you're running a dev server, right? You sandbox in a subprocess, whatever it is, right?

52. 9:15

    You can't really make that, uh, durable using a log.

53. 9:19

    And how do we get this to work, right? So you, you have to wait for some amount of time for the next user message, right? And we can't just keep the machine running.

54. 9:29

    It'd be nice, but we can't. It'd be too expensive.

55. 9:33

    So instead of recreating the execution state from a log, we should use snapshot and restore. So this allows us to snapshot the machine, shut it down, save it to disk, and then when the user message comes in, we just restore it, right?

56. 9:47

    So this gives us durability across turns. So when the user goes to lunch, right, we don't have to run the machine the whole time. Uh, it allows us to preserve everything that the agent was doing.

57. 9:59

    Uh, and you know, effectively, compared to running the machine, um, live, it's pretty cheap.

58. 10:06

    So I think if you combine these two things, then you sort of get a durable agent, right? Um, you've got the context, so you're sort of, uh, yeah, you-- context durability and execution durability, right?

59. 10:20

    Um, and this also allows you to cov- recover from errors. So one of the whole, whole points of having these, like, durability guarantees is to recover, right? And so it depends on what happened, what went wrong, and you can cov- recover in different ways.

60. 10:32

    So say the LLM isn't working for some reason. That never happens, but you never know. It could happen. Um, and it takes a long time to like retry. Maybe it says like, "Wait, uh, wait, you know, fifteen minutes so you retry your next message."

61. 10:44

    Well, you don't wanna wait in memory, so you snapshot, and then you restore when you can retry. But if there's something wrong with the machine, uh, maybe you've like shipped a bug, or maybe there's just an issue with the machine, right?

62. 10:56

    But it crashes. You have the context log, and you can recover that.

63. 11:02

    So I think, you know, for thirty years, we sort of had this, uh, stateless compute as the sort of core of backend infrastructure, and I think agents are sort of forcing this, uh, move to become stateful compute.

64. 11:16

    So, and sort of at the heart of that, I think, is, is gonna have to be this snapshot and restore, uh, capability. Um, but sort of, you know, this isn't actually new.

65. 11:27

    Um, this is an IBM mainframe from nineteen sixty-six, and it actually has checkpoint and, and restore. Um, 'cause they would run these super expensive jobs for hours and, you know, if something ha- went wrong and they c- they couldn't afford to run it all again, so they would add these like checkpoints into their code, right?

66. 11:51

    Fast-forward to twenty eleven, a thing called CRIU was, um, developed. It was a way to like suspend and restore s- uh, a process like from user space. So it would basically like inject a process with this like a parasite basically, and then they would force the process to like dump everything to memory, and then it would remove

67. 12:09

    all the traces of the parasite, and it actually worked. Um, in twenty twenty-four, we actually shipped this, um- And we've done millions of, uh, snapshot restores since. You know, it's transparent for the process, so the process doesn't have to, like, participate in it, and it's compatible with container runtimes, which is good.

68. 12:27

    So the downsides are you sort of can only checkpoint, like, a process. So if you're doing stuff with, like, FFmpeg or, like, you've got a Chrome instance running or anything else, right, it sort of doesn't work.

69. 12:38

    Uh, it only captures open files, so if you're working with the file system, it has to be open at the time of snapshot or you won't get a snapshot.

70. 12:45

    And then also if you [laughs] ... It, it, yeah, it's nice that it's compatible with containers, but once you are compatible with containers, you have to work with registries and push and pull, and it gets very slow.

71. 12:56

    Uh, so last year we moved to, um, Firecracker microVMs, and this allows us to sort of snapshot, like, the entire machine, right? So everything that's just on a machine, on a VM, we can snapshot it, and then we can restore it, and it pick up right where it left off, no matter what was happening in the machine,

72. 13:12

    right? But if you do that sort of, uh, in a naive way, uh, it can be quite expensive. So say you have a def- a default machine size of 512 megabytes, you know.

73. 13:25

    Uh, if you do a snapshot, it's 512 megabytes on disk, so that's, that's not great. Um, so obviously you've got, like, networ- network transfer costs, you've got storage costs, and there's a lot of memory there that's not actually being used.

74. 13:38

    Uh, so we actually solved this with, uh, compressing it. Uh, we actually use a, a seekable compression. Um, so when we restore, we actually on- don't restore all the memory pages at once.

75. 13:51

    We actually, like, capture when it needs to be restored and just, like, com- decompress, like, that little bit that needs to be, um, restored at time. We also have a couple other techniques for layering the snapshot, and we can get the, the, um, snapshot down to, like, 14 megabytes compressed, and that's sort of like a knob you

76. 14:08

    can tweak and, um, depending on how perform... what kind of performance you want, um, you can compress more or less.

77. 14:16

    Um, so that's pretty much all we had to do other than all that. [laughs]

78. 14:22

    Um, and once we did that, uh, we, we got super fast snapshot and restore times. So this is a sort of a stupid graph, uh, comparing [laughs] Kreon and Firecracker, but it's basically the, the moral of the story is that snapshots are, like, slightly under a second and restores are a couple hundred milliseconds.

79. 14:40

    Um, we've actually bundled all of this into a tool that's gonna be open source here soon. Uh, it's called FCRun or F-Run, depending on who you ask. Um, so this allows you...

80. 14:52

    It's like a Docker-like CLI, so you can drop in replacement for, like, the Docker command, um, for running containers in, in Firecracker VMs and snapshotting and restoring them.

81. 15:01

    So for example, um, you can run Alpine and it's super fast, and you can snapshot a running VM and it's super fast. You can, like, fork a VM, also very fast.

82. 15:15

    Um, this is a little benchmark for TTI, so basically how long it takes the VM to become, uh, interactable with the internet. So this is, uh, we're doing, like, 15,000 VM starts per minute.

83. 15:29

    Um, you can almost render, uh, like, a video. The F- the FPS would be about 30 FPS. Um, so it's, it's extremely, extremely fast. Um, so this is gonna be powering sort of our future, like, compute layer, but it's open source.

84. 15:45

    Um, not yet, but very soon. Um, so kind of back to where we started with our little agent loop here,

85. 15:55

    and, um, we've sort of made it durable now by doing two different things: context log and execution snapshots. So we get durability across versions, durability across turns, across failures, and, uh, so this will lead to a future of, you know, stateful compute.

86. 16:12

    So that's it. Yeah. [audience applauds] [upbeat music]