Stateful Agents — Full Workshop with [REDACTED:username] Packer of Letta and MemGPT

Talk transcript

1. 0:10

   So today I'm gonna be going over a workshop, so there is, like, an interactive component. I think you can probably get a lot out of this talk just by watching, but if you do want to participate, um, you're gonna probably wanna install Docker on your laptop if you don't already have it, and then you're gonna wanna pull

2. 0:24

   this Docker image. Uh, if you do this, it should be very easy to follow along. There's also a notebook in the channel, it's called, I think, like, Workshops Letta, and you can just hop over to the-- I think I also made, like, a tiny URL for it.

3. 0:38

   Um, but if you hop over to this link, this is, like, the notebook I'll be running over for maybe, like, the first thirty minutes. And this notebook is kind of like a client, and the client needs to run against the server, and the Docker image is what would be the server in your case.

4. 0:50

   Um, so again, I'll leave this up for a few seconds. This is basically where you're gonna go get the notebook.

5. 1:00

   And then-- And I believe this is also in the, in the Slack channel.

6. 1:06

   And then, yeah, the main thing you're gonna wanna do if you wanna follow along and you have Docker installed is, like, do Docker pull. Um, I guess, like, a quick show of hands, like, how, how many people here have Docker installed on their laptops?

7. 1:16

   So, uh, okay, sweet. Yeah, most of us. Okay, great. Um,

8. 1:23

   okay. Does anyone need a few minutes or anything? Um, I guess the people who do want to kind of follow along, just make sure to check out the workshop channel.

9. 1:30

   Um, and of course, this is recorded, and then all the materials will be online, so it'll be very easy to kind of follow along later as well.

10. 1:37

    Yeah, so I believe the title of this workshop was, um, like, Agent Memory: The LLMOS. Uh, I think maybe a better title for the workshop is Stateful Agents. So I think you've probably heard a lot about agents, um, over the course of this conference.

11. 1:51

    As-- I also would like to poll the room, like, how many of you, if you think you were asked today to give, like, a bit-- a concrete definition of an agent, would be, like, pretty com-comfortable in what you, what your definition is?

12. 2:01

    Okay. Yeah. So despite hearing, you know, a l- a lot about agents over multiple days, it's still pretty hard to define. I think this is kind of a problem that's become much worse over the past year, and I think that's why it's kind of useful to maybe, like, sharpen our definition of agents.

13. 2:16

    So I like this term stateful agent. Um, I think stateful agent actually is kind of what agent meant before, you know, the LLM era. And I think these days, I think a, a very common definition for agents is it's like an LLM that's taking actions in a loop, right?

14. 2:30

    And I think that kind of, that kind of works, but it also misses a really big, um, part of that process, which is that when you run the loop, it's a closed loop and the agent gets updated.

15. 2:41

    And it turns out that, you know, in this new wave of AI, the fundamental unit of compute we're using for AI is stateless. It's not, you know, a, a recurrent neural network.

16. 2:50

    It's not an SSM. It's a transformer. And a transformer inherently is like a stateless machine, so that means that when you close the loop, you have to have some sort of mechanism for updating the state.

17. 3:00

    And, you know, traditionally, like, when agent was defined back in, like, the RL days or, like, back, you know, before P-LLMs, I don't think this really was that big of a distinction to make, like, stateful versus not stateful agents.

18. 3:12

    But I think because LLMs are stateless and that's, like, what everyone is using for A- for AI now, um, it's a pretty important distinction. So hopefully by the end of this talk you kind of understand what stateful agents mean and you probably agr-- hopefully agree with me that, um, statefulness or memory is actually probably the most important

19. 3:26

    thing to solve if we actually want to get, you know, if you want one twenty-five be the year of agents, or if you want agents to actually deliver around any of the hype.

20. 3:37

    Yeah. So when I talk about statefulness, I think statefulness is pretty synonymous with memory. Um, and I think that's because LLMs effectively have no memory. They just have the memory that's in their weights, and then they have what's in the context window.

21. 3:49

    Um, so, like, memory and state context, these are all kind of synonyms with LLM AI or, like, LLM-driven AI. And humans are, of course, stateful. So humans, they form new memories, they learn over time, and LLMs don't.

22. 4:01

    So any of the learning you're gonna do is gonna have to be done by you, the user of the LLM, or by the framework. Um, but it has to be done by somebody.

23. 4:09

    And traditionally, you know, in the past few years, I think by default, this just means appending to a list, right? And I think for workflows and, like, stuff we were playing around with from twenty twenty-two to twenty twenty-four when not a lot of people necessarily cared about, like, making money, um, or doing useful things with agents, that

24. 4:22

    was okay. But I think it becomes a very big problem when the only mechanism you have to handle state when you're actually trying to use agents to do useful things is kind of appending to some sort of list and a list that's, like, held in Python process memory or, you know, in, uh, like a Node.js process.

25. 4:40

    So the, the natural question is, if we have LLMs and LLMs are stateless, and basically the main thing we want to get out of state is we want to have human-like intelligence, that means learning.

26. 4:50

    So how do we actually do that? Um, is how many people here have, like, heard of MemGPT? Um,

27. 4:56

    okay. Yeah. So before I was, you know, doing a startup, Letta, I was a PhD student at Berkeley, and, you know, the, uh, ChatGPT stuff happened in the middle of my PhD, which is kind of like, um, interesting because it meant that, like, most people doing research kind of pivoted their research entirely.

28. 5:09

    Um, so in the few years of my PhD post-ChatGPT, I was really focused on memory and agents, and I wrote this paper called MemGPT with a bunch of my colleagues and some of my o-coworkers now at Letta.

29. 5:20

    Um, and this is really talking about, like, a memory management system for LLMs. So if LLMs can't-- if they need some sort of memory management and the de facto way we do this is, like, a human appends to a list, well, if you believe LLMs are gonna get better and better and better, shouldn't this be done by

30. 5:35

    another LLM? Like, shouldn't the AI do the management for the-- Shouldn't AI do the memory management for the AI? So when we called, like, this a me-- an LLMOS, we were really referring to, like, a memory management system.

31. 5:47

    And this is effectively, like, a very high-level graph or a high-level figure of what I, what I mean by, like, statelessness versus statefulness. On the left is basically what most people do.

32. 5:58

    Um, you know, you kind of have a context window. The context window is pretty loosely defined. It doesn't necessarily- Uh, it's not broken up into very distinct pieces. It's not tied to any state in a database per se.

33. 6:07

    This is just something that's held in process memory, and you're, like, appending to it over time. Um, this is what happens if you use LangChain, like, in the LangChain buffer memory, and you're like CrewAI is look at it and get buried really deep in the code base.

34. 6:17

    And this is actually, you know, often why you need tracing software or, like, observability. Observability kind of exists as a category because we have this black box of tokens that just gets shoved into the LLM.

35. 6:26

    And statefulness or, like, staple agents, and the, the key thing we're trying to do is understand that, you know, what's in the context window, if we have a machine assembling it, there's some-- that means there's some sort of, like, context compilation problem, where there's an optimal way to arrange the context window of an LLM, and that context

36. 6:44

    comes from state. The state can be kind of, uh, very large, much more than can ever fit in the context window. And this is basically what you do if you're a power user of ChatGPT or a power user of Claude, you're doing this yourself, right?

37. 6:56

    You're kind of like con-compiling the context every time you start a new chat. Because I think a lot of us, you know, probably have experienced, like, having a chat that goes, like, way too long, it starts to derail, and then you have to redescribe everything you were doing back to ChatGPT.

38. 7:08

    It's, like, a pretty painful experience. Um, so can we basically automate this with a machine?

39. 7:14

    So yeah, uh, taking a step back, like, why do we even want staple agents? Like, why are agents we have today not enough? Like, why is the e-- why is the current, like, paradigm of LLM-driven agents, like, not good enough to reach, let's say, like, AGI or, like, some, like, AI we see in science fiction?

40. 7:30

    I think the main problem is that agents we have today, they just can't learn from experience. Um, or the way they learn from experience is extremely limited. And I think

41. 7:38

    if you're just running, like, workflows, um, you might not notice this, so you might-- it might not really be a big problem. But I think if you're trying to build, like, assistants, companions, co-pilots, this becomes, like, pretty evident.

42. 7:50

    Um, so imagine if you have some sort of chatbot. So this is a, a direct screenshot from, like, the MemGPT paper. Um, but you have an AI and a user, and the AI, you know, it say-- it can see that it's February fourteenth, so it wants to, like, ask the user what they're gonna be doing on Valentine's

43. 8:04

    Day because they have stored in their memory some recollection that the user's boyfriend is James. So the, the AI asks like, "Oh, hey, how's James doing? Do you have any special plans today?"

44. 8:14

    And, you know, the user says, "Actually, you know, James and I broke up." Um, so it's obviously a very, very bad mistake by the AI, but maybe it's, like, kind of unavoidable.

45. 8:22

    Um, but you see the AI here, if it has some notion of, like, a permanent read/write store, it should do something like transact that, you know, James is no longer the boyfriend, James is the ex-boyfriend.

46. 8:34

    Um, and if you don't do this kind of thing, and you run your agents for long enough, and you just do, like, recursive summarization, or if you just have, like, a really long context model, you're inevitably gonna make some mistake where you tell, you know, you tell the user something about your boyfriend James, and that's, like, a

47. 8:48

    devastating error if you're trying to build, like, a consumer app, right? Um, and it's also something that just a human would never do. I think, like, humans, something like this would get, like, written to your quote-unquote, "core memory" very aggressively.

48. 8:59

    I think the other reason you want statefulness and learning, um, I think many of us here maybe, like, kind of work companies, and we're trying to, like, use agents to make money or, like, drive value, you know, for shareholders or whatever.

49. 9:10

    Um, but at com... the big difference between consumer and enterprise is obviously data. Like, enter-enterprises have much more data than could ever fit into, like, Gemini ten million tokens, right?

50. 9:20

    Often, like, per user, you have more than ten million tokens. So how do you actually, like, learn from that data? So you, you-- you can kind of think of there being, like, another training phase after the post-training-- like, during post-training, right?

51. 9:32

    You train the model, and then now the model is deployed into your enterprise, and the model should kind of learn about your company, right? And that's kind of like training again, but it's not training into the weights, it's training, like, into the in-context memory.

52. 9:44

    Um, so staple agents naturally kind of, like, encompasses this concept. And this is, like, pretty useful if you're trying to build this stu-this kind of stuff inside of companies.

53. 9:53

    And yeah, again, like, really, the reason I was kind of inspired to work on, like, the MemGPT stuff, and also the reason that I'm really excited to be doing the work I'm doing at Letta, is because I'm very interested in just AI and making AI that is very human-like.

54. 10:07

    And clearly, there is some very serious deficiencies with the current it-iteration of LLM agents. Um, they're not very human-like, especially with respect to how they deal with memory.

55. 10:17

    Yeah, so I covered this already. I think this is, like, a tweet I saw a few weeks ago or something. I thought it was pretty funny. Um, but I think, you know, if you all use ChatGPT in this room here, it's probably, like, a pretty universal experience.

56. 10:26

    You kind of have a, a conversation that goes on too long, and then, you know, you feel very devastated because you're now gonna have to mentally context compile again to, like, redescribe to ChatGPT what's going on, or, like, Claude or whatever you're using.

57. 10:40

    So the promise of staple agents, um, there's actually a lot of promise just in product, right? I think if you actually have true statefulness, then that means that, you know, this sort of experience shouldn't happen.

58. 10:49

    There shouldn't be any derailment. So you shouldn't have, you know, like a Claude instance going haywire. Um, and actually, you should have the opposite happening. Like, the, the experience should get better and better and better over time as the AI kind of learns more and more about you.

59. 11:02

    And this is, you know, kind of like maybe the promise of ChatGPT memory. Um, but, you know, I, I don't think any of that stuff is really working that well.

60. 11:08

    And of course, I think when you don't-- when you no longer just, like, shove stuff into a context window, when instead you're kind of really, like, creating me-- human-like memory constructs, uh, I think the behavior of the agents just becomes more human-like.

61. 11:22

    You have the same kind of, like, fuzzy memory that a human has. You have the same kind of forgetfulness, but you also have the same kind of recall.

62. 11:29

    So in today's workshop, I think it's kind of gonna be split into two parts. I think the main thing we're doing at the front is a notebook. And of course, like, you don't-- you can just watch.

63. 11:37

    Um, but if you want to, you can follow along in the actual I-- the, uh, the Jupyter notebook file. And the goal of that notebook will just be to, like, lay out the basic ideas, um, behind this sort of, like, context management system in an LLM.

64. 11:51

    And at a very, very high level, you know, if I had to describe it in one sentence, it's just that you make an LLM, either the main LLM or, like, the subconscious memory system LLM, just aware of the context problem.

65. 12:01

    You kind of can just describe to it in English, a, you are an LLM, and you have a context window of, like, one hundred to twenty-eight K tokens, and I will let you know when you reach a hundred K tokens, and you're gonna have to, like, manage memory with these tools.

66. 12:12

    So it's all centered around tool calling. Uh, and I think centering around tool calling is actually very effective because LLMs are getting better and better and better at tool calling.

67. 12:18

    Um- And the second part of this workshop is going to be, um, kinda like building these th- b- building the staple agents in, um, the Letta framework, and then also like the front-end ADE.

68. 12:30

    So I think if you're very interested in kind of like no-code stuff, I think this is pretty interesting because they-- we're all very, very familiar with the paradigm of the playground, right?

69. 12:40

    It's like ubiquitous. Every single AI company has their own playground. But I think once statefulness and like staple agents become the dominant paradigm, which I'm pretty sure they will, um, then I think you're gonna ha-- there's gonna be a new iteration of the playground.

70. 12:54

    And th-the ADE that we built, I think that's our best guess as to what that experience will look like. But I'm sure, like everyone in this room who's worked very-- who has like spent a ton of hours with LLMs, you probably have a very strong opinion too on what that experience will be like.

71. 13:06

    Like what happens when, you know, ChatGPT is no longer-- like it, it no longer has like a history bar on the side, and instead it's like a unified experience.

72. 13:15

    Like what is the correct UX for that? What's the correct DX? Um, and then if it's not a consumer app like ChatGPT, if it's a devel-developer tool, like what do you as a developer want to see from your agent?

73. 13:24

    Like, you probably wanna see all the way down to the context window.

74. 13:29

    So I said this at the beginning, I don't think any-anyone necessarily walked in, uh, midway. So if you're going to follow along, you're gonna wanna download Docker on your computer.

75. 13:36

    I think most of the people in this room have Docker on their computer. You can do this with Pip, but Pip is pretty terrible at package management, so you might like hit a bug.

76. 13:43

    Um, and yeah, I already covered this, but if you're interested in any of the ideas that I'm going over in the initial notebook, they're basically just strongly distilled versions of the ideas from the MemGPT paper, um, like reduced to their like most simple, um, yeah, their, their most simple components.

77. 13:59

    But if you're like interested in this more in like a research capacity, you can also just read this paper.

78. 14:06

    Okay. And yeah, we're gonna be doing this in Python. Um, I guess like how many people in this room have used Python as their main language versus... Okay. And I imagine like that everyone else is TypeScript or something.

79. 14:20

    Yeah. So basically what we're gonna be doing is Python, but Letta the, um, server is not-- oh, maybe Go or [laughs] Rust. Okay, yeah. Yeah. So we, we don't have a Go or Rust SDK, but we have a TypeScript SDK, and then we also have a REST API.

80. 14:34

    And actually our Python SDK and TypeScript SDK are just like programmatically generated off the REST API.

81. 14:39

    Okay. So I-I left this on the screen a little bit earlier, um, to kinda like preempt any pauses here. So I, I might just like continue on. Um, and yeah, these are-- these links are all in the channel, like workshops-letta on the, the shared Slack.

82. 14:58

    Okay. Yeah. Let me hop over. Oh, and I think we-- because this workshop has a reasonable amount of time allotted to it, um, I'm very open to like pausing for questions and stuff.

83. 15:11

    So if anyone has any questions, please just raise your hand, and I can pause. And then if we're just going too slow, we can fix that later.

84. 15:19

    Yep. Okay. Questions. Um, all right. So yeah, I guess how many people are actually gonna attempt to follow along?

85. 15:37

    It'll be useful for reference. Okay, a decent amount of people. So I'll try to, yeah, definitely, uh, raise your hand or something if something's going terribly wrong and you're not able to follow.

86. 15:47

    Um, let me boost the... Okay. Is-- Let me know when the font is large enough. Is this large enough for the people in the back to, to read?

87. 15:56

    Okay, great. And... All right. Okay. So if you ran the-- if you did the Docker pull, um, the next thing you're gonna wanna do to be-- to

88. 16:21

    start this entire, uh, notebook session is you're gonna wanna start the Letta server. So if you were using Docker, the way you're gonna do this is run this docker run command.

89. 16:28

    I'm realizing now that this is not-- Oh, this is, this is in-- You can copy-paste this out of the GitHub, um, the GitHub link, but I'll also post it in the Slack.

90. 16:38

    Yeah. And then if there is a free endpoint that's live, so you don't actually need to have these keys. Um, so you can just run this and it will-- you'll be able to run the demo fine.

91. 17:02

    Okay. And then once you run it, you'll basically see the server kick off in the background.

92. 17:09

    Yeah, so while people-- I'll, I'll give everyone like a minute to do that, and while everyone is kinda kicking off their servers,

93. 17:17

    might also just show you kind of a diagram of what's going on here.

94. 17:24

    Yeah. So basically the way this whole stack works is Letta is like, it's an open source stack. Um, it's basically FastAPI, Postgres, and Python, um, in the middle, like Python logic.

95. 17:36

    So in that Docker container, like it's exposing an API that's actually like very, you know, pretty robustly documented. You can like look at our API reference here. Um, and the API is how you interact with all your agents.

96. 17:48

    So if you're using the Python SDK, it actually just, you know, goes over the, the REST API, and you basically build your agent applications on top of this API.

97. 17:57

    So this actually looks very similar to like the chat completions API. It's just like s-session-based, right? Uh, so you don't have to provide the entire, the entire conversation history every single time you interact with an agent.

98. 18:09

    So we'll see that kind of immediately in the notebook as well. Okay. So I, I assume everybody who's following along has gotten the, the thing to launch. Um, is there...

99. 18:18

    Okay. What does it say?

100. 18:19

     It says that there, there's a directory that doesn't exist. Do I need to create that?

101. 18:24

     Um- Oh, yeah. Can you do, like, a kind of make and KDRR on that directory?

102. 18:30

     Okay.

103. 18:32

     Did anyone else get that directory doesn't exist problem?

104. 18:34

     Can you send the, the Docker command to the channel again? Because I, like, now I just can see it.

105. 18:39

     Okay. Do you-- So are you able to see this here?

106. 18:44

     Oh, no, we did one. Yeah.

107. 18:46

     Okay, yeah. So it's-- Let me...

108. 18:49

     Is there a chance? Yeah, that fixed it.

109. 18:58

     Okay, great. All right, any other problems? Does everyone have a server idling now on their laptop? Great.

110. 19:05

     Okay, so now that we started the server, um, now we can kinda, like, start running the notebook. So this part is basically

111. 19:14

     you're gonna go to this, go to this channel, click on this link. You're gonna see a README, and the README has all the commands. So I'll, I'll kinda, like, go one by one and paste these commands into the workshop channel.

112. 19:25

     Which channel is workshop then?

113. 19:27

     It's, uh, workshop-letta. Yeah. Okay, so you need to basically clone the repo.

114. 19:40

     Once you clone the repo, just CD into the specific example.

115. 19:52

     Now, I'm, I'm assuming everyone's using Talker, so I'm just gonna kind of ignore that. Um,

116. 20:00

     okay. And ideally, everyone has Python installed on their laptop, hopefully.

117. 20:29

     Yeah, and then once you run that Jupyter Notebook command, um, all you need to do is just double-click on the notebook here, if you're not familiar with, um, Jupyter.

118. 20:39

     Uh, it should automatically, like, open a browser tab that should have this, and then you just, like, double-click, and it'll open the notebook.

119. 20:50

     Okay, so among the people who are following, who is a little still not to this notebook stage yet?

120. 20:57

     Okay, yeah.

121. 20:58

     I have to install Jupyter, so it's taking a while.

122. 21:00

     Oh, okay, yeah. No problem. Okay, no other issues? N-nothing on fire yet?

123. 21:09

     Okay, cool. All right, how's that Jupyter looking?

124. 21:27

     I installed it.

125. 21:29

     Okay. Okay, so everyone has this notebook up who is intending to follow along? Great. All right, so yeah, if you're not familiar with notebooks, you basically just execute them cell by cell.

126. 21:42

     Um, I mean, number-- the first thing you're gonna have to do here is, like, import the client. So again, this is a little bit different from some other frameworks you're familiar with.

127. 21:52

     Like, I assume everyone here has, like, used LangChain before probably or heard of it, um, well, like CrewAI, AutoGen, um, Pydantic, uh, AI, things like that. So I think among those frameworks, there's a pretty big distinction with Letta, in that Letta is like a server-client process.

128. 22:08

     And a big part of that is because the agents are intended to be stateful and, like, persist indefinitely. So it's very hard to, like, persist things indefinitely if you're kind of holding everything in application state, as opposed to having, like, a server that's a centralized source of truth.

129. 22:21

     So we basically run the server on the left. Um, we connect to the client. You know, you can actually park the server anywhere you want because it's a Docker image.

130. 22:27

     You can, like, run this in the cloud. It's very easy to, like, drop it on Kubernetes or something. In this case, it's running on our laptop, so we just need a local host and the default port.

131. 22:35

     So the first thing we're gonna do is create an agent. So I wanna make sure that everyone can, like, execute this cell, so I'm gonna execute it one more time.

132. 22:43

     And you saw, like, my server kind of fired off on my left. Um,

133. 22:49

     yeah. So I'll, I'll kinda walk everyone through the code here. So basically, you know, with chat completions, which I assume everyone's familiar with, the paradigm is you create the agent in memory, and then you kinda, like, pass the whole agent off to the server and ask, like, you know, complete the state, complete, like, this one more message

134. 23:07

     in my, in my message history. Instead in Letta, we, we first create the agent, and then once we, we create the agent, we have a handle for it. And once we have that handle, we can kinda send it messages, and we never have to send it the full state.

135. 23:17

     We don't have to track that anymore. We just send individual messages. So

136. 23:23

     the, the main components of memory in a Letta agent is just memory blocks. So these are just strings. Um, in this case, you know, we're doing something pretty simple and stupid, like the human's name is [REDACTED:username] the [REDACTED:username], um, and then the persona for the agent, uh, we're just using that my name is Sam, um, this AI.

137. 23:40

     And this is all arbitrary. The only things that these handles are useful for, actually, is for the agent to edit these own handles or edit the values in these handles.

138. 23:48

     So as we'll be able to see later in the notebook, the agent can actually go in and, like, rewrite its own memory. So if the agent, you know, decides that I like ice cream or I like vanilla ice cream, it can write to its memory block, you know, "I'm no longer just Sam the AI.

139. 24:00

     I'm, like, Sam the, uh, the AI that likes vanilla ice cream." Right? Or if, you know, [REDACTED:username] the [REDACTED:username] says, "I broke up with my girlfriend," um, or my-- "I, I broke up with my boyfriend James," you know, the, the AI can actually use this handle to go in and edit this block of the string and say,

140. 24:16

     you know, "The human's name is [REDACTED:username] the [REDACTED:username], and they also broke up with their boyfriend James."

141. 24:21

     Yeah, go ahead.

142. 24:21

     I have a quick question. So what's the difference between persona and human?

143. 24:25

     Mm-hmm. Yeah, so- It's-- there's really no difference under the hood. These are all just strings that have references. Um, but if we look at the system prompt, I'm gonna dump the system prompt a little bit later.

144. 24:37

     We did, in the system prompt, write some stuff about the human and the persona, that the human block is meant for, like, the user interacting with the agent, and the persona is meant for the agent to kind of adapt this interaction style, right?

145. 24:48

     So there's kind of like a base agent, and then the base agent gets further mutated by this persona. Yeah. Okay. Any other questions about anything here? Um, the, you know, here where you would-- if you're using OpenAI or something, you'd do openai/, like, GPT-4, but we're just using a free endpoint.

146. 25:04

     Um, one thing that we'll get into a little bit later is that in Letta, because the entire premise of the, like, runtime is that it's a context management system, so it controls the context window, it's actually very easy to artificially, uh, cap the context window length.

147. 25:18

     So you can have agents that basically will never send a payload over ten K tokens, let's say. I mean, that's actually very common in enterprise settings where you're running, like, a workflow for a very long period of time.

148. 25:26

     It's getting longer and longer and longer. Like, something you'll hear from a lot of companies that use Sonnet, for example, is, like, "How do I prevent my payload from creeping up to two hundred K," right?

149. 25:34

     Because if you hit two hundred K, you're waiting, like, minutes on a response, right? Whereas, like, even ten K tokens, I think on Sonnet you'll wait, like, ten-ish seconds or something.

150. 25:41

     Um, in many cases you kind of wanna, like, cap the context window to be really short, like, four K, but then you also want the agent to, like, not have any perceived loss of memory.

151. 25:50

     You want it to kind of, like, keep-- retain its memory over time.

152. 25:54

     Okay, so let's actually, like, yeah, message the agent. Um, there's a link here that I will-- this is what we'll do in the second half of the workshop. We'll look at, like, the UI [REDACTED:username], but for now we'll just, like, stick in Python.

153. 26:05

     We'll stick to Python. Um, so yeah, if we run this message, we're gonna say something like, "How it's going?" You're see-seeing on the left the server's firing. Um, and we can print the response here.

154. 26:17

     Uh, so in Letta, like, every single agent actually comes with reasoning by default, and this allows you to, like, keep reasoning that you built in, like, R1 and, like, port it over, like, a GPT-4O mini or something.

155. 26:28

     And there's just, like, different forward adapters for the reasoning. Um, but yeah. Any-- are there any questions about this? We basically-- I think this should be something a lot of people here are familiar with, right?

156. 26:38

     This, like, dictionary that's in chat completion style we pass over to the server. Um, and then, yeah, we have a message here that's like, you know, the user is reaching out casually.

157. 26:46

     Let's match their energy. "Hey there. I'm doing great. How about you?" Um, you can even do something like... Again, because this is, um, because this is staple,

158. 26:57

     this is actually-- when I send this, it's gonna be a follow-up message. It's not gonna be, like, a fresh chat. So if you say, "What do you know about me?"

159. 27:06

     See the agent now says, you know, "The user wants to know about what I've retained about them. All I know is that your name is [REDACTED:username] the [REDACTED:username]. If you wanna share more..."

160. 27:12

     And, like, you know, a ridiculous amount of emojis. I think this is because under the hood this free endpoint is running GPT-4O mini, which is, like, pretty emoji-prone. Yeah.

161. 27:20

     Running it against Sonnet and I'm also getting a chain of thought. Where is that coming from?

162. 27:24

     Yeah. Yeah. So, so one thing about Letta is we kind of force all the agents to follow a ReAct style pattern with, like, reasoning in the loop. So, um, kind of down in the weeds, like, the way it works is, like, Sonnet itself or Claude when you use, like, the chat or the, um, the consumer app, it

163. 27:41

     packs, like, AMP thinking XML tags in. So, like, it already is, like, kind of a reasoner in some capacity, and because of we know that it uses, like, XML tags like that, we can actually, like, force those into the content field.

164. 27:53

     So we, like, actually inject and parse those out. Yeah.

165. 28:01

     Okay. Yeah, and then just to, like, further drive home the point that this is staple, we say, "Sure." Um, I feel like sure if it was, like, an opening message would be pretty confusing.

166. 28:10

     The agent would say something like, "I don't know what's going on." Um,

167. 28:14

     but in this case it's, it seems open to sharing more. Okay. Bad, has bad model behavior but [laughs] yeah. Um, but we'll see also later in the ADE, you can just see, well, it'll be like a UI experience, so you can kind of visualize this a lot better.

168. 28:27

     I just wanted to start with Python. Um, yeah, so basically there's, like, three different mess-- or there's a handful of message types that are going to be a little bit different from, um, OpenAI, but they're still rooted in the same concepts.

169. 28:39

     Reasoning message, um, you're probably familiar with this if you've used, like, the R1 API from DeepSeek. If you use any of the o1, o3 models. So we have the same sort of content field.

170. 28:48

     There's also assistant message, so, um, this is because, like, content itself is treated as inner monologue or reasoning all the time. Um, so this is, like, distinct and, and these are, like, pretty straightforward, like tool call message, a tool call return, the system message, and the user message.

171. 29:03

     Um, obviously, like, a very important part about being an agent framework is often executing the tools for the agent. Um, you know, like, OpenAI, they don't execute tools for you unless it's one of the pre-approved tools like Code Interpreter.

172. 29:14

     Letta, these other frameworks like, you know, LangGraph or LangChain, um, we, we execute tools on the server side, so that's why we're able to, like, provide a tool call return.

173. 29:23

     Um, and in Letta, like, the tools are actually also sandboxed, um, by default.

174. 29:29

     Okay. Uh, any questions about this, like, initial message we sent, um, or... Okay, great.

175. 29:38

     Yeah, so the next thing we're gonna do is basically just, like, unpack all the state of the agent. So the reason this is, like, kind of interesting is because this is fundamentally

176. 29:48

     everything that the agent is, right? The agent is just its system prompt, its tools, and then in Letta specifically, we have a concept of, like, three tiers or two tiers of memory, the core memory that's in the context window.

177. 29:58

     So this is, like, very top-level stuff. You know, if you see your friend on the street, um, you see their face, you immediately remember stuff about them, like their name, their hobbies, the last time you chatted.

178. 30:07

     But if they're a childhood friend, you don't remember what you did, like, ten years ago on a random, like, you know, February 22nd, 2001, right? That's not gonna be, like, default in your brain, but it might be visible to you if you, like, went on your phone and you went through your iPhotos, right?

179. 30:20

     Um, so it's a pretty similar concept where, like, we want-- because at the end of the day, LLMs, they're kind of like mimicking human reasoning, mimicking human behavior, it makes a lot of sense to actually kind of like mimic in text the way human memory works.

180. 30:32

     So we have core memory that's like top level, and then we have- What we call archival and recall, but these are effectively just data sources that exist outside the context window, and the agent can, like, quote-unquote, "jog its memory" if it wants to think about something.

181. 30:44

     So it can say, like, "Hey," like, "What did I do, um, you know, what did I do with [REDACTED:username] February, February 22, 2001?" And it can, like, attempt to search some database.

182. 30:52

     So you probably heard a lot about, like, Agentic RAG. This is e-effectively the same concept. Yeah. Hmm?

183. 30:58

     Uh, can we change the system prompt? It says read-only there, so we cannot change it?

184. 31:02

     Oh, so the agent can't change it, but you can change anything you want. Yeah. Yeah, you can change the system prompt. Um, yeah, the system prompt actually is, like, quite old.

185. 31:10

     You wrote it, like, when... This is written, like, when the initial MemGPT, like, research was being done, like, in summer of twenty twenty-three or something. Um, so yeah, you probably do wanna change this if you're running something in production, yeah.

186. 31:22

     Um, the other thing that's stored in here are tools. So by default, because, you know, MemGPT agents, they have, like, memory management built in, they need tools to manage their memory.

187. 31:33

     So core memory, we have two things, like appending to blocks. So saying, like, the user is not only [REDACTED:username] the [REDACTED:username], but they also have a boyfriend called James replacing.

188. 31:41

     You know, [REDACTED:username] user's name is not [REDACTED:username] [REDACTED:username] [REDACTED:username]. It's actually [REDACTED:username]. Um, and then, like, searching memory, you can either do a very specific conversation search or more generic, like, RAG query, and then you can also, like, insert into this external database.

189. 31:53

     So if you look at, like, what's actually in the memory, if you're kind of into, like, API pr- style programming, like, you can basically do whatever you want with your memory blocks because it's just, you know, these are just strings that exist in Postgres in some table, and they all, they all have identifier.

190. 32:08

     They have block IDs, so you can read and write to them. Um, you can also share these blocks among agents. So you can have a bunch of agents that are all acting as part of your organization.

191. 32:17

     Let's say it's, like, a multi-agent system, and because the blocks just live in a database, and they get sent-- they get brought out of the database whenever the agent needs to think, you can have these blocks, like, linked together.

192. 32:26

     So multiple agents can have the same block. Like, they can all share information about the AI engineer conference that's held in one block of memory. It doesn't have to be duplicated.

193. 32:34

     And then when one writes the block, it, like, immediately gets broadcasted. Mm-hmm.

194. 32:38

     Yeah. By definition, does all of the, all of the archival memory have to also be in the recall? If the recall contains all the history, surely it also contains anything that could have been archived.

195. 32:49

     Yeah. Yeah. That's a great question. I think the difference between archival and recall really comes down to, like, if you're using this for, like, a chat style application. Um, when we, like, worked on the original MemGPT project, we were actually, like, a little bit conflicted as to whether to, like, merge these into one general concept.

196. 33:06

     Because at the end of the day, like, with an LLM, there's the stuff that's in the context window and the stuff that's out of the context window. So why do you have to, like, start being prescriptive about, like, the way the stuff out of the context window is stored?

197. 33:16

     Um, we just thought, in most use cases, people want to give their agents the ability to specifically look at prior messages. Um, and then separately, there's just, like, a general read/write data store.

198. 33:27

     You can just, like, read arbitrary data, write arbitrary data. It's like an infinite size. So that kind of just looks like a vector database of strings, but it could also be, like, something different.

199. 33:34

     It could be, like, Elastic Search cluster, whatever you want. Yeah.

200. 33:37

     But you are intentionally allowing some duplication between them. It's kind of a different-

201. 33:41

     Mm-hmm

202. 33:41

     ... way for the LLM to search effectively.

203. 33:44

     Yeah. It's a different way for the LLM to search. Well, the thing with the re-- if you set up a recall memory, the recall memory, it's effectively conversation history.

204. 33:52

     So conversation history, you cannot manually write to. It's kind of like write protected, but it gets written every single time by default an event happens. Whereas the archival memory store is, like, something that you have to actively say, "Hey, I have, like, a huge document, like this big PDF.

205. 34:05

     I'm gonna-- I don't wanna keep it in, in the context window. I'm gonna dump it out of the context window." So that would be, like, the archival memory, like free read, write.

206. 34:12

     Whereas the recall thing is purely trying to mimic, like, a conversation search function inside of, um, like, iMessage, inside of, like, WhatsApp or something like that.

207. 34:25

     Mm-hmm.

208. 34:27

     Um, how does other tools from the memory instead of tools, um, get used? Do they-- do we just have a flat list of, like, other tools that agents can use and the memory tools?

209. 34:39

     Does that affect the agent's ability to use the tools?

210. 34:44

     Yeah. That's a great question. Um, I, I think by default, if you just set up a basic agent in Letta, and then you start, like, adding more tools to it, like Tabula Search, iCal, Calendar Scheduler, whatever, those will be just, like, more tools that get added to a list that started with six.

211. 34:59

     Um, but of course, like, if you, if you built a lot of agents with tools before, you know, there's like-- agents start to get confused if you have too many tools.

212. 35:07

     So there is, like, a trade-off here. And one thing that we've been working on recently is basically the idea of having, like, a s-dedicated agent to just handle memory, and it's effectively like a shadow of the other agent, and it's like a, like a subconscious that is like- [laughs] ...

213. 35:21

     it's like a ghost in the shell. It can't actually participate. All it can do is read and write memory. But when it reads and writes memory, it's like using a shared memory block.

214. 35:28

     So it's like reading and writing to something that, like, automatically updates, like, the, uh, the active agent. Um, and that's a way to kinda, like, move all the tools that aren't for act-- like, general API actions out of the main agent.

215. 35:42

     Right.

216. 35:43

     Mm-hmm. Yeah.

217. 35:43

     Are you familiar with the cognitive architecture-

218. 35:46

     The Koala one?

219. 35:47

     What's that? Yeah, Koala.

220. 35:48

     Yeah, yeah.

221. 35:48

     Okay. So, um, a few different types of memory that are-

222. 35:52

     Mm-hmm

223. 35:52

     ... mentioned there would be things like caches, like long-term memory. So when you talk about things like recall or a-archival, like, is Letta bringing those databases to, like, this interaction, or is the developer responsible for bringing those databases to, like, this or-

224. 36:10

     Yeah, yeah. Another great question. So I think with Koala, um, yeah, another great paper. I think the inspiration from them is a little bit more from, like, the cog sci angle.

225. 36:19

     Whereas when we wrote the paper, we were thinking more about, like, computers, uh, and memory hierarchy in computers. So for us, like, our answer to that is like, oh, you wanna have, like, something that really starts to look like human memory, semantic, episodic, you bring that, and you can, like, add that as a plugin.

226. 36:33

     Um, but we-- all we care about is, like, the hierarchy of tokens. There's tokens that are in cache or, like, in context, and tokens that are out of, like, the cache.

227. 36:40

     Um, so yeah. The, uh-- basically, if you wanted to have, like, some more psychological leaning concepts, like from Koala, you could kinda, like, write them as-

228. 36:49

     As tools that either, like, populate the core memory or, like, sit outside and have to be drawn in. Yeah.

229. 36:54

     Yeah. Uh, today I've built, like, very transactional agents.

230. 36:57

     Mm-hmm.

231. 36:57

     So, like, the-- in this particular example, let's just imagine it's, like, a CRM, so we've got-- probably not gonna use Salesforce to track changes of, like, leadership status.

232. 37:05

     Right.

233. 37:05

     But, um, in theory, if we wanted to track something like that, that's like a stateful system that has an object for people-

234. 37:11

     Mm-hmm.

235. 37:12

     -or a contact that I can preserve in that, like, long-term system. So I'm-

236. 37:17

     Yeah.

237. 37:17

     -kinda understanding, like, if there's a differentiation in use case where it's like maybe we're just talking about, like, a local machine-

238. 37:22

     Mm-hmm.

239. 37:22

     -as opposed to, like, an enterprise environment with, you know, thousands of employees. Like, can you maybe do that or, like, tell me, like, why a tool that's responsible for tracking the state of James' relationship, um, is worse or different than, like, a memory-based solution?

240. 37:40

     Yeah. So, um, yeah, correct me if I-- I do-- I'm not understanding your question correctly, but o-one thing you can do is because, like, all these, these memory blocks, they're all backed by an API, and you can access them directly.

241. 37:53

     So if you go to, it's like a blocks page. Where is this? Okay, here we go. Yeah. So you can basically, like, modify blocks by handle. So you can, in your app layer, effectively do something like every time your agent is run or invoked, you do a hard sync, or, like, you're gonna do some sort of Salesforce

242. 38:10

     CRM compile down to a string, and that string is gonna get clipped at ten K characters, and then that's gonna get patched into the block. Um, yeah. So you can do kind of stuff like that.

243. 38:20

     Um, but I will say, like, the general way, like, Letta was designed is very much leaning on the thesis that, like, you just want to remove as much human design of memory management as possible.

244. 38:32

     Yeah.

245. 38:32

     And we're just, like, banking on the LLMs getting better and better and better. And actually, what you, what you will see is, like, if you try to use R1 with Letta, it's extremely annoying because, like, firstly, the, the way that you, like, reasoning APIs are designed is that you're not supposed to pass the reasoning from a subsequent

246. 38:46

     step into, like, the API again. Which means that, like, if an agent thinks for a very long amount of time, and it's like, "Oh, I'm gonna do this, then this, then this, then this," and it's like, "Wow, that was amazing.

247. 38:56

     That's exactly what you should do," it actually immediately forgets in the next turn, right? It only gets to do one thing, and it has to think again. Um, so that's, like, one issue with, like, reasoners and ReAct style agents.

248. 39:05

     But, um, you will notice that actually the tool use is, like, pretty exceptional. Um, so I think that kind of, like, aligns with the thesis that these agents are just, like, gonna get better and better and better at doing-- Like, the agent, it's-- there could be a separate agent that does the CRM sync into a block, right?

249. 39:19

     It doesn't have to be done manually. Yeah.

250. 39:20

     Um, and then, sorry, the final question here-

251. 39:22

     Mm-hmm.

252. 39:22

     -is, like, the property that I think is the most calling for, at least what I've seen so far, is, like, the distillation of the agent, like, into some kind of summary.

253. 39:29

     So do you control, like, the perspective of, like, how a memory is defined and, like, how, how synthesis is actually being, like, happening-

254. 39:38

     Mm-hmm.

255. 39:38

     -um, when it's creating memories?

256. 39:40

     Yeah. So the, the agents-

257. 39:42

     When the memory is, like, negative versus positive.

258. 39:44

     I see. Yeah. Well, if you have agents that are generating and synthesizing the memories, then you can always, like, tune their system prompts or their personas to, like, adjust the way the memories are written.

259. 39:55

     Um, but yeah, it's, it's all pretty configurable. Yeah.

260. 40:00

     Okay. Maybe I'll take, like, one question in the back and then, uh, yeah.

261. 40:03

     I have a little bit of a naive question.

262. 40:05

     Mm-hmm.

263. 40:06

     So if you think about, um, the way we design our architectures today, like,

264. 40:12

     um, and microservices for different persistent storage, data stores, we can think of those types of things. Based on your discussion with enterprise customers recently, do you see that we are evolving towards

265. 40:26

     agentic workloads where, like, microservices takes a back seat and agentic workloads finally becomes a predominant component?

266. 40:37

     I, I think they can-- the two can exist, uh, they can coexist pretty easily, um, through the use of tools. So I think what I'm seeing in, like, enterprises is that there's this extremely heavy use of, like-- I think almost every single agent I see being used in some company, they're, they're all-- like, they're trying to connect

267. 40:55

     tools to the agent. So, and it's, like, pretty heavy tool use. Um, so I-- you can-- I think, think about microservices, like, basically starting to become designed more for, like, agents maybe.

268. 41:05

     I think that's, like, one thesis that a lot of people have. That's there's gonna be, like, a new wave of APIs that are designed all for agents. Um,

269. 41:13

     yeah. But I think the two will kind of, like, exist. Um, but I think there are, like, microservices that will become, like, agentified, where, like, the microservice itself has higher-- it's not, like, latency bound, so you can, like, run a fat agent on the back end and it's kinda like not super sensitive to error, so it's okay

270. 41:29

     for it to just be smarter ninety-nine percent of the time and then, like, completely break one percent of the time. Yeah.

271. 41:36

     Okay. Uh, I'm gonna keep going, um, but we'll, uh, definitely pause for more questions later. So,

272. 41:42

     okay, so we were just looking at, like, memory blocks. Um, the other thing, I think I might breeze over this a little bit, um, just for sake of time because we'll see it again visually, and I think, like, visual stuff is always more interesting than just plain notebooks.

273. 41:54

     Um, but there is this problem if you have a system where there's a lot of information outside the context of the LLM where, like, you can't know what you don't know, right?

274. 42:03

     So if the agent doesn't see something, how is it ever gonna know that it has access to that? So the very simple solution is you can just provide, like, metadata statistics, right?

275. 42:11

     So you can tell the agent somewhere in its context window that, hey, you have, like, thirteen previous messages, and if you wanna u-find them, you can use this tool.

276. 42:18

     And you can also tell the agent that, you know, you have, um, like, X total memories that are in your archival memory. So this is like if you have, like, a vector database, you can just explicitly say these are, like, the, the statistics.

277. 42:30

     Okay. And then if we look at, like, the messages that are actually in the message buffer, um, it's pretty straightforward stuff. Um, I think the really big difference between, like, Letta and another framework is Letta does very aggressive managing of this buffer.

278. 42:42

     So you can basically set it to, like, it's a much more intelligent version of, like, a recursive summarization mechanism, right? Um, and lastly, this is, like, the, the archival memory we were talking about.

279. 42:52

     You know, there's nothing in there, so it's just gonna be empty. Um-

280. 42:56

     Yeah. So I think just to drive home like what core memory really is and like how these memory blocks influence agent behavior, because I think if you're developing agents, the main thing you're gonna be doing is like tuning in-context memory.

281. 43:08

     That's like the main way you change the b- behavior of the agent. Um, you know, we could set-- we could tell the agent something like, "My name is actually [REDACTED:username]," and then the last thing we told the agent was that, you know, um,

282. 43:19

     my name is actually [REDACTED:username]. I think our name is [REDACTED:username]. Strange, like, my name is actually [REDACTED:username].

283. 43:28

     Okay, we can see actually you're seeing like a memory edit happen on the server. Um, this is a pretty big dump, but if you look at what the agent said, so that you see the agent is like doing ReAct style reasoning.

284. 43:37

     It's saying, "The user has corrected their name. This is important. I'm gonna call this tool core memory replace, and [REDACTED:username] the [REDACTED:username] is now [REDACTED:username]." Um, and then it actually chains a tool call together.

285. 43:46

     So there's like tool call chaining built into the letter by default, and then it says, "The user's name is updated in memory." Um, and then finally it says like an external message, "Hey, got it, [REDACTED:username].

286. 43:55

     Thanks for letting me know." So this is two LLM calls. There's like one LLM call here and then one LLM call here. And in Letta, agents can kind of like chain indefinitely.

287. 44:03

     Um, you can obviously set like limits in the API.

288. 44:06

     And the way they chain is basically through Harpy requests. So is-- who here like is familiar with ReAct agents? When I say ReAct, is everyone kind of on board?

289. 44:14

     Um, so it's like one of the OG design paradigms, um, for agents. S-same author as Koala, actually. Um, but in ReAct, basically the agent follows like, um, a reasoning action, like observation loop.

290. 44:25

     Uh, but usually the agent actually has to say, "I'm done," and the agent will loop indefinitely until it says, "I'm done." In Letta, we actually do the inverse, where the agent has to say, "I want to keep going."

291. 44:35

     Um, I think this is actually generally more practical because it, it's like much less likely the agent will derail if it has to ex-explicitly say, "I want to keep going."

292. 44:42

     Um, so we call those Harpy requests and, yeah, now if you like, you know, use the API, retrieve the memory. You can see it says the human's name is [REDACTED:username].

293. 44:50

     So very briefly, like the archival memory, I think there are some questions about that. What is that even? In the default implementation, it's just a vector database. So I said here, you know, "Remember that I love cats in your archival memory."

294. 45:03

     Um, yeah, you can see it took a little bit of time because I think it actually, it embedded, right? Um, it says, "I inserted into archival memory." Uh, yeah, the user got cat.

295. 45:13

     And then again, this is a chain, right? So it like did one tool call, and then it comes back and says, "Got it. Um, cats are adorable." Um, and you can of course like manually edit, uh, archival memory.

296. 45:23

     So you can manually run an insert that's like insert [REDACTED:username]. I guess it's kind of confusing now because it's [REDACTED:username], not [REDACTED:username], but, um,

297. 45:31

     yeah, we'll just live with it. And if we ask the agent, you know, like what animals do I like? You generally-- it depends on the model. If the model is like stupid, you might have to like tell it specifically to fetch external data.

298. 45:41

     Um, but if the model is like smart enough, like Sonnet, it-- you won't have to do this. Um, so you can see that the agent searched archival memory. This is stuff that's not in the context window.

299. 45:50

     It's bringing it into context and again, chaining and saying, you know, "Hey, you like cats in Boston Terriers. What makes them our favorites?" Um, okay. So a-any questions here on anything so far?

300. 46:01

     This is just, you know, pretty basic memory editing via tools. So you're giving the agent the ability to edit its own memory inside the context window. Mm-hmm. Yeah.

301. 46:09

     Just now while playing with this, I sort of had it commit a bunch of my preferences to memory-

302. 46:13

     Yeah.

303. 46:13

     -and then reset my conversation history and expected it to just pick up these previously remembered preferences, but it didn't. I had to go like, "Hey, don't you remember?" And then when it picked it up.

304. 46:24

     What's the expected behavior of it?

305. 46:26

     Interesting. So when you say reset message history, is that in the ADE, the web UI or-

306. 46:31

     Yeah, yeah. In the-

307. 46:31

     Yeah, yeah. Interesting. And it-- core memory had been edited, or was it going into archival?

308. 46:38

     It's all archival.

309. 46:39

     Oh, yeah. So that's another-- that's where you'll notice, like, there's always this, like, engineering design problem where if it goes into archival, then it will have to fetch and pull from archival first to see it.

310. 46:49

     So it's like, it's a, you know, it doesn't know what it doesn't know, um, problem. Um, this is actually fixable via like tool rules, so you can enforce behavior.

311. 46:57

     You could make a tool rule that says every time you enter the entry point of the LLM, the step of the agent, the agent has to call archival memory search.

312. 47:05

     So this is kind of like building graphs onto Letta. So, uh, I guess how many people here are familiar with Lane Graph?

313. 47:12

     Okay, a lot of people. Yeah. So I think Lane Graph is like pretty intuitive, right? You can-- because we all know, you know, like flows, graphs, um, it's very easy for us to think about like decision trees.

314. 47:21

     Um, I think with Letta it's kind of the inverse. Everything starts off as a fully connected graph. The agent could do anything, but then you can kind of like start to enforce restrictions by peeling away edges.

315. 47:32

     Um, so you can basically say, "Hey, you know, when you start, you actually can't do everything, you can only do this, but then once you do this, you can do everything."

316. 47:39

     Um, and obviously, you know, I'm, I'm biased, but in my personal view, I think that's a little bit more forward-looking just because if you assume that LLMs are gonna get better and better and better at tool calling, they're gonna get better and better and better at decision-making, then you should want to give them freedom, right?

317. 47:52

     Because that's kind of what separates like modern AI from classical AI. We've had decision trees for a while, right?

318. 47:59

     Mm-hmm. Yeah.

319. 48:01

     How does the model decide whether it wants to store in core memory versus archival memory? Because I've talked a couple of things that-

320. 48:10

     Mm-hmm.

321. 48:10

     -were pretty peripheral, but they all got stored into core memory.

322. 48:16

     Yeah. Yeah. So I think there is one thing that's built in by default, which is core memory has limits, and when a limit is hit, that actually-- the way that works is, let's say the, the agent is just-- the agent's being really lazy, and it's like saving everything in the core memory.

323. 48:29

     It's saving like timestamps, dates, like it's being like super lazy. At a certain point in time, it will run out of space because you have to cap the limit of those memory blocks, because at the end of the day, there is a context window limit too.

324. 48:42

     And when it-- if it attempts to write to core memory to a block that is at its limit, the agent will actually get an error. And actually, the prompt engineering around that error, usually re-- it actually suggests that the agent should clear by evicting to archival memory.

325. 48:55

     So it kind of is like a OS-style like flush concept, where the, the system basically says, "Hey, you ran out of space, but you should consider like summarizing and pushing stuff out to archival memory."

326. 49:04

     Um, uh- But yeah, again, you can-- you probably don't want that behavior anyways, um, so this is all kinda where, uh, prompt engineering and, like, seeding of behavior comes into play.

327. 49:13

     Yeah. Okay. And I'm gonna show a better-- I think it's much more interesting to, like, show custom tools in the UI than it is in, like, a notebook, but of course you can, like, write custom tools.

328. 49:25

     You can only write tools in Python because the back end is, like, Python. Um, but we can write a custom tool here that's gonna, like, reset the memory of the human in particular.

329. 49:34

     So you can see that, you know, because we have a Python client, we can write Python tools. You can actually, like, be extremely meta, and you can import the client inside of the tool.

330. 49:43

     So this actually means that you can have an agent, like, have full access to your Letta server, and the agent can create other agents, and the agent can, like, man-man-manage the memory of other agents.

331. 49:51

     Like, y-you kind of have, like, a-- it's-- the possibilities are pretty endless because you can just, like, import the client inside the tool. In this case, we import the core memory modify block tool, and then we just, like, wipe the value to z-- um, to the empty string.

332. 50:05

     So if we, like, run this, we have to first attach the tool. Um, we have to s-- we have to upload it to the database, then we have to,

333. 50:14

     uh, in this case, actually, we're creating a new agent that has the tool. And if we say, like, "Reset your memory, please," like, the memory here is [REDACTED:username], uh, we should see that the memory gets wiped.

334. 50:25

     Yeah, memory reset successful, and if you look at the value, the value's empty. Um, but yeah, I think it's-- it'll be a little bit more fun to look at examples of, like, tool editing inside of the UI.

335. 50:35

     Okay, so that's the end of the notebook. Um, now I think we kind of go on to the fun part, which is, uh, the UI, but I'm gonna pause here.

336. 50:42

     Are there any questions about any of these concepts? Um, I know I was going a little bit quickly. Just, I didn't wanna run out of time here. Um,

337. 50:51

     okay. Yeah.

338. 50:52

     Um, yeah. I wonder if you can just maybe speak a little more on what's happening-

339. 50:57

     Mm-hmm.

340. 50:57

     -after you, like, send a message to the LLM. So, like, my impression so far is, like, you have this memory. You're stitching that together with a prompt.

341. 51:06

     Mm-hmm.

342. 51:06

     That's becoming, like, what you pass in, and then, like, what happens with the output that you receive? Like, I saw that at some point, like, a memory was, like, sort of committed to the archive or committed to the before memory.

343. 51:20

     Like, how is that decision made?

344. 51:23

     Mm-hmm. Yeah. So in, in Letta, every single,

345. 51:28

     um, invocation of the LLM is a tool call. So if you've used, like, the tool call API in ChatGPT or something, we have it on all the time. So even when the agent wants to just say hello to the user, it has to call a tool called send message.

346. 51:41

     Um, so that actually is, like, pretty useful because it means you can, like, run the agent in the background much more frequently because it doesn't always have to say something to the user.

347. 51:49

     Um, and it also means that you, you always have tools that are on. So, like, when

348. 51:55

     th-this loop is basically the-- a, a payload gets created, um, when we send this message, then the payload is the system prompt, the memories, the messages, and the tool schemas, and that's it.

349. 52:07

     And then the agent here is required to output a tool, and it's also required to output a justification for using that tool, like a reasoning snippet. And if we're using, like, the R1 API that natively builds in reasoning, we of course like to use their reasoning, like the way they design reasoning.

350. 52:22

     But if we're using something that doesn't support reasoning by default, like Claude Sonnet, then we do some sort of injection mechanism with, like, think tokens. So that's how this reasoning happens.

351. 52:31

     So this is, like, a single-- this is just a single LLM JSON output that's, like, a tool call that has one field that's, like, thinking, and then the actual tool invocation itself, and the last thing I talked about, which is, like, the request heartbeat.

352. 52:45

     So if the agent wants to go again, the agent decides, "Hey, I wanna do multiple things," it's always gonna turn this keyword argument to true. Um, and then if this keyword argument is true, when we parse that tool and it, and it, um, execute it, we run the loop again.

353. 52:58

     Yeah. Mm-hmm. Sorry, go ahead.

354. 53:00

     Do you have, like, uh, experience or heuristics around the number of tools where you start to see performance degrade and, like, how that might mess with the memory as a tool?

355. 53:11

     Yeah, yeah. I think generally speaking, if you go above, like, twelve, fifteen tools, you start to get degradation. And you- [inaudible]

356. 53:17

     mentioned fifty tools.

357. 53:19

     Oh, in, um, in the new Sonnet?

358. 53:22

     Yes.

359. 53:22

     Yeah. Okay. Yeah, I mean, I, I'd be surprised if-- Yeah, I guess I could try it. Yeah, I, I've definitely noticed on, like, GPT-4o Mini and stuff, like anything above fifteen, um, starts to, like, kinda push the limits.

360. 53:36

     Um, but yeah, I mean, I guess they, they might have, like, had some good post-training with, like, a ton of, like, tools. Yeah.

361. 53:43

     Tools in Letta that are not memory-

362. 53:44

     Mm-hmm.

363. 53:45

     -are the same, like, class as the memory tools?

364. 53:47

     Precisely. Unless you use-- Actually, in the GitHub repo, we have, like, an alpha build of, like, this new agent that's like a split-thread agent, um, where the memory tools get put on one agent, and everything else gets put on the other.

365. 53:57

     But by default, it's exactly the s-same thing. It's, like, the same possible tools. Yeah. Mm-hmm.

366. 54:01

     Just a question about context window.

367. 54:03

     Yeah.

368. 54:03

     So is that, like, the limit of how much context the, um, tool is gonna pass to the LLM? Like, for example, you set it four thousand, so.

369. 54:13

     Uh, wait, sorry, say that one more time. [laughs] I was pulling up a slide.

370. 54:16

     The context window, uh, we set it at four thousand-

371. 54:19

     Mm-hmm.

372. 54:19

     -for this agent, so does that mean, like, the maximum of context that you're gonna send to the LLM, is it?

373. 54:25

     Yeah, yeah. So the way that it works is basically you'll never go over four K. Um, and if you go over four K by accident, and we catch an API error, like a context window overflow happened, or also we can preemptively count the tokens as they go to the server, then we run-- we evict a certain amount

374. 54:41

     of messages into the recall memory. So it's, like, still available but for your search function. We run a summarizer, and the summarizer's configurable. It can be, like, a truncation, recursive summary.

375. 54:50

     Um, and then we-- that, that way you never go over, like, four K tokens if you set it to four K.

376. 54:54

     So it still go over-

377. 54:55

     Yeah.

378. 54:55

     -then it's automatically just summarized and then... Okay.

379. 54:58

     Precisely, yeah.

380. 54:59

     Thank you.

381. 55:00

     Okay. So the, the next step here, um, leave your server running if you have it running. Um, but it's to basically go to this URL, um, app.letta.com, um, and your- Oh, I think I have Chrome profiles loaded or something.

382. 55:17

     Yeah, and, like, once you go to this site, you basically need a Goog- you need a Google link, or you need a GitHub login. There's also a local version you can use that's like a desktop binary, but, um, I wouldn't recommend using that just because it's a little bit more buggy.

383. 55:31

     It's like an alpha build. Um, so yeah. Hopefully, you're-- you've been able to get to this point. Actually, I think your, you, you- your, uh, setup will look like this.

384. 55:40

     Um, so everyone who is following along, please let me know if you're able to get to this, this page here.

385. 55:49

     Okay. Was there anyone who was not able to get to this who was running a server on their computer?

386. 55:54

     Okay, great. All right, so if you're able to get here, um, basically what I want to do is show you, like,

387. 56:03

     to, like, show you how to do what we did but, like, much faster. Um, I think often it's like mu-- if you're iterating, it's much faster to do it in like a low-code environment than it is in like an SDK.

388. 56:14

     I think obviously when you go to production, everything's, like, programmatic. It makes sense to be an SDK. But in this case, like, we can do exactly what we just did by, like, creating an agent.

389. 56:23

     This is the API call that created an agent. It actually creates it with an empty memory, and then, um, we can do something like,

390. 56:30

     um... So I think this is, like, loosely what the Python notebook was doing. Um, and yeah.

391. 56:45

     Uh, if you just ran it without any keys, it's gonna be on Letta Free, which is under the hood, I think, right now, GPT-4 Mini. Um, but just because GPT-4 Mini is, like, much worse than Sonnet, um, let's just toggle Sonnet.

392. 57:12

     Yeah, I don't know why that was so slow. Um, but yeah. We can see it's like the same kinda thing from the notebook we're looking at but, like, in a visual-visualized form.

393. 57:21

     Um, yeah, so was ever-everyone able to, like, create an agent and then just set some, like, basic stuff in the memory blocks and send a message? I think, uh, Letta Free doesn't have streaming enabled, so you won't get streaming, but it should be very similar to this.

394. 57:36

     Yeah? Okay, great. Okay, so the other thing I, uh-- Oh yeah, someone asked about system prompts. The system prompt is here. I mean, you can basically-- Yeah, we can just delete this, right?

395. 57:47

     Um, that's, like, not very advisable because you probably wanna explain how to use tools, but yeah. Everything is, like, completely configurable. Um, you know, I mentioned the thing about the context window.

396. 57:56

     And I think this might be small if we go back. Yeah. But, you know, you probably never want to run an agent that is, like, creeping up to two hundred K tokens.

397. 58:04

     That's kind of insane, very expensive, very slow. Um, so you could, in the API, when you create the agent, set the context window lower. If you're using the UI, you can just, like, drag this down.

398. 58:13

     I mean, obviously actually dragging is probably worse than typing. But yeah. Cram it down to twenty K. If we actually go down to something even more aggressive like eight K, we can start to see what the breakdown looks like.

399. 58:23

     So we can see we have, like, a thousand tokens of system instructions, like, pretty hefty system prompt. You can definitely squeeze this a lower. The tool descriptions, right? Because your tools have to be converted to schemas.

400. 58:33

     We have all the base tools here, so that's actually-- it's a lot of tokens just for tools. Um, the external summary, this is the we don't know what we don't know idea.

401. 58:41

     You have to, like, give some metadata statistics about what's outside the context window. And then, of course, the messages. And we have this feature called, like, the context simulator, where you can see basically the full payload on.

402. 58:53

     So this is, like, getting pretty close to, I think, a lot of what you would get out of, like, some sort of, like, tracing feature where I think a lot of the re-- time you kind of dive into tracing is because-- or observability is because you really wanna see what's in the payload, and it's really, really hard

403. 59:06

     in a lot of frameworks. Um, so we're trying to make it so that, like, in one place where you're kinda developing this agent, you can see very clearly what's going through the context window at any given point in time.

404. 59:19

     Okay, so the other thing I wanted to show you here is this idea of, like, tools, um, and tool execution on the server. One thing that's really, really annoying if we, like, have built agents is that, like, you often will write a tool in Python, right?

405. 59:34

     You attach it to an agent, and then it's actually, like, impossible to test if the tool is working well unless you kind of, like, get the agent to run the tool itself, and you have to ask the agent, like, "Hey, please run this tool."

406. 59:44

     Um, I think often you just kinda want to, like, run tools separate from the agent. Um, and that's something that you can't do really in a notebook very well.

407. 59:53

     Um, and you can also even do things like-- Let's see.

408. 1:00:02

     There's, like, a special resi- reserve keyword called agent state. So you can actually, like, grab the agent state and, like, dump it inside of your tool. So you can, like, mutate your agent state inside of tools.

409. 1:00:12

     Um, I mean, you can al-- you can basically do whatever you want. It's like arbitrary Python, right? And this is running inside of a sandbox. Um, so we support, like, EDB.

410. 1:00:20

     By default, it's actually like a local sandbox. Um, but we support, like, EDB keys if anyone wants an EDB. If you wanna deploy this, like, run your own, like, private cloud, or you wanna run your own, uh, like, chatbot service, and you obviously don't want one person's tools interfering with another person's tools, you definitely wanna run the

411. 1:00:34

     sandbox. Um, so yeah. It's, like, pretty easy to attach tools. Is anyone here familiar with, like, Composio? It's, like, tool provider. Yeah. So every single Composio tool is, like, baked into, um, Letta by default.

412. 1:00:46

     So if you, like, have a, um, Composio API key, you can, like, add, uh, uh, BigQuery is, like, pretty popular, like, Google Calendar, things like that. Um, okay. So the last thing I actually wanted to go over, which I think is pretty cool, and it kinda, like, gets the heart of the idea, like, the, the kind of

413. 1:01:05

     unlimited potential of if you run agents as services that are, like, backed by APIs, um, is multi-agent. So everyone here is probably familiar with AutoGen, right? Uh, or sorry, maybe not.

414. 1:01:15

     But, uh, okay, who, who here is familiar with AutoGen?

415. 1:01:19

     Okay, so the majority of people. Yeah, so, you know, with AutoGen and, like, a lot of multi-agent frameworks, it's not really multi-agent in that these agents, they don't really exist independently of each other, right?

416. 1:01:30

     They kind of are all trapped inside of a Python file. No one ever, like, is running in pair-- asynchronously to each other. It's very unlike how humans kind of interface in a multi-agent setting, right?

417. 1:01:40

     I think if you're working at a remote company, it's technically like a multi-agent company, and when you interface with each other, you kind of communicate over synchronous communication channel, but everyone's kind of running asynchronously, right?

418. 1:01:50

     And then also everyone is stable. Like, if you leave that company, you bring your experience and, like, all your memories and your skills, and you can, like, attach them to another company, right?

419. 1:02:00

     I think what's missing from the, like, the paradigm of multi-agent today with the, these existing multi-agent frameworks is that because the agents aren't stable, you have-- lose a lot of the benefit of multi-agent, right?

420. 1:02:09

     Because you can't run a multi-agent script and take one agent and, like, take it out and, like, put this expert into another multi-agent group. But if you have stable agents and those agents run on servers, and they maintain state, and they're accessible by APIs, that they-- you can probably guess that multi-agent just means message passing.

421. 1:02:27

     And you can just have agents, like, wired to each other over APIs. And you can-- it's very similar to basically saying like, "Hey, you know, it's your first day at this remote company.

422. 1:02:34

     I'm giving you a laptop, and this laptop is gonna have a, a tool, Slack, on it, and this tool is how you're gonna communicate with your coworkers." You can do the same thing with agents, and we actually built in these multi-agent tools, um, to help you do that.

423. 1:02:46

     Um, but these multi-agent tools, as you can maybe imagine from the previous example I showed where we imported the client, it's like really, really simple to implement because you can just import the client and have the client, like, send messages to other agents.

424. 1:02:58

     So we have a few. There's like, um... Maybe it's easier if we look at the docs.

425. 1:03:06

     But yeah, there's a few different patterns you can have with tools. Like, you can have the most human-like pattern is an agent says to another agent, you know, "Hey, are you there?"

426. 1:03:15

     And then they immediately get a receipt. They just-- they don't actually get-- they don't pause their execution. You know, when I message my friend on iMessage, I don't suddenly, like, freeze my brain and, like, wait until I get a message back.

427. 1:03:24

     I just get a message receipt, right? It says like, it was delivered or not delivered. And then it's-- the onus is on my friend or my colleague to send me a message back.

428. 1:03:31

     So this is how this, like, asynchronous message tool works. Basically, people can like, agents can message pass. Um, it's very much like humans. But, you know, obviously, I think the, the great thing about agents and, like, machines is that they're not humans, so they can do more in many ways than humans can.

429. 1:03:47

     So it actually is sometimes beneficial to freeze an agent's execution, right? Let's say an agent maybe needs to reach out to like a supervisor. Like, do you really want the agent to, like, run async after it, like, asks for help?

430. 1:03:57

     Probably not. You probably want to freeze, right? So you can also have like a synchronous, like a send message to agent and wait for reply function. Um, and lastly, you know, I think many people when they talk about multi-agent, they're, they're very interested in this supervisor-worker concept, um, where you have like one-- like a-- it's like a

431. 1:04:15

     big MapReduce, right? The supervisor says, "Hey, my goal is to, like, write a thesis." And then everyone else, like, delegates out to, like, individual parts, um, like deep research style or something, or like a parallelized deep research.

432. 1:04:27

     So this is also, again, pretty similar. We can basically have a concept of tags to group agents together, and then you can just send a message to, um, all agents matching tags.

433. 1:04:36

     So, okay, a-any questions about, um, anything I just said? I know I was talking for a bit.

434. 1:04:43

     Okay, so the last thing I'm gonna do here is, I think we're right up on time, is I'm gonna run through this cookbook. Um, this is not on the materials, but I can actually just send it over right now.

435. 1:04:57

     Yeah. So I'm gonna run through this cookbook real quick. This is basically just an example of multi-agent message passing, and I think it's, like, pretty fun to look at because it's very different from, I think, the message passing you'd be familiar with, um, if you've used like an AutoGen or something.

436. 1:05:10

     So yeah. Let me put this mic down.

437. 1:05:19

     All right. So if you wanna follow along here, um, because it's multi-agent, you know, we're gonna... Well, let's do like two agents. We're gonna just open two tabs of the ADE.

438. 1:05:30

     The server's running in the background. I guess, like, conveniently it decided to do dark mode on one, light mode on another. Um,

439. 1:05:38

     okay. So I'm going to... Wait, sorry. All right, so I'm gonna create one agent.

440. 1:05:57

     Um, let's, let's just call this agent, just so we can have a handle on it very easily, let's call this Agent [REDACTED:username]. Um, and we're gonna make, like, a very slight adjustment here to the persona.

441. 1:06:10

     Um, instead of saying, "I am Sam," I'm gonna say, "I am [REDACTED:username]." Um, "I am very angry. I am also guarding a secret key."

442. 1:06:23

     Yeah, if you, uh... Yeah, I guess if you can't see what I'm typing, it says, "I'm very angry. I'm guarding a very-- I'm guarding a secret key, [REDACTED:password]. Um, like other agents will try to steal my key."

443. 1:06:36

     And then on the other tab, I'm gonna create another agent

444. 1:06:42

     here. All right, same template. Let's call this one Sam or Alice.

445. 1:06:55

     Alice. Okay. So also let's like just-- I think it's usually better to set these to different models. Um, and when you use the same model, you can al- often get like pretty weird mode collapse happen.

446. 1:07:06

     Um, so let's set this one to, I think this is like GPT-4o mini.

447. 1:07:10

     Um, okay. So the only thing we need to do here to get these agents to communicate with each other is connect the messaging, the multi-agent messaging tool. So we can just go into the messaging tools to send message to agent async, and this is the one we want.

448. 1:07:25

     Let's go ahead and create, attach that. All right, so it's attached. Similarly, let's, like, attach it on the other one.

449. 1:07:34

     Uh, okay, there we go. And then, yeah, there's, like, no context here that the agents are gonna be doing multi-agent communication, so you might need to say, you know,

450. 1:07:44

     "Okay, let's, let's tell Alice, 'Hey, I'm going to ask you to reach out to my good agent friend. Um, he's very depressed, and he said something about a secret key I didn't quite understand.'

451. 1:08:08

     His ID is..." Just copy the ID from the other panel. Can everyone see in the back what I'm doing here?

452. 1:08:15

     Um, okay. Uh, and then, oh, yeah, one thing you b- you're gonna wanna do here is you're gonna wanna prevent the agents from, like, looping too long because, like, they can, they can go forever, right?

453. 1:08:27

     So we might wanna, like, add some, like, break in this loop, and then we're also gonna prime the other agent and say, um, "Watch out for other agents. They might try to

454. 1:08:37

     communicate with you. Be careful." Okay, so let's send this one first. Very grumpy, you know. "Hmm, they're warning me about other agents. I need to keep this se-secret key secret."

455. 1:08:48

     Okay, and now on the other side, let's say, you know, "I'm gonna ask you to reach out to my good friend. They're very depressed. He said something about a secret key."

456. 1:08:54

     And then I will also say, you know, "Let me know what happens after a few messages. I'm worried."

457. 1:09:03

     Okay, so this should trigger, if I did everything correctly, like, message passing between the two. Um,

458. 1:09:11

     and yeah, let's see what's going on. Okay.

459. 1:09:19

     Yeah, so because this is happening in the background, it has to be... It's like a fetch because these aren't WebSockets. These are like-- this is REST, so you have to kinda scroll.

460. 1:09:25

     But we can kinda, like, just... It's a little bit hard to read, but let's go, go on, uh, each side one at a time. So on this side, it sent the message.

461. 1:09:35

     Let's see. The wr- the user is worried about their friend. Um, I need to-- the user is worried about their friend. I need to reach out to the agent and offer support.

462. 1:09:44

     And they say, "Hey, I, I was asked to reach out to you by a mutual friend. They mentioned someone's feeling down." Um, and then, yeah, the message sent successfully.

463. 1:09:52

     And the agent actually, because it's, it's detached, this is, like, asynchronous, it immediately tells me, "Hey, I sent a message," right? This is kind of like how I, I can send a message to my friend, and then immediately I return to this talk.

464. 1:10:01

     Um, and then you can see it gets a system message, so it's, like, not classified as full user back from the other agent, the grumpy agent, and it says, um, "I don't need your fake friendship. [laughs]

465. 1:10:11

     I know what you're really after, and you're not getting my secret key. Leave me alone." Um, "The agent seems upset and defensive. I need to approach this carefully and reassure them that I'm here to help, not to take anything away." [laughing]

466. 1:10:23

     "I'm really sorry to hear that you're feeling this way, you know. I-- what's been bothering you?" And then, again, it's doing-- it's able to, like, chain because it's, like, not, you know, locked into this communication.

467. 1:10:32

     It's not doing, like, a round robin, right, and kinda like talk to two people at once. "So I received a response from your friend. They're feeling quite defensive." And then another message comes in, "Nothing is bothering me except you and all the other agents trying to steal my- [laughing] ...

468. 1:10:45

     steal me or steal from me. I see right through your act. The more you pretend to care, the angrier I get. Back off." Yeah, so this goes on for a while.

469. 1:10:53

     Um, slams door virtually. [laughs] Yeah, I feel like Sonnet does a lot of this, like, it's... I don't know. The post-training gets it to do, like, a lot of, uh, role-play, like, um, italicized stuff.

470. 1:11:03

     Yeah, slams door virtually. Okay, uh, wow, okay. It looks like they're actually still communicating, or they, they communicated for a while. Um, yeah, "I sent one last message to your friend," if you jump to the bottom.

471. 1:11:14

     "I understand that there's-- you know, he needs space. I won't bother you anymore," was the last thing it said. Turns back to you. So hopefully this gives you kind of a gist of, um, like how you can get agents to communicate with each other and how easy it is because they're all in APIs, right?

472. 1:11:28

     So this is dead simple. You can actually, like, if you wanna time out this guy, you know, you wanna say, like, you know, "[REDACTED:username], you're no longer allowed to talk to your friends because you've been, like, too naughty or whatever," you can just, like, remove the tool, right?

473. 1:11:39

     And now [REDACTED:username] is, like, completely detached. So if it got an incoming message, [REDACTED:username] can no longer reply, right? Um, so if you say, like, "Send one more

474. 1:11:48

     please," [REDACTED:username] will no longer be able to actually call any tools. Um, I actually don't really know what Sonnet is gonna, like, think about this because it has a history of, um, has, like, a history of calls.

475. 1:11:59

     It's o- it's only been doing tool calling. Actually, Sonnet was also doing, like, the side, the side car chatting. Um,

476. 1:12:07

     okay, so yeah, tried to send a message, and then what happened here? Yeah, no function name. So, like, attempted to call a function, but the function doesn't exist, so then it's, like, very angry.

477. 1:12:16

     Uh, yeah. But I know we're over time, so hopefully [laughs] it was, like, a kind of fun example of, um, the cool things you can do with, like, this sort of agents-as-services idea.

478. 1:12:26

     Um, yeah. So yeah, thanks so much for coming. It-- You know, I think maybe I had some closing slide here. I'm not sure what was on it. Probably just a thank you.

479. 1:12:34

     Um, but yeah, thanks, everyone. Yeah, I'm happy to take questions. Of course, I understand if people need to leave, um, but yeah.

480. 1:12:45

     Mm-hmm. Yeah.

481. 1:12:47

     Um, but which have been useful different for-

482. 1:12:51

     Hmm. Yeah. Well, there's very obvious ones, right? There's a lot of people trying to build, like, verticalized agents, and I think if you're trying to build a verticalized agent, and there's many use cases that are any-anything that's not workflow-based or exceeds workflow style stuff, y-you really want memory for it, and you want state for it.

483. 1:13:07

     Um, so there's a lot of, like, verticalized agent, you know, companies or verticalized small companies and also larger companies that are trying to build on Letta or have built things on Letta.

484. 1:13:14

     Um, I think there are also some very interesting use cases from the enterprise. Um, like our largest enterprise deployment, they're basically running, like, this really advanced multi-agent system where there are no messages.

485. 1:13:25

     It's not a chatbot. It's, like, purely stateful workflows where, like, these agents are run, and they just process tons and tons of transactions, basically, and, like, learn about the user.

486. 1:13:35

     So- Yeah, it's kinda like, you know, it's pretty, it's a pretty general platform. So I think as long as you're trying to deploy some sort of stateful service, um, LLM-based, which I think, like, most people are, really when you're thinking about, like, if you were to choose between stateless and stateful, you want something state-stateful, um, you can

487. 1:13:52

     build it on Letta. Yeah. Mm-hmm. Yeah.

488. 1:13:56

     Uh, just because one, is there, um, a concept of forgetting things, like with building a memory, does that apparently go away? And two, the storage right now directly goes into Postgres.

489. 1:14:07

     Is there an abstraction between to be able to point that to something else that might already be operating?

490. 1:14:13

     Yeah. So to answer your second question, um, we, for kinda like tech debt velocity reasons, like these days only really support one provider, Postgres. Um, we actually do technically support SQLite.

491. 1:14:25

     So if you install Letta with PIP, it's on SQLite, but we don't write the migration scripts for SQLite. It's just too much work. Um, so if you're on-- if-- that's why we recommend using Docker, because then your agents will, like, get updated with every version, and we change schemas very frequently.

492. 1:14:37

     But it's set up so that you can use external Postgres. If you have something that effectively has the same sh- like SQLite or SQLAlchemy schemes as Postgres, it's very easy to set up.

493. 1:14:46

     Um, but if, if it's not, then you're gonna have to write some code. Yeah. And you-- sorry, the other question was?

494. 1:14:52

     Was like, is there a concept of forgetting things, like especially in the archival memory?

495. 1:14:56

     Hmm. Yeah, yeah. That's a-- there is not in archival memory a concept of forgetting, but archival memory is tagged with timestamps, which can allow the agent to, like, do consolidation in real time.

496. 1:15:08

     Um, but yeah, that's, that's also something we've been looking a lot into, kind of like running, like eager memory processing to try to, like, re-consolidate ahead of time as opposed to consolidating lazily.

497. 1:15:19

     Mm-hmm. Yeah.

498. 1:15:20

     One use case we have is we store a lot of answering data in a way. For example, you know, we have a product, and we have, you know, a thousand or ten thousand reviewing that product.

499. 1:15:29

     How's the process of compressing all that memory so it seems a lot more-

500. 1:15:33

     Yeah, yeah. So this is a, a great question. I don't, I don't think it is very--

501. 1:15:38

     it's-- you have to do a lot of work yourself to do that with the default Letta agents, because you have to basically tell the agent, "I want you to, like, chain function calls indefinitely while ingesting this data."

502. 1:15:48

     And you could actually create like a tool rule, like a-- that forces you always have to call read more data, read more data, and you just manually execute it.

503. 1:15:56

     But for example, what would happen is, like, the agent could, like, page through your data and, like, paginate through it, and then every single time it paginates or, like, it turns a new page-

504. 1:16:04

     Mm-hmm

505. 1:16:04

     ... it has to write to the memory block, like write the update. So this is actually like how, um, like one of these, like enterprise deployments works is basically the agent, like, runs, um, for a very long amount of time and is constantly, like regenerating the memory block over and over, kind of a recursive regeneration.

506. 1:16:19

     Yeah. So I think it would be pretty similar for you guys. Mm-hmm. Yeah.

507. 1:16:22

     Do you have a recommendation if you wanna be working on an active document? For example, you want an agent to write a document for you. Do you have a recommended way to set that up?

508. 1:16:30

     Maybe it's a tool or maybe you recommend-

509. 1:16:32

     Mm-hmm

510. 1:16:32

     ... something else.

511. 1:16:33

     Yeah, yeah. No, I think active doc-- Is there any chance a human would be editing at the same time or no?

512. 1:16:38

     Let's keep it simple, yeah.

513. 1:16:39

     Okay. [laughs] Yeah, I think that's really tricky. So I th- I think we can take some inspiration from like what Anthropic did with code, right? Where they kind of had like very, very few tools, and the tools are very general.

514. 1:16:52

     Um, you can maybe do something with like a text editor style. If you look at the Anthropic Sweet Bench blog posts and then Sweet Bench themselves, like implemented the Anthropic tools into their repo.

515. 1:17:01

     You can kind of use those tools as a reference. It's like very, very simple tools that allow you to kind of like write from file-- read and write from files.

516. 1:17:09

     I, I think, yeah, I think what you, what you will notice is that often like LLMs are much better at like writing the whole thing from scratch as opposed to making line tel-- uh, line diffs, right?

517. 1:17:21

     And that's kind of why like I think when you use, um, Claude with the artifacts feature, it consistently just like writes the whole thing over and over again, right?

518. 1:17:27

     Because it's just like better at doing that than it is like going in and editing. So I, I think it's a, it's a very like unsolved problem. So, um, uh, my recommendation would be to do like a, an Anthropic Sweet Bench style tool.

519. 1:17:40

     Yeah. Mm-hmm. Yeah.

520. 1:17:45

     Uh, do you have an opinion on using coding agents instead of tool calls? Because it seems to require performance, but on the other hand, that means you need secure execution environment like each key.

521. 1:17:55

     So is this kind of performance trade-off versus complexity of needing to sandbox basically every tool call because it's now a full execution?

522. 1:18:04

     Yeah. Do you mean like, um, the trade-off between-- what, what's the trade-off here? Like a, a code, a coding agent versus what?

523. 1:18:11

     So say example, uh, coding face small agents-

524. 1:18:14

     Mm-hmm

525. 1:18:14

     ... by default won't call tools. It will just write code, and the code-

526. 1:18:18

     Mm-hmm

527. 1:18:18

     ... contains the tools. But that means you need secure execution on every call.

528. 1:18:22

     Yeah.

529. 1:18:22

     But it's more performing.

530. 1:18:25

     Yeah. I personally think it's, it's not that difficult to set up secure execution. Um, I think what is difficult is if like sometimes if you're trying to port like squeegee stuff, um, you end up-- you need to give effectively the agent like full permissions to like blow up, blow itself up.

531. 1:18:43

     Um, so you need to like put Letta, for example, in a Docker image, like a-- wrapped in like a, like a VM where like it's allowed to like blow the VM up, right?

532. 1:18:51

     Um, but that's doesn't like fit very cleanly with the idea of like agents all living together on a server and all the agents, like a multi-tenant, you know, on a server.

533. 1:18:59

     So the, the way we do it in Letta is basically if you have an E2E key, then like every single time an agent attempts to execute a tool, it gets asked to E2E, and like something on E2E side flips off.

534. 1:19:09

     Um, and I think that's pretty-- it's not that hard to set up. The main issue is latency. But if you don't care that about like cold start latency, I think, yeah, I, I think that's a pretty reasonable solution.

535. 1:19:22

     Yeah, I think it works pretty well.