Lessons from Studying Every Memory System

Talk transcript

1. 0:04

   [upbeat music] Okay. Uh, hi everyone. I'm Shlok, and I've spent the past year studying different memory systems.

2. 0:23

   Now, before I get started, one thing I've realized speaking to people over the last two days is that memory is a very overloaded term now. It can mean a lot of different things.

3. 0:34

   So when I talk about memory today, it is going to be in the context of personalization, especially for consumer AI applications.

4. 0:43

   Now, a little bit about me. My claim to fame, the reason I get to speak to you here, is that I've spent the past year trying to reverse engineer how products like ChatGPT, Claude, Gemini, and Poe implement their memory systems.

5. 0:58

   And I've then worked with multiple teams across different domains in helping them design their memory.

6. 1:05

   I'm gonna break the talk down into two parts. Uh, first, we're gonna look at how memory has evolved over the past three years, especially in the context of ChatGPT and Claude.

7. 1:17

   And then in part two, I'm gonna discuss some of the lessons I've learned, maybe a rant, and, uh, where I think all of this is going.

8. 1:28

   To kick things off, we go back to ancient times, uh, which in our industry is 2023.

9. 1:36

   This is, uh, ChatGPT just after the launch of GPT-4.

10. 1:41

    Now, you could have back-and-forth conversations within a single thread, and context was maintained inside that thread, but as soon as you started a new conversation, nothing was carried over.

11. 1:54

    Now, for early adopters, this wasn't a problem. GPT-4 was such an amazing model that if we ever had the need to carry context, we would do so by hand.

12. 2:05

    But as ChatGPT started becoming more popular, as regular people started using it for things like learning, cooking, uh, as a companion, the need for some sort of memory system became really apparent.

13. 2:20

    So in February of 2024, we got ChatGPT memory V1,

14. 2:26

    and what you could do is you could ask ChatGPT to remember things about you. So you could say things like, "Hey, remember that I'm vegetarian." And ChatGPT would extract what it thought was a fact, which is that the user is vegetarian, store it in a list of memories, and this list was then added to the context window

15. 2:47

    for every single conversation. You could also then go into settings and view this list of memories, and if you thought that something didn't apply anymore, you could delete a memory.

16. 3:00

    Now, as the first serious memory implementation within our industry, I think this was a really decent effort, but there were also some fundamental flaws with it.

17. 3:11

    The biggest one was that as a user, because you could see every time a memory was created, it felt like you were responsible for both creating memories while you were just trying to have a conversation.

18. 3:22

    So the burden of memory management fell to the user.

19. 3:26

    Also, if you notice this list of memories here,

20. 3:30

    these held true at the time they were being created, but that doesn't necessarily hold true over time. For example, it says that Shlok is going to Bengaluru. Now, I obviously am in SF right now.

21. 3:42

    I am not going to Bengaluru, but this fact, this memory, is still added to my context window today. So staleness was another huge problem with this version of ChatGPT's memory.

22. 3:56

    A little more than a year later, April of 2025, ChatGPT released V2 of its memory, and this was a little more sophisticated.

23. 4:06

    The most important addition was this thing called user knowledge memories.

24. 4:11

    Uh, I'm just going to call it a running profile for the rest of this talk, and what a running profile really is, is that every few days, ChatGPT looks at all the conversations you've had with it, it extracts anything it thinks is important for it to know about you, and it updates this profile that it maintains on

25. 4:28

    you. Now, this updation process is also what a bunch of folks called dream, uh, now call dreaming. Uh, how many of you all were there for Lance Martin's talk yesterday?

26. 4:41

    Okay, not many, but he did a great talk on this.

27. 4:45

    So every few days, ChatGPT looks at the new conversations you're having, uh, updates your profile, and then this updated profile is added to the context window for every single new conversation.

28. 4:58

    These are two excerpts from my running profile. I want you to notice a few things. First, these are extremely dense memories, so ChatGPT tries to pack in as much context as it can within e- every single memory.

29. 5:13

    What's, what's essentially happening here is that they're trying to put in keywords almost like clues, and because LLM, especially the frontier models today, are so good at inferring context from limited information, when you're having a conversation, it connects these clues to what you're talking about.

30. 5:30

    Also, these are just two of sixteen different sections in my profile. Other sections include my personal life, things I'm working on. Uh, in total, my profile is almost four thousand tokens long.

31. 5:41

    And because these updates are happening, happening asynchronously, they're happening in the background, this new version does away with the flaw we discussed in V1, which is the burden of user management was taken a-- The burden of memory management was taken away from the user.

32. 5:56

    But I want you to notice the highlighted memory. This is about places I traveled to in 2025, but if you pay attention, it says Thailand and Turkey, but the dates are overlapping.

33. 6:08

    And that's because the source of this memory was conversations I was having with ChatGPT deciding between where to go bet- among these two places. Now, I did end up going to Thailand.

34. 6:18

    I've never been to Turkey, but ChatGPT still says that I've been to Turkey in twenty twenty-five. So the stainless problem with V2 didn't completely go away.

35. 6:29

    Another very important thing is that if you go to your settings, ChatGPT doesn't let you view this raw profile. So you could view your memories from V1. Your raw profile is not visible to you.

36. 6:41

    Now, you may ask, Shlok, how did you see your profile then? That's because this prompt works really well if you want to jailbreak ChatGPT, uh, and view your raw profile.

37. 6:50

    You might have to attempt a few times, try different thinking modes, but prod enough, and you shall receive.

38. 6:59

    August of twenty twenty-five, uh, Claude released its first version of Memory. This surprised me a bit because if you compare ChatGPT and Claude, they are very similar applications, right?

39. 7:10

    You have a chat box, you have back-and-forth conversations. You have a list of previous conversations. You can start a new conversation. My assumption going into study Claude was that the memory systems would also be similarly designed.

40. 7:24

    Not the case, at least for V1. So in V1 of Claude, you had no user profile, you had no list of facts. Instead, the model was given two tools.

41. 7:35

    It was given a tool to search over previous conversations by keyword or topic, and it was given another tool to search over conversations by time period. So queries like, "What did we discuss last week?"

42. 7:45

    Or, "What did we discuss at the start of November of twenty twenty-five?"

43. 7:49

    So in V1, every single context-- every single conversation starts fresh with no context on the user, and when the model thinks that it needs to retrieve something, it can do so on demand.

44. 8:04

    Uh, on September eleven of last year, I released a blog post saying Claude's memory architecture is the opposite of ChatGPT's. This hit the Hacker News front page. Funnily, on that very day, Claude released V2 of its memory

45. 8:20

    and, uh, they added a running profile similar to ChatGPT, but with a few differences. First, Claude made this profile visible to users. So you could go to settings, and you could view your raw profile.

46. 8:34

    Second, this profile was a thousand tokens, so it was much smaller than ChatGPT's four thousand tokens. And also, if you notice, these are complete sentences rather than a dense keyword approach of ChatGPT.

47. 8:45

    So less dense and smaller. Claude's profile updates every twenty-four hours. For ChatGPT, it's every few days. And Claude also let you-- let the user make explicit edits to this profile.

48. 8:58

    So you could request for an edit, and that edit would le-lead to a resynthesis of the profile.

49. 9:04

    It gave you an interface to manage previous edits, and you could delete the things that no longer held true. And this is how Claude's memory works even today, so it hasn't changed since September of last year.

50. 9:17

    We have seen two updates within ChatGPT's memory this year, though.

51. 9:21

    The first was it added a tool to look over past conversations, like we just saw with Claude. Uh, so the model can retrieve summarized context based on queries it makes.

52. 9:33

    And then a month ago, uh, start of June, ChatGPT finally made user profile visible to them, somewhat. So what you can see is a LLM-generated summary of your profile, which is weird because your profile is already an LLM ge-generated summary of your conversations.

53. 9:53

    Uh, it's all a bit confusing. I've written about it, but it is visible in some sense.

54. 9:58

    You could also ma-- request explicit edits to your profile, and with this update, ChatGPT deprecated V1, the fact list, from its memory system.

55. 10:09

    So what we've seen here is a convergence after three years of each of these products evolving independently, where they both now have a running profile. This profile is visible and editable, again, somewhat, uh, and the model has tools to look over past conversations.

56. 10:26

    Okay. So what can we learn from this evolution, and where are things going?

57. 10:32

    I think the biggest lesson for me is that there is no single way to do memory.

58. 10:39

    It wasn't too long ago that everyone, including me, assumed that RAG was the way to go, go about memory. Where you would take conversations, you would chunk them, uh, create embeddings, put them in a vector store, and then as user queries came in, do some sort of semantic search.

59. 10:56

    But as we saw, neither ChatGPT nor Claude really do this. Instead, they both evolved independently using different approaches. And while the general architectures have converged, the specific implementation details are still very different.

60. 11:12

    And then if you look at Gemini, it also has a running profile, but each memory comes with detailed timing logs. So when was it created? When was it last updated?

61. 11:21

    And then if you look at agents like Claude Code, OpenClau Hermes, they have completely different memory systems, right? With Markdown files, Heartbeat, knowledge bases, skills. The point being that there is no one way to do memory.

62. 11:34

    The implications of this is that memory cannot be outsourced. If you're a serious team, you do not outsource memory. It is something that you build alongside your product. Your memory system evolves with your product, and it cannot be thought of, uh, as an afterthought.

63. 11:50

    And there is plenty of evidence for this. So if you look at all of the top consumer products today across different categories, each of these has some form of memory,

64. 12:01

    yet none of them outsource it. All of them build memory in-house.

65. 12:08

    Lesson two: Memory is a function of compute. What does that mean?

66. 12:13

    Let's look at the costs associated with a running profile. So there are two types of costs. There is a cost to maintain a profile, and that depends on how frequently you update it and how much compute you apply to each update.

67. 12:27

    And then because these profiles are part of the context window for every single conversation, there's a cost of serving, which is the lo-longer the profile, the more it costs to serve.

68. 12:37

    Now, thought experiment: if you were to design the ideal memory system with no restraints, what would you do?

69. 12:44

    You might want to update, um, your profile every hour or maybe after every conversation.

70. 12:53

    Uh, you might want to task Fable with a bunch of Opus sub-agents for the update itself. Uh, and why stop at four thousand tokens? Why not make it four hundred thousand tokens, store every single thing you would want about the user?

71. 13:08

    Unfortunately, we live, we live in a GPU-constrained world, and trade-offs have to be made, and you can see that happening here. So ChatGPT, the profile length is four thousand tokens.

72. 13:18

    It updates every few days. So they have a higher serving cost for a lower update cost. And for Claude, it's a thousand tokens, updates every twenty-four hours, so they make the exact opposite trade-off.

73. 13:30

    And this is what I mean by memory is a function of compute. You have to really think about how much compute you want to put into memory.

74. 13:37

    Third, uh, we had a bunch of talks about continual learning today. I'm not an expert here, but what I would say is that continual learning is already here.

75. 13:47

    Going back to running profiles, what exactly is happening here? Your running profile starts with something that the model knows about you.

76. 13:55

    This is then applied to every single conversation.

77. 14:00

    Each of these conversations bring in new information.

78. 14:04

    This new information is then synthesized throu-through the trimming process back into the profile, and then this profile dictates further conversations. And this loop keeps repeating itself again and again and again, and what you have is a continual learning process.

79. 14:18

    Now, obv-obviously, this learning loop is happening outside the weights.

80. 14:24

    And a big question, particularly for consumer AI, is will this process ever make its way into the weights? Now, I don't-- Obviously, updating weights, um, training models is an expensive process.

81. 14:39

    Uh, continuous learning does make sense at an en-enterprise level because the costs of these models are amortized across different employees, different customers. But that's not the case at an, at an individual level, right?

82. 14:52

    So big, big open questions that I don't know-- don't yet know the answers to, which is: Will each of us get our own self-learning model?

83. 15:01

    What data do we need to kick the CL process off, and how do we generate it? And finally, who's gonna pay for this? How would the economics for this work?

84. 15:12

    Uh, Gwern recently wrote an essay called "Guardian Angels", uh, where he explores this topic in

85. 15:20

    beautiful detail. And if you're interested in what the future for one model a person looks like, I would recommend reading this.

86. 15:32

    Finally, uh, my rant is that we have a massive context problem.

87. 15:39

    You could have the best memory architecture in the world. You could pour infinite amounts of compute into it. You could have continuous learning working at an individual level where the-- every single data point you bring up is somehow perfectly integrated into the model weights.

88. 15:57

    Yet, your memory system is capped by how much context it can gather about you.

89. 16:03

    Let's go back to the example we discussed earlier, which was the conflict between where I travel to, uh, in the summer of twenty twenty-five. These are the two source conversations.

90. 16:15

    Again, I was trying to use ChatGPT to decide between which of these two countries to go to.

91. 16:21

    Now, the decision to go to Thailand was actually made in a conversation I had with my partner in person, and ChatGPT couldn't, couldn't reason over this or couldn't listen to this.

92. 16:30

    But there were also traces of this conversation in my emails because I, I had flight and hotel bookings for Thailand. But because even if ChatGPT is connected to my email, it doesn't reason over my email, and it doesn't update my profile over my email, it couldn't resolve this conflict.

93. 16:51

    And I think that's okay, uh, it's understandable, but what really bothers me is that

94. 17:00

    ChatGPT today doesn't realize that there is a conflict. It's not curious about trying to fill in gaps in the information it knows about me. And this is particularly interesting and also infuriating because the tech-- it's not a technology problem.

95. 17:18

    It's a product problem. There is no fundamental reason from an LLM level that these things can't be solved. Uh, it's just that our products today are not designed to help us with this.

96. 17:34

    So my personal stack today is a bunch of chatbots, assistants, vertical-specific applications, agents, and even hardware devices.

97. 17:45

    Each of these products is trying to build its own memory of me.

98. 17:49

    None of these memories are shared with each other, so I have to rebuild context within every single product from scratch every time. When something in my life changes, I have to individually update all of them.

99. 18:04

    And then I have a bunch of very rich existing context sto-sources, like my email, calendar, uh, photos. None of these products are able to reason over my existing very rich context sources.

100. 18:21

     So for me, none of this feels like twenty twenty-six, and what I keep asking myself every day is: When will personal AI feel like personal AI?

101. 18:33

     All of that frustration aside, I still think we're very early. Memory for AI is just a three-year-old field.

102. 18:42

     Memory is also foundational to how humans interact with AI. And because I hope to be talking to AIs all my life, and I know that's going to be the case for every single one of us here today, memory is something that's going to be important for the rest of human history.

103. 18:58

     And, uh, there's so much left to build.

104. 19:02

     That's it from me. Uh, thank you so much. You can find my website. You can find me on Twitter. [audience applauding] Have a great rest of the conference. [on hold music]