1
03:48:31,420 --> 03:48:32,160
All right.

2
03:48:32,740 --> 03:48:34,080
It's very nice to be here.

3
03:48:35,560 --> 03:48:37,560
The clicker works, the computer works.

4
03:48:37,860 --> 03:48:39,460
I'm Lance from Anthropic.

5
03:48:39,640 --> 03:48:42,400
I know we've had some issues with the prior keynotes,

6
03:48:42,500 --> 03:48:44,220
so we're set to go.

7
03:48:44,500 --> 03:48:47,120
I'm going to tell two stories about getting the most out of Claude

8
03:48:47,120 --> 03:48:48,940
and exploring Claude's frontier capabilities.

9
03:48:49,600 --> 03:48:51,180
So the first is multi-aging.

10
03:48:51,300 --> 03:48:53,920
This is a blog I wrote with the Fable 5.2 launch,

11
03:48:53,940 --> 03:48:56,120
and I want to kind of tell the story of what happened here.

12
03:48:57,260 --> 03:48:58,660
So for math slash theorem,

13
03:48:58,820 --> 03:49:01,440
it's one of the more kind of famous claims in mathematics.

14
03:49:01,460 --> 03:49:03,500
It's stood and proven for around 350 years,

15
03:49:04,120 --> 03:49:05,740
and it's actually a very simple statement.

16
03:49:05,980 --> 03:49:07,840
Just A to the N equals B to the N,

17
03:49:07,880 --> 03:49:09,520
equals C to the N has no solution

18
03:49:09,520 --> 03:49:11,700
for any positive whole numbers greater than 2.

19
03:49:12,460 --> 03:49:15,120
Now, this deceptively simple kind of statement

20
03:49:15,120 --> 03:49:17,560
has proven to be quite difficult to actually prove

21
03:49:18,040 --> 03:49:22,320
in the mid-90s, Sir Andrew Wiles did prove it.

22
03:49:23,040 --> 03:49:25,640
He put out a proof, and it took him seven years of work.

23
03:49:25,760 --> 03:49:26,820
It was extremely painstaking.

24
03:49:27,500 --> 03:49:31,280
And since that time, a major effort and interest in mathematics

25
03:49:31,280 --> 03:49:32,560
has been formalizing that proof,

26
03:49:32,680 --> 03:49:34,580
or basically computationally checking every step.

27
03:49:34,660 --> 03:49:36,980
So proofs are kind of long chains of logic,

28
03:49:37,060 --> 03:49:39,760
and you actually can computationally check proofs

29
03:49:39,760 --> 03:49:42,760
using different systems, which I'll talk about, for example, using Lean.

30
03:49:44,140 --> 03:49:46,640
So this is just kind of a visual kind of explaining the,

31
03:49:46,640 --> 03:49:49,440
There's a very interesting video showing the effort that he went through.

32
03:49:49,780 --> 03:49:52,560
It was an extremely pain-stricken process to establish this proof.

33
03:49:54,200 --> 03:49:59,200
And in September, we published that we used CLAW to formalize it.

34
03:49:59,400 --> 03:49:59,980
This was the

35
03:50:00,000 --> 03:50:04,560
first very important result that we shared along these lines.

36
03:50:05,140 --> 03:50:07,800
And we used Lean to do it.

37
03:50:07,940 --> 03:50:10,940
So Lean allows you to computationally verify every step in a proof.

38
03:50:11,580 --> 03:50:14,100
Now doing this, though, is extremely painstaking.

39
03:50:14,680 --> 03:50:17,040
We started with a team of down a dozen agents.

40
03:50:17,580 --> 03:50:20,400
And this is kind of just showing a schematic of that team.

41
03:50:20,820 --> 03:50:23,780
Very early on, that team didn't work very effectively.

42
03:50:24,660 --> 03:50:26,840
We had to give it something called ProofToMe.

43
03:50:26,900 --> 03:50:30,320
Now, this is kind of a platform for orchestrating mathematics proofs,

44
03:50:30,320 --> 03:50:34,720
and it helped the agents significantly manage the state of the project.

45
03:50:35,960 --> 03:50:37,760
Lean, you can think about as a harness.

46
03:50:38,220 --> 03:50:41,200
So basically, it provides the agents with a graph that they can query.

47
03:50:41,380 --> 03:50:44,260
So it's basically lots of scaffolding around this problem statement.

48
03:50:44,420 --> 03:50:47,080
You can think of it as like a very domain-specific harness, okay?

49
03:50:48,080 --> 03:50:50,340
Now, this is all setting up what we actually did.

50
03:50:50,660 --> 03:50:52,720
We reported. This was back in September.

51
03:50:53,620 --> 03:50:59,700
the formalization. Now what I want to do for 5.2 is show can we have a much larger team of agents,

52
03:50:59,760 --> 03:51:05,140
so going from around a dozen to a thousand, do this formalization with no domain-specific

53
03:51:05,140 --> 03:51:11,580
harness, so without proof to me, and with no human steering with respect to the solution.

54
03:51:11,800 --> 03:51:15,740
So the report we talked about in September had both this domain-specific harness, proved to me,

55
03:51:15,880 --> 03:51:21,380
and a lot of support from a researcher. This is testing whether or not a large team of FABL2 agents

56
03:51:21,380 --> 03:51:25,480
can autonomously develop this formalization themselves.

57
03:51:26,660 --> 03:51:28,720
So this is just giving you a sense for the harness.

58
03:51:29,020 --> 03:51:31,620
It looks kind of complicated, but it's actually pretty simple.

59
03:51:32,360 --> 03:51:34,500
Each of the 1,000 agents has its own sandbox

60
03:51:34,500 --> 03:51:36,240
with restricted network access.

61
03:51:36,940 --> 03:51:39,460
Most of this is directed on safety and security.

62
03:51:40,320 --> 03:51:41,800
So we have 1,000 agents.

63
03:51:42,000 --> 03:51:43,100
Each have their own sandbox.

64
03:51:43,400 --> 03:51:44,920
They can connect to two different things,

65
03:51:45,140 --> 03:51:46,400
a Git and a SQL database.

66
03:51:46,400 --> 03:51:47,060
That's it.

67
03:51:47,420 --> 03:51:49,320
Those are the core primitives that we give this team.

68
03:51:50,200 --> 03:51:54,580
And we have real-time monitoring looking at every tool call for things like escapes.

69
03:51:55,260 --> 03:51:59,580
We also have offline monitoring looking at runs of all the agents every 10 minutes.

70
03:51:59,920 --> 03:52:01,020
This is kind of the setup.

71
03:52:01,580 --> 03:52:05,120
1,000 agent team, very simple primitives, just Git and a database.

72
03:52:06,100 --> 03:52:08,380
The prompting is pretty simple.

73
03:52:08,820 --> 03:52:12,780
It establishes one lead agent, 999 helper agents.

74
03:52:12,960 --> 03:52:17,920
It gives them awareness of Git and the database and the problem statement.

75
03:52:18,200 --> 03:52:19,180
That's really it.

76
03:52:19,320 --> 03:52:21,820
We want to see if this team could then do this formalization.

77
03:52:23,720 --> 03:52:26,480
What's interesting is this is kind of the story of what the team did.

78
03:52:26,740 --> 03:52:29,560
The lead, very early on, five minutes into the run,

79
03:52:30,420 --> 03:52:32,140
basically created these database tables.

80
03:52:32,880 --> 03:52:35,760
So agents now are sufficiently advanced

81
03:52:35,760 --> 03:52:37,340
that they can actually build primitives

82
03:52:37,340 --> 03:52:38,580
that they need to solve hard problems.

83
03:52:38,780 --> 03:52:39,940
We didn't tell them to these tables.

84
03:52:40,040 --> 03:52:41,860
These are the tables that the agent actually just kind of built.

85
03:52:43,000 --> 03:52:44,960
A table where the lead can announce things,

86
03:52:45,220 --> 03:52:46,880
assign roles to workers.

87
03:52:47,680 --> 03:52:52,220
everyone, you know, agents can post notes, a subtask table, a table that indicates whether

88
03:52:52,220 --> 03:52:56,020
or not things are landed. So the lead just creates these tables to orchestrate this team.

89
03:52:56,920 --> 03:53:03,040
And this is a visualization showing how the agents wrote to and read from these tables over the

90
03:53:03,040 --> 03:53:07,720
course of this run. Now, red on the top is showing writes to the tables, green is showing reads.

91
03:53:08,020 --> 03:53:11,020
What's really interesting is you can see occasionally the lead right into the announce

92
03:53:11,020 --> 03:53:16,180
table. This kind of announces major global, like, kind of announcements about the problem.

93
03:53:17,480 --> 03:53:20,440
it also signs workers, these helpers,

94
03:53:20,500 --> 03:53:23,160
different statements in this theorem as they proceed.

95
03:53:23,660 --> 03:53:26,400
Notes are where they communicate and share kind of progress.

96
03:53:26,880 --> 03:53:29,460
Land is where they kind of actually write the lean code

97
03:53:29,460 --> 03:53:30,840
as part of the proof.

98
03:53:31,200 --> 03:53:33,500
And so the agents all autonomously,

99
03:53:33,580 --> 03:53:34,700
without specific prompting,

100
03:53:35,260 --> 03:53:37,660
use the database in this way to coordinate their work

101
03:53:37,660 --> 03:53:39,800
on this extremely hard long horizon task.

102
03:53:40,180 --> 03:53:41,960
This ran for around 30 hours.

103
03:53:42,900 --> 03:53:46,320
And for particularly challenging parts of this proof,

104
03:53:46,320 --> 03:53:48,500
They actually developed their own abstractions.

105
03:53:48,560 --> 03:53:50,740
This is actually a task tree for this Lagrange tunnel.

106
03:53:50,900 --> 03:53:52,340
This is one of the harder statements in the proof.

107
03:53:52,880 --> 03:53:56,800
And this is actually showing the different dependencies of the tasks within this tree.

108
03:53:57,280 --> 03:54:02,860
And this is actually showing over time how, you know, the progress from, like, in progress to done in red.

109
03:54:03,860 --> 03:54:07,140
So the point of all this is really just to kind of highlight,

110
03:54:07,280 --> 03:54:11,120
agents can solve very long horizon problems without much scaffolding.

111
03:54:11,120 --> 03:54:19,040
This is showing kind of the overall kind of allocation of the agents across these different tasks over the course of this run.

112
03:54:20,380 --> 03:54:30,380
And this kind of shows how the lead and the agents themselves kind of reallocate themselves and kind of reassign themselves to different parts of the problem as it progresses.

113
03:54:30,540 --> 03:54:34,560
And as these statements, there's 60 statements in this proof, are all solved.

114
03:54:34,560 --> 03:54:38,100
So the agents using simple primitives, Git, and the database

115
03:54:38,100 --> 03:54:41,940
can coordinate themselves across this very hard problem

116
03:54:41,940 --> 03:54:43,780
to solve it progressively over time.

117
03:54:44,420 --> 03:54:47,140
That's really the key point, and you can see time progressing here.

118
03:54:47,900 --> 03:54:50,600
From one of our runs, this is one of the notes that the agents left,

119
03:54:50,820 --> 03:54:53,820
basically saying total freeze of the origin, effective now,

120
03:54:54,080 --> 03:54:56,060
magnificent work, all 999 of you.

121
03:54:57,480 --> 03:55:00,680
So basically, what is really interesting about this

122
03:55:00,680 --> 03:55:04,640
is we were able to show this formalization,

123
03:55:04,660 --> 03:55:06,540
which was expected to take many, many years.

124
03:55:07,120 --> 03:55:08,560
Back in September, we showed initially

125
03:55:08,560 --> 03:55:10,580
we could do it with like a dozen agents

126
03:55:10,580 --> 03:55:12,260
with a lot of scaffolding.

127
03:55:12,460 --> 03:55:13,580
This proved to be a harness

128
03:55:13,580 --> 03:55:16,680
and a lot of poking and support from a researcher.

129
03:55:17,160 --> 03:55:18,620
This new result is showing

130
03:55:19,800 --> 03:55:21,560
we provide a lot of structure

131
03:55:21,560 --> 03:55:23,040
in terms of safety and monitoring,

132
03:55:23,060 --> 03:55:26,340
but the agent team was able to accomplish this task

133
03:55:26,340 --> 03:55:27,880
with very simple primitives,

134
03:55:28,100 --> 03:55:30,240
without a domestic harness in this particular case.

135
03:55:31,280 --> 03:55:34,260
and agents are getting increasingly good at orchestrating themselves

136
03:55:34,260 --> 03:55:37,360
and knowing how to coordinate using these simple primitives

137
03:55:37,360 --> 03:55:39,220
for very challenging long-horizon tasks.

138
03:55:39,340 --> 03:55:40,980
That's really the key kind of observation,

139
03:55:40,980 --> 03:55:42,580
which is very interesting from this work.

140
03:55:44,160 --> 03:55:46,500
Now, I want to shift gears and talk about memory,

141
03:55:46,520 --> 03:55:50,000
which is entirely different when I talk about it in terms of multi-agent,

142
03:55:50,100 --> 03:55:51,940
but it's highlighting another area

143
03:55:51,940 --> 03:55:54,020
where we're seeing Claude gain significant capabilities.

144
03:55:54,200 --> 03:55:56,600
So previously, we talked kind of about Claude's ability

145
03:55:56,600 --> 03:55:59,140
to orchestrate large teams to solve very long-horizon problems.

146
03:55:59,140 --> 03:56:01,180
memory is a different capability.

147
03:56:01,380 --> 03:56:03,240
It's showing that Claude is increasingly capable

148
03:56:03,240 --> 03:56:04,580
to organize this context

149
03:56:04,580 --> 03:56:06,580
to help solve long horizon problems.

150
03:56:08,180 --> 03:56:10,820
So I published an Eng blog on this fairly recently

151
03:56:10,820 --> 03:56:13,040
and this is actually the lead line of the blog.

152
03:56:13,120 --> 03:56:15,680
I think this actually encapsulates the idea of the blog

153
03:56:15,680 --> 03:56:17,680
and like one of the central ideas around memory

154
03:56:17,680 --> 03:56:18,620
I think is very interesting.

155
03:56:19,140 --> 03:56:21,120
And this is from the philosopher Kierkegaard

156
03:56:21,120 --> 03:56:23,060
so life can only be understood backwards

157
03:56:23,060 --> 03:56:24,600
but it must be lived forwards.

158
03:56:24,960 --> 03:56:27,380
This is a fundamental paradox of agent memory

159
03:56:27,380 --> 03:56:30,660
because agents write to memory typically as they go about a task.

160
03:56:30,840 --> 03:56:33,300
They encounter something weird or interesting, they write to memory.

161
03:56:34,120 --> 03:56:38,500
But memory needs to be a coherent map of an agent's knowledge

162
03:56:38,500 --> 03:56:40,460
that agents can look back on and reflect on.

163
03:56:40,780 --> 03:56:43,740
So when they're writing to memory, they're writing kind of myopically.

164
03:56:43,900 --> 03:56:45,540
They're looking greedily within a session,

165
03:56:45,560 --> 03:56:47,500
saying, ah, this seems important for this task at hand.

166
03:56:47,640 --> 03:56:51,060
But memory needs to be something that's globally coherent and not redundant.

167
03:56:51,300 --> 03:56:54,300
So this is like a very tricky paradox of memory.

168
03:56:54,300 --> 03:56:57,900
And frankly, we as humans have the same kind of challenge.

169
03:56:58,540 --> 03:57:01,060
We go about our day, we write to short-term memory.

170
03:57:01,160 --> 03:57:04,040
It's believed that the hippocampus kind of is the place that that's orchestrated.

171
03:57:04,580 --> 03:57:08,980
At night, it's believed that the hippocampus replays to the neocortex

172
03:57:08,980 --> 03:57:10,540
is where long-term knowledge is stored.

173
03:57:11,120 --> 03:57:13,980
So it's believed that actually this dreaming process for humans

174
03:57:13,980 --> 03:57:17,500
helped consolidate the fragments or like the most important aspects

175
03:57:17,500 --> 03:57:19,500
of our daily experience into long-term knowledge.

176
03:57:19,500 --> 03:57:24,540
There's this decomposition between daily memory and kind of long-term knowledge.

177
03:57:24,940 --> 03:57:27,460
And we found agents can benefit from the same decomposition.

178
03:57:27,960 --> 03:57:29,400
That's really the key point of the post.

179
03:57:30,420 --> 03:57:32,680
Now, here's some results kind of telling the story.

180
03:57:32,900 --> 03:57:37,300
So this is actually an open-source continued learning benchmark called CLBench.

181
03:57:38,380 --> 03:57:44,900
And this is showing that different models, as you go from like Sonnet up to like Opus 5.5,

182
03:57:45,120 --> 03:57:46,940
get increasingly good at using memory.

183
03:57:47,040 --> 03:57:48,480
Now, this task is a very simple setup.

184
03:57:48,480 --> 03:57:53,760
it asks Claude to solve 40 questions about a database. In between each question

185
03:57:53,760 --> 03:57:57,460
it's a fresh session and Claude can write to notes like a memory directory

186
03:57:57,460 --> 03:58:01,460
between each session. So it's kind of like a set of sequential questions and Claude

187
03:58:01,460 --> 03:58:07,300
can write memories to help with future questions, right? So what happens is the

188
03:58:07,300 --> 03:58:11,500
dark bars are showing when you give Claude the memory store and then the

189
03:58:11,500 --> 03:58:16,340
hash is like without memory, okay? Now what's interesting is with Sonnet 4.6

190
03:58:16,340 --> 03:58:22,340
giving Sonnet memory, you can see in the dark blue, has very little benefit. Can't

191
03:58:22,340 --> 03:58:26,180
really use memory. And we looked in the results and we found Sonnet doesn't

192
03:58:26,720 --> 03:58:31,640
doesn't reliably know when to write to memory. It also doesn't really naturally

193
03:58:31,640 --> 03:58:36,480
read from memory, okay. It's also worth noting that these experiments are done

194
03:58:36,480 --> 03:58:40,360
with no specific memory prompting. It just tells the agent the directory is

195
03:58:40,360 --> 03:58:45,000
available. Use memory if you want. That's really it. So Sonnet 4.6 doesn't natively

196
03:58:45,000 --> 03:58:51,320
memory effectively sonnet 5 does indeed better opus and so on and so basically the models get

197
03:58:51,320 --> 03:58:57,120
better and better with the same simple instruction at using memory and really the key idea here is

198
03:58:57,120 --> 03:59:02,420
that what happens is with stronger models they know intrinsically when to write to memory

199
03:59:03,180 --> 03:59:07,640
and they kind of know intrinsically when to read from memory to retrieve information their

200
03:59:07,640 --> 03:59:11,840
memories are also better they're like they're highly verified statements or predictions about

201
03:59:11,840 --> 03:59:16,800
kind of the state of the database or what may happen next versus earlier models tend to write

202
03:59:17,360 --> 03:59:21,380
lower quality memories. I'll show you a very fun example of this from a different game.

203
03:59:22,120 --> 03:59:29,740
This is from Pokemon. So this is showing you Cloud 3 7's memory directory from Pokemon versus Fable 5.

204
03:59:31,740 --> 03:59:35,820
Here is the key point. The size of each box indicates the

205
03:59:36,460 --> 03:59:39,820
the number of memory files in each of these directories.

206
03:59:40,260 --> 03:59:42,160
Okay, so when we have Claude Pay Pokemon,

207
03:59:42,320 --> 03:59:43,640
we give Claude a memory directory,

208
03:59:43,800 --> 03:59:45,560
just like I did in that prior benchmark, right?

209
03:59:46,080 --> 03:59:47,900
Claude can write whatever it wants that directory

210
03:59:47,900 --> 03:59:48,840
while it plays the game.

211
03:59:49,460 --> 03:59:50,780
What's really interesting is,

212
03:59:51,140 --> 03:59:53,940
Sonnet 3.7, we see it writes a huge amount to memory.

213
03:59:54,340 --> 03:59:56,580
And when you look at the memories, it's indiscriminate.

214
03:59:56,780 --> 03:59:59,120
It's basically writing down all sort of experience.

215
03:59:59,120 --> 03:59:59,800
It creates this

216
04:00:00,000 --> 04:00:04,960
huge, unwieldy, just disgusting memory directory of things that are not really relevant or strategic for gameplay.

217
04:00:05,420 --> 04:00:09,520
And it proceeds much worse in the game than Fable 5.

218
04:00:09,920 --> 04:00:16,820
Fable 5's memory directory is much tighter. It's really strategic insights, tactical knowledge, things you can't just get from game state very easily.

219
04:00:17,100 --> 04:00:23,520
So what we see is better models have much better instincts for salience and what to write to memory.

220
04:00:23,520 --> 04:00:26,100
and you can see this reflected in the size of the memory directories,

221
04:00:26,240 --> 04:00:28,560
it's much tighter, it's much more strategic,

222
04:00:28,800 --> 04:00:32,500
and gameplay, indeed, with Fable 5, gets much further than Sonic 3.7.

223
04:00:32,720 --> 04:00:35,420
So it gives another intuition around the fact that Claude

224
04:00:35,420 --> 04:00:37,280
is much better at writing to memory,

225
04:00:37,480 --> 04:00:40,520
knowing what to write in salience and what's salient,

226
04:00:41,860 --> 04:00:44,420
and also using and reading from memory.

227
04:00:44,500 --> 04:00:47,060
So models are getting more and more effective

228
04:00:47,060 --> 04:00:50,000
at knowing salience, knowing what to write, and also reading.

229
04:00:51,320 --> 04:00:58,860
Now both of these represent what I would consider in my initial framing of like writing to memory during experience

230
04:00:58,860 --> 04:01:02,360
So basically in both of these tasks we let Claude write to memory while plays this game

231
04:01:02,360 --> 04:01:05,920
We let Claude write to memory while it does the CL bench benchmark

232
04:01:06,480 --> 04:01:09,940
But as I mentioned there's also the dreaming side of memories

233
04:01:09,940 --> 04:01:15,180
This is allowing Claude to reflect over memories to fix things and I'll show some interesting examples of that

234
04:01:16,580 --> 04:01:19,500
This is kind of a framing of dreaming

235
04:01:19,500 --> 04:01:25,420
And so you can think about memory in terms of there are transcripts.

236
04:01:25,520 --> 04:01:27,720
Transcripts you can think about as kind of the foundation of memory.

237
04:01:27,980 --> 04:01:31,740
So imagine a memory system could very simply be just transcript search.

238
04:01:32,540 --> 04:01:34,900
Claude writes your transcript as it goes about a task.

239
04:01:35,300 --> 04:01:36,740
Future sessions can read from the transcript.

240
04:01:37,400 --> 04:01:38,940
Of course, transcripts are large.

241
04:01:39,140 --> 04:01:40,100
It's very unwieldy.

242
04:01:40,520 --> 04:01:44,840
We talk a lot about in the blog post the fact that transcripts often contain information that,

243
04:01:44,960 --> 04:01:48,240
you know, a very, very small amount of information in transcripts is actually salient and matters.

244
04:01:48,240 --> 04:01:51,440
So we also allow Claude to do things like writing memory notes.

245
04:01:51,500 --> 04:01:52,440
That's what I just talked about.

246
04:01:52,580 --> 04:01:54,220
As it goes, it can write notes to memory.

247
04:01:54,640 --> 04:01:56,120
But dreaming lets you do more.

248
04:01:56,260 --> 04:01:59,180
Dreaming lets you reflect over transcripts and these written notes

249
04:01:59,180 --> 04:02:01,320
and re-represent them.

250
04:02:01,900 --> 04:02:04,240
Build memory representations like wikis, as an example.

251
04:02:05,000 --> 04:02:07,240
Actually promote certain memories to injected context

252
04:02:07,240 --> 04:02:09,240
and load it preemptively into every future session.

253
04:02:09,740 --> 04:02:14,960
So dreaming is this mechanism where you can kind of process existing transcripts

254
04:02:14,960 --> 04:02:17,380
or existing memory notes and build new things from them,

255
04:02:17,380 --> 04:02:20,100
including representations, injected contexts, and more.

256
04:02:21,520 --> 04:02:23,300
So some fun examples of this.

257
04:02:23,440 --> 04:02:24,940
This is from Pokemon Gameplay.

258
04:02:26,100 --> 04:02:28,300
Dreaming is an effective way to correct mistakes in memory.

259
04:02:28,380 --> 04:02:30,840
So as I talked about, when Claude's going about a task,

260
04:02:30,940 --> 04:02:33,020
it can write incorrect notes to memory.

261
04:02:33,120 --> 04:02:34,160
It makes a mistake.

262
04:02:34,560 --> 04:02:36,760
This happened in this particular game.

263
04:02:39,120 --> 04:02:41,200
Claude incorrectly wrote the location

264
04:02:41,820 --> 04:02:44,180
of a particular landmark in the game.

265
04:02:44,180 --> 04:02:47,820
Now what happened is with this incorrectly written memory,

266
04:02:48,180 --> 04:02:52,240
Claude got trapped in a loop where basically it would mislocalize itself

267
04:02:52,240 --> 04:02:54,580
at this particular point in the game, fall down this trap door,

268
04:02:54,620 --> 04:02:56,700
and it would basically send it back to the prior level.

269
04:02:56,900 --> 04:02:58,220
That's like the high-level intuition.

270
04:02:58,400 --> 04:03:00,100
The details aren't that interesting or important.

271
04:03:00,480 --> 04:03:04,300
The key idea is a poison memory was injected and it's stuck there.

272
04:03:05,040 --> 04:03:06,480
Dreaming can rescue that.

273
04:03:06,620 --> 04:03:09,160
That's really the intuition on why dreaming is extremely powerful

274
04:03:09,160 --> 04:03:11,260
for reflecting over memories and correcting them.

275
04:03:12,440 --> 04:03:14,740
So this is showing some trajectories of the game.

276
04:03:15,060 --> 04:03:18,820
Where orange is memory and dreaming, up is good.

277
04:03:19,040 --> 04:03:21,160
So basically up represents game progress,

278
04:03:21,360 --> 04:03:23,520
and down represents kind of game regression.

279
04:03:23,780 --> 04:03:25,980
And what's interesting is raw memory,

280
04:03:27,580 --> 04:03:30,540
basically without dreaming, we can see crashes out.

281
04:03:31,380 --> 04:03:33,080
An incorrect memory is planted,

282
04:03:33,300 --> 04:03:36,240
and Claude continues to basically regress or fall in the game

283
04:03:36,240 --> 04:03:37,300
when it hits that particular point.

284
04:03:37,480 --> 04:03:38,980
Where dreaming it rescues that mistake,

285
04:03:39,400 --> 04:03:40,620
Claude proceeds to the next level.

286
04:03:41,260 --> 04:03:42,460
That's really the key observation.

287
04:03:42,640 --> 04:03:43,880
And this is a very general principle,

288
04:03:44,340 --> 04:03:46,640
that Claude can use dreaming to correct mistakes in memory

289
04:03:46,640 --> 04:03:47,960
that are otherwise stuck there.

290
04:03:48,080 --> 04:03:51,500
It's a very important way you can kind of correct from errors.

291
04:03:52,280 --> 04:03:54,000
Now, this is another fun example

292
04:03:54,000 --> 04:03:55,700
of building representations in memory.

293
04:03:56,380 --> 04:03:58,500
So there's been a lot of interest in wikis

294
04:03:58,500 --> 04:04:01,000
as kind of general kind of ways

295
04:04:01,000 --> 04:04:02,320
that Claude can organize knowledge.

296
04:04:03,020 --> 04:04:05,840
This is a really fun example from,

297
04:04:06,440 --> 04:04:08,960
we acquired a company, Stainless, back in May.

298
04:04:09,800 --> 04:04:12,420
And we had access to the company corpus,

299
04:04:12,620 --> 04:04:14,800
all its documents and Slack messages and so forth,

300
04:04:14,800 --> 04:04:16,480
and we anonymized things,

301
04:04:16,580 --> 04:04:20,740
and we built kind of a data set from it

302
04:04:20,740 --> 04:04:22,200
which allowed us to test Dreaming.

303
04:04:22,420 --> 04:04:25,180
So what we did was we created kind of an email set

304
04:04:25,180 --> 04:04:27,020
of 25 questions about the company

305
04:04:27,020 --> 04:04:28,760
that can only be answered from this corpus.

306
04:04:29,160 --> 04:04:33,940
We allow Claude to build a wiki from this information

307
04:04:33,940 --> 04:04:35,140
to help it answer questions.

308
04:04:35,860 --> 04:04:38,640
What you see is we compare a raw search tool

309
04:04:38,640 --> 04:04:42,060
That's just like some kind of semantic search over this kind of documentation.

310
04:04:42,760 --> 04:04:45,540
We compare that as a baseline, which does reasonably well.

311
04:04:45,740 --> 04:04:48,420
You can see it's like 53% just for search.

312
04:04:48,900 --> 04:04:52,960
We also test letting Claude build a wiki over this knowledge.

313
04:04:53,260 --> 04:04:56,160
You can see as you push, you add more and more dreaming tokens,

314
04:04:56,480 --> 04:04:59,420
the wiki gets better and better and better allowing Claude to answer questions.

315
04:05:00,080 --> 04:05:03,620
So Claude natively can kind of build up over time

316
04:05:03,620 --> 04:05:08,020
a representation of this kind of corpus of knowledge through a wiki.

317
04:05:08,020 --> 04:05:11,320
And as you add more dreaming compute, the wiki gets more and more useful.

318
04:05:11,520 --> 04:05:16,180
We think there's a really nice mental model to think about how agents can kind of build knowledge over time.

319
04:05:16,340 --> 04:05:20,120
They can be given access to kind of raw corpuses of text, of transcripts,

320
04:05:20,160 --> 04:05:24,780
and they can kind of build these representations that allow them to search those contexts much more easily,

321
04:05:24,920 --> 04:05:27,740
to answer questions and to connect their knowledge in effective ways.

322
04:05:28,120 --> 04:05:33,380
We also find that adding the wiki and the search tool together actually bumps further beyond just raw search.

323
04:05:33,380 --> 04:05:43,660
So it's kind of another intuition that basically Claude can also build representations like wikis in its kind of memory in order to help it solve problems.

324
04:05:44,580 --> 04:05:52,060
And a final example, this is from a different benchmark, but this is showing two different wikis that Claude built.

325
04:05:52,140 --> 04:05:54,460
So this is prior to dreaming and after dreaming.

326
04:05:55,080 --> 04:05:59,820
What we saw is when Claude was building wikis in this particular case, there were some gaps.

327
04:06:00,420 --> 04:06:03,900
And, for example, this initial wiki had a bunch of placeholders

328
04:06:03,900 --> 04:06:05,040
for pages that didn't exist.

329
04:06:05,220 --> 04:06:06,640
This can happen when Claude's kind of building

330
04:06:06,640 --> 04:06:07,800
different memory representations.

331
04:06:08,260 --> 04:06:10,580
And without Dreaming Compute, there may not be opportunities

332
04:06:10,580 --> 04:06:12,560
to rescue or fix those errors.

333
04:06:13,440 --> 04:06:15,240
What we saw is when we ran Dreaming,

334
04:06:15,500 --> 04:06:18,420
Dreaming can look over Claude's existing representation,

335
04:06:18,620 --> 04:06:21,140
like, for example, a wiki, and can fill in mistakes.

336
04:06:21,240 --> 04:06:23,180
So in this case, it fills in all the missing gaps.

337
04:06:23,220 --> 04:06:24,200
It creates an index.

338
04:06:24,380 --> 04:06:27,360
And so, again, Dreaming is a way that Claude can reflect

339
04:06:27,360 --> 04:06:30,620
and kind of apply compute across its memory to improve it

340
04:06:30,620 --> 04:06:32,600
by correcting mistakes, like I talked about,

341
04:06:32,920 --> 04:06:36,120
by filling in gaps in its knowledge, like we see here,

342
04:06:36,820 --> 04:06:39,540
or, for example, by creating new representations like wikis.

343
04:06:41,100 --> 04:06:44,240
So with that, I thank you all for listening,

344
04:06:44,460 --> 04:06:48,400
and I'll be at the Anthropic booth,

345
04:06:48,500 --> 04:06:51,020
so please come see me and happy to talk about any of this.