# Exploring Claude's Frontier

Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ

Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session.

[03:48:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13711s) All right.

[03:48:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13712s) It's very nice to be here.

[03:48:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13715s) The clicker works, the computer works.

[03:48:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13717s) I'm Lance from Anthropic.

[03:48:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13719s) I know we've had some issues with the prior keynotes,

[03:48:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13722s) so we're set to go.

[03:48:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13724s) I'm going to tell two stories about getting the most out of Claude

[03:48:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13727s) and exploring Claude's frontier capabilities.

[03:48:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13729s) So the first is multi-aging.

[03:48:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13731s) This is a blog I wrote with the Fable 5.2 launch,

[03:48:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13733s) and I want to kind of tell the story of what happened here.

[03:48:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13737s) So for math slash theorem,

[03:48:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13738s) it's one of the more kind of famous claims in mathematics.

[03:49:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13741s) It's stood and proven for around 350 years,

[03:49:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13744s) and it's actually a very simple statement.

[03:49:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13745s) Just A to the N equals B to the N,

[03:49:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13747s) equals C to the N has no solution

[03:49:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13749s) for any positive whole numbers greater than 2.

[03:49:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13752s) Now, this deceptively simple kind of statement

[03:49:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13755s) has proven to be quite difficult to actually prove

[03:49:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13758s) in the mid-90s, Sir Andrew Wiles did prove it.

[03:49:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13763s) He put out a proof, and it took him seven years of work.

[03:49:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13765s) It was extremely painstaking.

[03:49:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13767s) And since that time, a major effort and interest in mathematics

[03:49:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13771s) has been formalizing that proof,

[03:49:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13772s) or basically computationally checking every step.

[03:49:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13774s) So proofs are kind of long chains of logic,

[03:49:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13777s) and you actually can computationally check proofs

[03:49:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13779s) using different systems, which I'll talk about, for example, using Lean.

[03:49:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13784s) So this is just kind of a visual kind of explaining the,

[03:49:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13786s) There's a very interesting video showing the effort that he went through.

[03:49:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13789s) It was an extremely pain-stricken process to establish this proof.

[03:49:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13794s) And in September, we published that we used CLAW to formalize it.

[03:49:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13799s) This was the

[03:50:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13800s) first very important result that we shared along these lines.

[03:50:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13805s) And we used Lean to do it.

[03:50:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13807s) So Lean allows you to computationally verify every step in a proof.

[03:50:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13811s) Now doing this, though, is extremely painstaking.

[03:50:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13814s) We started with a team of down a dozen agents.

[03:50:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13817s) And this is kind of just showing a schematic of that team.

[03:50:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13820s) Very early on, that team didn't work very effectively.

[03:50:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13824s) We had to give it something called ProofToMe.

[03:50:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13826s) Now, this is kind of a platform for orchestrating mathematics proofs,

[03:50:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13830s) and it helped the agents significantly manage the state of the project.

[03:50:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13835s) Lean, you can think about as a harness.

[03:50:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13838s) So basically, it provides the agents with a graph that they can query.

[03:50:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13841s) So it's basically lots of scaffolding around this problem statement.

[03:50:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13844s) You can think of it as like a very domain-specific harness, okay?

[03:50:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13848s) Now, this is all setting up what we actually did.

[03:50:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13850s) We reported. This was back in September.

[03:50:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13853s) the formalization. Now what I want to do for 5.2 is show can we have a much larger team of agents,

[03:50:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13859s) so going from around a dozen to a thousand, do this formalization with no domain-specific

[03:51:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13865s) harness, so without proof to me, and with no human steering with respect to the solution.

[03:51:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13871s) So the report we talked about in September had both this domain-specific harness, proved to me,

[03:51:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13875s) and a lot of support from a researcher. This is testing whether or not a large team of FABL2 agents

[03:51:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13881s) can autonomously develop this formalization themselves.

[03:51:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13886s) So this is just giving you a sense for the harness.

[03:51:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13889s) It looks kind of complicated, but it's actually pretty simple.

[03:51:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13892s) Each of the 1,000 agents has its own sandbox

[03:51:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13894s) with restricted network access.

[03:51:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13896s) Most of this is directed on safety and security.

[03:51:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13900s) So we have 1,000 agents.

[03:51:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13902s) Each have their own sandbox.

[03:51:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13903s) They can connect to two different things,

[03:51:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13905s) a Git and a SQL database.

[03:51:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13906s) That's it.

[03:51:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13907s) Those are the core primitives that we give this team.

[03:51:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13910s) And we have real-time monitoring looking at every tool call for things like escapes.

[03:51:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13915s) We also have offline monitoring looking at runs of all the agents every 10 minutes.

[03:51:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13919s) This is kind of the setup.

[03:52:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13921s) 1,000 agent team, very simple primitives, just Git and a database.

[03:52:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13926s) The prompting is pretty simple.

[03:52:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13928s) It establishes one lead agent, 999 helper agents.

[03:52:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13932s) It gives them awareness of Git and the database and the problem statement.

[03:52:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13938s) That's really it.

[03:52:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13939s) We want to see if this team could then do this formalization.

[03:52:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13943s) What's interesting is this is kind of the story of what the team did.

[03:52:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13946s) The lead, very early on, five minutes into the run,

[03:52:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13950s) basically created these database tables.

[03:52:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13952s) So agents now are sufficiently advanced

[03:52:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13955s) that they can actually build primitives

[03:52:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13957s) that they need to solve hard problems.

[03:52:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13958s) We didn't tell them to these tables.

[03:52:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13960s) These are the tables that the agent actually just kind of built.

[03:52:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13963s) A table where the lead can announce things,

[03:52:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13965s) assign roles to workers.

[03:52:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13967s) everyone, you know, agents can post notes, a subtask table, a table that indicates whether

[03:52:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13972s) or not things are landed. So the lead just creates these tables to orchestrate this team.

[03:52:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13976s) And this is a visualization showing how the agents wrote to and read from these tables over the

[03:53:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13983s) course of this run. Now, red on the top is showing writes to the tables, green is showing reads.

[03:53:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13988s) What's really interesting is you can see occasionally the lead right into the announce

[03:53:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13991s) table. This kind of announces major global, like, kind of announcements about the problem.

[03:53:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=13997s) it also signs workers, these helpers,

[03:53:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14000s) different statements in this theorem as they proceed.

[03:53:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14003s) Notes are where they communicate and share kind of progress.

[03:53:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14006s) Land is where they kind of actually write the lean code

[03:53:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14009s) as part of the proof.

[03:53:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14011s) And so the agents all autonomously,

[03:53:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14013s) without specific prompting,

[03:53:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14015s) use the database in this way to coordinate their work

[03:53:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14017s) on this extremely hard long horizon task.

[03:53:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14020s) This ran for around 30 hours.

[03:53:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14022s) And for particularly challenging parts of this proof,

[03:53:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14026s) They actually developed their own abstractions.

[03:53:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14028s) This is actually a task tree for this Lagrange tunnel.

[03:53:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14030s) This is one of the harder statements in the proof.

[03:53:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14032s) And this is actually showing the different dependencies of the tasks within this tree.

[03:53:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14037s) And this is actually showing over time how, you know, the progress from, like, in progress to done in red.

[03:54:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14043s) So the point of all this is really just to kind of highlight,

[03:54:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14047s) agents can solve very long horizon problems without much scaffolding.

[03:54:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14051s) This is showing kind of the overall kind of allocation of the agents across these different tasks over the course of this run.

[03:54:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14060s) And this kind of shows how the lead and the agents themselves kind of reallocate themselves and kind of reassign themselves to different parts of the problem as it progresses.

[03:54:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14070s) And as these statements, there's 60 statements in this proof, are all solved.

[03:54:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14074s) So the agents using simple primitives, Git, and the database

[03:54:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14078s) can coordinate themselves across this very hard problem

[03:54:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14081s) to solve it progressively over time.

[03:54:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14084s) That's really the key point, and you can see time progressing here.

[03:54:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14087s) From one of our runs, this is one of the notes that the agents left,

[03:54:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14090s) basically saying total freeze of the origin, effective now,

[03:54:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14094s) magnificent work, all 999 of you.

[03:54:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14097s) So basically, what is really interesting about this

[03:55:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14100s) is we were able to show this formalization,

[03:55:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14104s) which was expected to take many, many years.

[03:55:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14107s) Back in September, we showed initially

[03:55:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14108s) we could do it with like a dozen agents

[03:55:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14110s) with a lot of scaffolding.

[03:55:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14112s) This proved to be a harness

[03:55:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14113s) and a lot of poking and support from a researcher.

[03:55:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14117s) This new result is showing

[03:55:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14119s) we provide a lot of structure

[03:55:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14121s) in terms of safety and monitoring,

[03:55:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14123s) but the agent team was able to accomplish this task

[03:55:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14126s) with very simple primitives,

[03:55:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14128s) without a domestic harness in this particular case.

[03:55:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14131s) and agents are getting increasingly good at orchestrating themselves

[03:55:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14134s) and knowing how to coordinate using these simple primitives

[03:55:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14137s) for very challenging long-horizon tasks.

[03:55:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14139s) That's really the key kind of observation,

[03:55:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14140s) which is very interesting from this work.

[03:55:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14144s) Now, I want to shift gears and talk about memory,

[03:55:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14146s) which is entirely different when I talk about it in terms of multi-agent,

[03:55:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14150s) but it's highlighting another area

[03:55:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14151s) where we're seeing Claude gain significant capabilities.

[03:55:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14154s) So previously, we talked kind of about Claude's ability

[03:55:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14156s) to orchestrate large teams to solve very long-horizon problems.

[03:55:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14159s) memory is a different capability.

[03:56:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14161s) It's showing that Claude is increasingly capable

[03:56:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14163s) to organize this context

[03:56:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14164s) to help solve long horizon problems.

[03:56:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14168s) So I published an Eng blog on this fairly recently

[03:56:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14170s) and this is actually the lead line of the blog.

[03:56:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14173s) I think this actually encapsulates the idea of the blog

[03:56:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14175s) and like one of the central ideas around memory

[03:56:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14177s) I think is very interesting.

[03:56:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14179s) And this is from the philosopher Kierkegaard

[03:56:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14181s) so life can only be understood backwards

[03:56:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14183s) but it must be lived forwards.

[03:56:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14184s) This is a fundamental paradox of agent memory

[03:56:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14187s) because agents write to memory typically as they go about a task.

[03:56:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14190s) They encounter something weird or interesting, they write to memory.

[03:56:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14194s) But memory needs to be a coherent map of an agent's knowledge

[03:56:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14198s) that agents can look back on and reflect on.

[03:56:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14200s) So when they're writing to memory, they're writing kind of myopically.

[03:56:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14203s) They're looking greedily within a session,

[03:56:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14205s) saying, ah, this seems important for this task at hand.

[03:56:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14207s) But memory needs to be something that's globally coherent and not redundant.

[03:56:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14211s) So this is like a very tricky paradox of memory.

[03:56:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14214s) And frankly, we as humans have the same kind of challenge.

[03:56:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14218s) We go about our day, we write to short-term memory.

[03:57:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14221s) It's believed that the hippocampus kind of is the place that that's orchestrated.

[03:57:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14224s) At night, it's believed that the hippocampus replays to the neocortex

[03:57:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14228s) is where long-term knowledge is stored.

[03:57:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14231s) So it's believed that actually this dreaming process for humans

[03:57:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14233s) helped consolidate the fragments or like the most important aspects

[03:57:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14237s) of our daily experience into long-term knowledge.

[03:57:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14239s) There's this decomposition between daily memory and kind of long-term knowledge.

[03:57:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14244s) And we found agents can benefit from the same decomposition.

[03:57:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14247s) That's really the key point of the post.

[03:57:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14250s) Now, here's some results kind of telling the story.

[03:57:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14252s) So this is actually an open-source continued learning benchmark called CLBench.

[03:57:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14258s) And this is showing that different models, as you go from like Sonnet up to like Opus 5.5,

[03:57:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14265s) get increasingly good at using memory.

[03:57:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14267s) Now, this task is a very simple setup.

[03:57:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14268s) it asks Claude to solve 40 questions about a database. In between each question

[03:57:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14273s) it's a fresh session and Claude can write to notes like a memory directory

[03:57:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14277s) between each session. So it's kind of like a set of sequential questions and Claude

[03:58:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14281s) can write memories to help with future questions, right? So what happens is the

[03:58:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14287s) dark bars are showing when you give Claude the memory store and then the

[03:58:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14291s) hash is like without memory, okay? Now what's interesting is with Sonnet 4.6

[03:58:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14296s) giving Sonnet memory, you can see in the dark blue, has very little benefit. Can't

[03:58:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14302s) really use memory. And we looked in the results and we found Sonnet doesn't

[03:58:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14306s) doesn't reliably know when to write to memory. It also doesn't really naturally

[03:58:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14311s) read from memory, okay. It's also worth noting that these experiments are done

[03:58:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14316s) with no specific memory prompting. It just tells the agent the directory is

[03:58:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14320s) available. Use memory if you want. That's really it. So Sonnet 4.6 doesn't natively

[03:58:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14325s) memory effectively sonnet 5 does indeed better opus and so on and so basically the models get

[03:58:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14331s) better and better with the same simple instruction at using memory and really the key idea here is

[03:58:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14337s) that what happens is with stronger models they know intrinsically when to write to memory

[03:59:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14343s) and they kind of know intrinsically when to read from memory to retrieve information their

[03:59:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14347s) memories are also better they're like they're highly verified statements or predictions about

[03:59:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14351s) kind of the state of the database or what may happen next versus earlier models tend to write

[03:59:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14357s) lower quality memories. I'll show you a very fun example of this from a different game.

[03:59:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14362s) This is from Pokemon. So this is showing you Cloud 3 7's memory directory from Pokemon versus Fable 5.

[03:59:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14371s) Here is the key point. The size of each box indicates the

[03:59:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14376s) the number of memory files in each of these directories.

[03:59:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14380s) Okay, so when we have Claude Pay Pokemon,

[03:59:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14382s) we give Claude a memory directory,

[03:59:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14383s) just like I did in that prior benchmark, right?

[03:59:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14386s) Claude can write whatever it wants that directory

[03:59:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14387s) while it plays the game.

[03:59:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14389s) What's really interesting is,

[03:59:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14391s) Sonnet 3.7, we see it writes a huge amount to memory.

[03:59:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14394s) And when you look at the memories, it's indiscriminate.

[03:59:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14396s) It's basically writing down all sort of experience.

[03:59:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14399s) It creates this

[04:00:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14400s) huge, unwieldy, just disgusting memory directory of things that are not really relevant or strategic for gameplay.

[04:00:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14405s) And it proceeds much worse in the game than Fable 5.

[04:00:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14409s) Fable 5's memory directory is much tighter. It's really strategic insights, tactical knowledge, things you can't just get from game state very easily.

[04:00:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14417s) So what we see is better models have much better instincts for salience and what to write to memory.

[04:00:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14423s) and you can see this reflected in the size of the memory directories,

[04:00:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14426s) it's much tighter, it's much more strategic,

[04:00:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14428s) and gameplay, indeed, with Fable 5, gets much further than Sonic 3.7.

[04:00:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14432s) So it gives another intuition around the fact that Claude

[04:00:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14435s) is much better at writing to memory,

[04:00:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14437s) knowing what to write in salience and what's salient,

[04:00:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14441s) and also using and reading from memory.

[04:00:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14444s) So models are getting more and more effective

[04:00:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14447s) at knowing salience, knowing what to write, and also reading.

[04:00:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14451s) Now both of these represent what I would consider in my initial framing of like writing to memory during experience

[04:00:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14458s) So basically in both of these tasks we let Claude write to memory while plays this game

[04:01:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14462s) We let Claude write to memory while it does the CL bench benchmark

[04:01:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14466s) But as I mentioned there's also the dreaming side of memories

[04:01:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14469s) This is allowing Claude to reflect over memories to fix things and I'll show some interesting examples of that

[04:01:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14476s) This is kind of a framing of dreaming

[04:01:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14479s) And so you can think about memory in terms of there are transcripts.

[04:01:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14485s) Transcripts you can think about as kind of the foundation of memory.

[04:01:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14487s) So imagine a memory system could very simply be just transcript search.

[04:01:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14492s) Claude writes your transcript as it goes about a task.

[04:01:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14495s) Future sessions can read from the transcript.

[04:01:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14497s) Of course, transcripts are large.

[04:01:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14499s) It's very unwieldy.

[04:01:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14500s) We talk a lot about in the blog post the fact that transcripts often contain information that,

[04:01:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14504s) you know, a very, very small amount of information in transcripts is actually salient and matters.

[04:01:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14508s) So we also allow Claude to do things like writing memory notes.

[04:01:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14511s) That's what I just talked about.

[04:01:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14512s) As it goes, it can write notes to memory.

[04:01:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14514s) But dreaming lets you do more.

[04:01:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14516s) Dreaming lets you reflect over transcripts and these written notes

[04:01:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14519s) and re-represent them.

[04:02:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14521s) Build memory representations like wikis, as an example.

[04:02:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14525s) Actually promote certain memories to injected context

[04:02:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14527s) and load it preemptively into every future session.

[04:02:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14529s) So dreaming is this mechanism where you can kind of process existing transcripts

[04:02:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14534s) or existing memory notes and build new things from them,

[04:02:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14537s) including representations, injected contexts, and more.

[04:02:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14541s) So some fun examples of this.

[04:02:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14543s) This is from Pokemon Gameplay.

[04:02:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14546s) Dreaming is an effective way to correct mistakes in memory.

[04:02:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14548s) So as I talked about, when Claude's going about a task,

[04:02:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14550s) it can write incorrect notes to memory.

[04:02:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14553s) It makes a mistake.

[04:02:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14554s) This happened in this particular game.

[04:02:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14559s) Claude incorrectly wrote the location

[04:02:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14561s) of a particular landmark in the game.

[04:02:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14564s) Now what happened is with this incorrectly written memory,

[04:02:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14568s) Claude got trapped in a loop where basically it would mislocalize itself

[04:02:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14572s) at this particular point in the game, fall down this trap door,

[04:02:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14574s) and it would basically send it back to the prior level.

[04:02:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14576s) That's like the high-level intuition.

[04:02:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14578s) The details aren't that interesting or important.

[04:03:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14580s) The key idea is a poison memory was injected and it's stuck there.

[04:03:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14585s) Dreaming can rescue that.

[04:03:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14586s) That's really the intuition on why dreaming is extremely powerful

[04:03:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14589s) for reflecting over memories and correcting them.

[04:03:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14592s) So this is showing some trajectories of the game.

[04:03:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14595s) Where orange is memory and dreaming, up is good.

[04:03:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14599s) So basically up represents game progress,

[04:03:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14601s) and down represents kind of game regression.

[04:03:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14603s) And what's interesting is raw memory,

[04:03:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14607s) basically without dreaming, we can see crashes out.

[04:03:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14611s) An incorrect memory is planted,

[04:03:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14613s) and Claude continues to basically regress or fall in the game

[04:03:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14616s) when it hits that particular point.

[04:03:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14617s) Where dreaming it rescues that mistake,

[04:03:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14619s) Claude proceeds to the next level.

[04:03:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14621s) That's really the key observation.

[04:03:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14622s) And this is a very general principle,

[04:03:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14624s) that Claude can use dreaming to correct mistakes in memory

[04:03:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14626s) that are otherwise stuck there.

[04:03:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14628s) It's a very important way you can kind of correct from errors.

[04:03:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14632s) Now, this is another fun example

[04:03:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14634s) of building representations in memory.

[04:03:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14636s) So there's been a lot of interest in wikis

[04:03:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14638s) as kind of general kind of ways

[04:04:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14641s) that Claude can organize knowledge.

[04:04:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14643s) This is a really fun example from,

[04:04:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14646s) we acquired a company, Stainless, back in May.

[04:04:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14649s) And we had access to the company corpus,

[04:04:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14652s) all its documents and Slack messages and so forth,

[04:04:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14654s) and we anonymized things,

[04:04:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14656s) and we built kind of a data set from it

[04:04:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14660s) which allowed us to test Dreaming.

[04:04:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14662s) So what we did was we created kind of an email set

[04:04:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14665s) of 25 questions about the company

[04:04:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14667s) that can only be answered from this corpus.

[04:04:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14669s) We allow Claude to build a wiki from this information

[04:04:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14673s) to help it answer questions.

[04:04:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14675s) What you see is we compare a raw search tool

[04:04:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14678s) That's just like some kind of semantic search over this kind of documentation.

[04:04:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14682s) We compare that as a baseline, which does reasonably well.

[04:04:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14685s) You can see it's like 53% just for search.

[04:04:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14688s) We also test letting Claude build a wiki over this knowledge.

[04:04:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14693s) You can see as you push, you add more and more dreaming tokens,

[04:04:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14696s) the wiki gets better and better and better allowing Claude to answer questions.

[04:05:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14700s) So Claude natively can kind of build up over time

[04:05:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14703s) a representation of this kind of corpus of knowledge through a wiki.

[04:05:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14708s) And as you add more dreaming compute, the wiki gets more and more useful.

[04:05:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14711s) We think there's a really nice mental model to think about how agents can kind of build knowledge over time.

[04:05:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14716s) They can be given access to kind of raw corpuses of text, of transcripts,

[04:05:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14720s) and they can kind of build these representations that allow them to search those contexts much more easily,

[04:05:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14724s) to answer questions and to connect their knowledge in effective ways.

[04:05:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14728s) We also find that adding the wiki and the search tool together actually bumps further beyond just raw search.

[04:05:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14733s) So it's kind of another intuition that basically Claude can also build representations like wikis in its kind of memory in order to help it solve problems.

[04:05:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14744s) And a final example, this is from a different benchmark, but this is showing two different wikis that Claude built.

[04:05:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14752s) So this is prior to dreaming and after dreaming.

[04:05:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14755s) What we saw is when Claude was building wikis in this particular case, there were some gaps.

[04:06:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14760s) And, for example, this initial wiki had a bunch of placeholders

[04:06:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14763s) for pages that didn't exist.

[04:06:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14765s) This can happen when Claude's kind of building

[04:06:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14766s) different memory representations.

[04:06:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14768s) And without Dreaming Compute, there may not be opportunities

[04:06:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14770s) to rescue or fix those errors.

[04:06:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14773s) What we saw is when we ran Dreaming,

[04:06:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14775s) Dreaming can look over Claude's existing representation,

[04:06:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14778s) like, for example, a wiki, and can fill in mistakes.

[04:06:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14781s) So in this case, it fills in all the missing gaps.

[04:06:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14783s) It creates an index.

[04:06:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14784s) And so, again, Dreaming is a way that Claude can reflect

[04:06:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14787s) and kind of apply compute across its memory to improve it

[04:06:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14790s) by correcting mistakes, like I talked about,

[04:06:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14792s) by filling in gaps in its knowledge, like we see here,

[04:06:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14796s) or, for example, by creating new representations like wikis.

[04:06:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14801s) So with that, I thank you all for listening,

[04:06:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14804s) and I'll be at the Anthropic booth,

[04:06:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=14808s) so please come see me and happy to talk about any of this.