Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
0:00:00I had some technical difficulties during
0:00:03my talk this morning, so I ran back home
0:00:05and I'm just going to record it from my
0:00:07computer instead. So, hi everyone. I'm
0:00:09Lydia. I work on the Cloud Code team at
0:00:11Anthropic. And the models have gotten
0:00:14really good lately. So we have Opus 55,
0:00:16which is my favorite daily driver. And
0:00:19earlier this week, we also launched
0:00:20Fable 52. And these models are extremely
0:00:23good at long-running tasks and
0:00:25understanding your codebase. And
0:00:27especially Fable is also really good at
0:00:29verifying its own output. But as these
0:00:31models get better, the way that we work
0:00:33with them in Cloud Code also changes.
0:00:36So earlier this year, I don't know if
0:00:38you remember, but sessions kind of used
0:00:40to look like this. So we'd use a model
0:00:41like Opus 45, which honestly for its
0:00:44time was really good. I really enjoyed
0:00:45working with Opus 45. But compared to
0:00:48today, it would just stop a lot earlier.
0:00:50So it would just run for shorter amounts
0:00:52of time.
0:00:53And we'd also keep correcting it. We'd
0:00:55keep reminding it to do things, add
0:00:57things like important, exclamation mark,
0:00:59all these things to our Cloud MD file.
0:01:01And also features like auto mode didn't
0:01:03exist yet. So we were either approving
0:01:05everything manually or taking the
0:01:08riskier approach with dangerously skip
0:01:10permission. So we were still very much
0:01:13in the loop as humans.
0:01:15But as these models got better at taking
0:01:17on bigger tasks, our sessions also
0:01:19changed. So nowadays, at least at
0:01:21Anthropic, our sessions look more like
0:01:23this. So we're running dozens of
0:01:24sessions at the same time, running for
0:01:26hours, verifying their own results along
0:01:29the way, and we even have sessions
0:01:31orchestrate other sessions. So whereas
0:01:33we used to be in the loop for almost
0:01:35every step and running one or just a
0:01:37couple sessions at a time, we've now
0:01:39essentially become managers for dozens
0:01:42of sessions.
0:01:43We also got a lot more done though. Now
0:01:45you may have seen our blog post in
0:01:46October about how Anthropic went from
0:01:48about 2,500 changes a day in May to more
0:01:51than 30,000 a day in September. But this
0:01:55also means that we as developers just
0:01:57have to rethink on what we're focused on
0:01:59instead.
0:02:00So, if you think about our best
0:02:01practices from earlier this year, it was
0:02:03mainly about steering sessions,
0:02:05approving commands, saying no to certain
0:02:08things or restricting what Claude could
0:02:09do,
0:02:11the output, and also starting the
0:02:12sessions. So, everywhere that we now
0:02:15step out of that loop, something else
0:02:17has to take over. And this is either the
0:02:19model, the harness, or the environment
0:02:22that it's running in. So, if we are not
0:02:24in the loop to approve every command and
0:02:26we're running so many sessions at the
0:02:27same time, how can we make sure that the
0:02:30code it's writing is still high quality?
0:02:32And most importantly, if these sessions
0:02:34are running for hours without our
0:02:36supervision, how can we make sure that
0:02:38they're running safely? So, let's go
0:02:40through that by taking this old session
0:02:42and seeing what we have to change to get
0:02:44to that end result that I showed
0:02:45earlier.
0:02:46Well, first of all, let's use the actual
0:02:47defaults we start sessions in nowadays.
0:02:49So, that's not really something you
0:02:51should think about. So, let's update the
0:02:52model here from Opus 4 5 to Opus 5 5 and
0:02:55change the manual permission to auto
0:02:57mode. And Opus 5 5 is a lot better at
0:03:00understanding your intent and
0:03:01understanding your code base. So, we
0:03:03really don't have to steer it as much
0:03:04anymore mid session. And you also no
0:03:06longer have to approve every command
0:03:08because we started making auto mode the
0:03:10default permission mode back in August.
0:03:12Now, being better at understanding your
0:03:14intent and your code base also means
0:03:16that we have to rethink the instructions
0:03:18we have in our code base. So,
0:03:20if you followed our best practices from
0:03:21earlier this year, you may still have
0:03:23things like important or never do this,
0:03:25exclamation mark, all these things in
0:03:27your cloud MD file. But with today's
0:03:29models, you really don't need that and
0:03:31it can actually make the model go off
0:03:32track. So, you end up with more tool
0:03:34calls and overall a worse results.
0:03:37So, nowadays your cloud MD file should
0:03:39only include the things the Claude
0:03:40cannot see from your code. So, for
0:03:42example, maybe you have a service that's
0:03:44still using some legacy source
0:03:46somewhere, but it's not directly
0:03:47referenced in the code. That's still
0:03:49good to keep in there. But everything
0:03:51else, like what stack you're on or where
0:03:53to look, you really don't need that
0:03:54anymore. And we actually have a command
0:03:56for this called doctor prompt audit,
0:03:58which makes it really easy to optimize
0:04:00your environment for a specific model.
0:04:02So, if on Opus 55 here and you run this
0:04:05command, it will automatically check
0:04:06your cloud MD and your skills to see
0:04:08what the new model no longer needs.
0:04:11Now, auto mode is a permission mode that
0:04:13runs a separate classifier on pretty
0:04:15much everything that isn't just reading
0:04:17or editing files in your project.
0:04:20So, it automatically allows commands
0:04:22that it knows are safe, like running
0:04:23tests or installing dependencies, but it
0:04:26still blocks commands where it's unsure
0:04:28whether they're safe.
0:04:29And normally that's pretty good. Uh
0:04:31normally that's kind of, you know, what
0:04:32we want. But for example here, if we ask
0:04:35it to ship something to staging, we
0:04:37would have just, you know, approved the
0:04:39push to our deploy repo. But by default,
0:04:42auto mode only trusts this folder and
0:04:44this repo, so it would block that
0:04:45command. So, we're back in the loop.
0:04:48Now, the defaults should be good in most
0:04:50cases, but if you still notice that you
0:04:51have to approve a lot of uh commands
0:04:54even in auto mode, the first thing to
0:04:56look at is settings here because you can
0:04:58configure auto mode in your own settings
0:05:00just in plain English, and this is what
0:05:01gets sent to the classifier, which then
0:05:03decides whether to reject or allow
0:05:05certain commands.
0:05:07So, here we can just write which repos,
0:05:09buckets, domains we trust um and what it
0:05:11should always allow or never do. Okay,
0:05:13so honestly this was pretty easy by just
0:05:15using today's defaults. We've already
0:05:17gotten rid of most of our in-the-loop
0:05:19stuff
0:05:20um because the model took over the
0:05:21steering and auto mode took over all the
0:05:23approving, so getting out of the loop is
0:05:25easy. But the harder question now is how
0:05:28do we make sure it's okay that we're not
0:05:29there anymore? So, first is keeping it
0:05:32safe and then also verification.
0:05:34So, let's start with keeping it safe
0:05:35because everything so far, so upgrading
0:05:37the model and using auto mode is just
0:05:40working with a model.
0:05:42And in most cases it does what you tell
0:05:44it to do, but that's not a guarantee.
0:05:46So, in a study we published back in
0:05:48August, we tested auto mode versus real
0:05:50people in the loop, and only about 14%
0:05:53of people caught a clearly dangerous
0:05:55command. Auto mode blocked 89% of them,
0:05:58so it is a lot better at it than we
0:06:00were.
0:06:01But about one in 10 of those commands
0:06:03still went through. So, if do want the
0:06:05guarantee, you can only enforce that in
0:06:08the harness. And this is where some of
0:06:10our older best practices still matters,
0:06:11and that's permissions. So, for example,
0:06:13if you have a deny rule for a specific
0:06:16tool called in your project, Cloud Code
0:06:18looks at this rule before the automo
0:06:20classifier even looks at it. In fact, if
0:06:22a tool called matches a deny rule, the
0:06:24classifier never even sees it. So,
0:06:26having deny rules is still the best way
0:06:28to make sure Cloud actually never does
0:06:30something.
0:06:32But then how do you know what needs to
0:06:33be a rule in your project versus what
0:06:35the model can just know because they've
0:06:37gotten so good at it? Um I think it kind
0:06:39of just comes down to if Cloud gets this
0:06:41wrong once, can I take it back? So, if
0:06:44it writes the wrong code, you can still
0:06:46revert that. Um so, you can still leave
0:06:48that to the model. But if it throws away
0:06:50our uncommitted changes or it reads a
0:06:52key that's now in your transcript and
0:06:53you don't don't want that, that should
0:06:55still be a rule. So, for example, let's
0:06:58say that we just never want Cloud
0:06:59reading our .env file. Now, by default,
0:07:02automo is fine with this, so it would
0:07:03have just approved it.
0:07:05So so, if you want to make sure that
0:07:07Cloud can never read this file, you
0:07:09should just add a deny rule. And then
0:07:11commands like cat cannot open this file
0:07:12either. But it's not always that
0:07:14straightforward cuz sometimes certain
0:07:16commands can have side effects. So, for
0:07:19example, let's say that npm test here, a
0:07:21very normal command, it's just a test,
0:07:23so automo runs it, that's fine.
0:07:25But when the script is running, it runs
0:07:27code that you didn't write, like your
0:07:28dependencies. Or maybe Cloud runs a
0:07:31script real quick that it wrote just to
0:07:33quickly check something.
0:07:34So, if one of those scripts is
0:07:36compromised, it could just open that
0:07:38same .env file or your SSH keys and send
0:07:41those to a server you've never heard of,
0:07:43right? And even if we were still there
0:07:46as humans approving everything,
0:07:47honestly, we would have just clicked
0:07:49allow cuz there's nothing to say no to.
0:07:50It's just part of the script that in
0:07:53this example case npm test runs. And
0:07:56permissions only decide if a tool call
0:07:59runs, but they cannot limit what it can
0:08:01reach once it's running. So, we want to
0:08:03make sure that these kinds of scripts
0:08:05can still run because this is just part
0:08:07of, you know, how Cloud can check its
0:08:08own work running tests.
0:08:10So we cannot just block npm tests, but
0:08:12we want to limit its side effects. So
0:08:14really, we just want to limit the
0:08:15environment that it's running in.
0:08:18So if you're running it locally or on
0:08:20your own machine, you can run commands
0:08:22like that in a sandbox. So then you have
0:08:24control over which files in a command it
0:08:26can read, which folders it can write to,
0:08:29which hosts it can reach. And the
0:08:31operating system enforces that.
0:08:34Also, this also applies to everything
0:08:35that command then starts, so any child
0:08:37processes.
0:08:39So on its own, that deny rule gets
0:08:41checked by Cloud before the command
0:08:42runs, but all it sees there is npm
0:08:45tests. So it doesn't see which files
0:08:47that script ends up opening. But with
0:08:49the sandbox on, that same rule gets
0:08:51passed on to the operating system. So
0:08:53now when when that dependency tries to
0:08:56open or end file, the operating system
0:08:57just doesn't let it.
0:08:59And by default, it can still read most
0:09:00of your machine, so you'd want to add
0:09:02another rule for your SSH keys if you
0:09:04care about that.
0:09:06And it also cannot reach a host that
0:09:08wasn't approved.
0:09:10But the sandbox only covers the shell
0:09:12commands that Cloud runs and whatever
0:09:14those start. But Cloud's own files, your
0:09:16MCP servers, and all of your hooks still
0:09:18run outside of it. So the rest of your
0:09:21session still runs on your own machine
0:09:22and everything that's on there. So
0:09:24ideally, you want this whole thing to
0:09:26run somewhere where none of that is even
0:09:28there in the first place. And that is a
0:09:31Cloud session because there it gets its
0:09:33own machine with a fresh copy of a repo.
0:09:36And this runs even when we close our
0:09:37laptop, which is definitely something
0:09:39you want for these longer running
0:09:40sessions that run for hours.
0:09:43So in this case, this automatically gets
0:09:45rid of the end file issue because it's
0:09:47not in Git, and neither are my SSH keys
0:09:49or our GitHub token.
0:09:51And if it still needs an API key, you
0:09:54can add that as a network secret where
0:09:55it will it will tell you like, hey, I'm
0:09:57missing an API key mid-session,
0:10:00so it can use it without actually ever
0:10:01sending it to the model. And by default,
0:10:03also in the Cloud session, network
0:10:06access is limited to things like package
0:10:08registries in GitHub. So you have to
0:10:10explicitly add those domains to your
0:10:12environment for Cloud to to reach other
0:10:14hosts.
0:10:16Okay, so with a Cloud session we can run
0:10:18these long-running sessions even with
0:10:20our laptop closed and it keeps us safe
0:10:23because it runs in its own environment.
0:10:26Now it's still good to remember that
0:10:28this machine is disposable, but the
0:10:29things that leave it aren't. So a commit
0:10:31pushed or a message it sent through a
0:10:33connector. So we still run it in auto
0:10:36mode with our deny rules just commit it
0:10:38into the repo so they come along. Uh so
0:10:40it it could still catch things like a
0:10:42forced push or a secret in a commit, all
0:10:44that kind of stuff.
0:10:47And beyond just safety um we also need
0:10:50to make sure that Cloud is actually
0:10:51working on the right thing. So next is
0:10:54verification. And the nice thing about
0:10:56being in a loop uh like we kind of knew
0:10:58the work was good because we'd already
0:11:00kind of seen every change, we'd approved
0:11:02everything.
0:11:03But now if Cloud works for hours in the
0:11:05background, we don't have that anymore.
0:11:07Um and we're not really going to read
0:11:09every file at the end either.
0:11:12So you know, say that we ask you to move
0:11:14our auth to a new API. Well Cloud is
0:11:17very good at understanding what we want,
0:11:19but there are still a lot of small
0:11:20decisions that it has to make along the
0:11:22way. So with the exact same prompt, you
0:11:24know, one run might move it over but
0:11:26leave the old API in there or one might
0:11:29move it but only for new logins and
0:11:31another one might move it but log
0:11:33everyone out. And these all look pretty
0:11:35reasonable, but it really helps to set
0:11:38up these verification points along the
0:11:39way. So Cloud keeps verifying its own
0:11:42results. And those can be anything that
0:11:44comes back as a pass or fail. So tests,
0:11:47a build, a screenshot, whatever you
0:11:48have.
0:11:49So now we just want to make sure that we
0:11:51review these verification points. Uh
0:11:53there are a lot shorter to review than
0:11:55if we just had to you know, review the
0:11:57whole change at the end. So so that's
0:11:58pretty reasonable.
0:12:00So if one of these runs would still log
0:12:01everyone out, it doesn't get past that
0:12:03first check and then it just has to find
0:12:05another way. And there's a couple ways
0:12:07to set this up. Um the easiest one is of
0:12:09course just in your prompt. So you know,
0:12:11you say what done looks like and ask it
0:12:13to split that into tasks so they have
0:12:15their own tests. Um, and I also like to
0:12:17ask it to show the actual output because
0:12:19you weren't there to see it. But again,
0:12:21prompting is not a guarantee that a
0:12:23model will do it. So for the ones that
0:12:25have to run
0:12:27guaranteed run, I would use hooks, um,
0:12:29which are just, you know, scripts that
0:12:31Cloud Code runs at a specific point.
0:12:33And
0:12:34those give you a lot of control because
0:12:36you decide when they run and what they
0:12:38run and what they check.
0:12:39So for example, every time it edits a
0:12:41file, we can run our linter. Every time
0:12:43it wants to mark a task as complete, we
0:12:45can run the tests for what it changed.
0:12:47Um, so you know, if it touched our auth
0:12:49code or something, it has to pass the
0:12:51auth auth tests.
0:12:52And when it wants to finish the whole
0:12:54session, we can just run the entire test
0:12:55suite.
0:12:56And if one of those fails, it goes, oh,
0:12:59that's still broken. It goes back until
0:13:01it passes, so it fixes it. Now, just one
0:13:04thing,
0:13:05um, when a test fails, by default, most
0:13:07tools that exit with a one exit code,
0:13:10um, but with a one, that task by Cloud
0:13:12Code just gets marked as completed
0:13:14anyway. So make sure that your scripts
0:13:17exit with a two exit code because that's
0:13:19what actually sends Cloud back to fix
0:13:21it.
0:13:22So
0:13:23okay, now we have one session that I'm
0:13:25pretty comfortable with leaving alone.
0:13:28So now we can start with a few more of
0:13:30those because they've all got their own
0:13:32machines, so they're, you know, not all
0:13:33working in the same folder and we don't
0:13:35really have to think about work trees
0:13:36too much ourselves anymore.
0:13:39But then we still start every session
0:13:41ourselves and we review every one of
0:13:43them. And honestly, this is already a
0:13:45pretty good place to be. We are still
0:13:47fully in control over what gets started
0:13:50and when we call a task done.
0:13:53But even there, um, you know, we're
0:13:55still the ones kicking off every session
0:13:57and we're still in the loop for all of
0:13:59them. So I think the most important
0:14:01thing to have in place here now is your
0:14:02CI because
0:14:04all of these sessions eventually open a
0:14:06PR and your CI runs the same checks on
0:14:08every one of them, no matter how that
0:14:10session was set up or, you know, what
0:14:11Cloud decided to do along the way. And
0:14:13And you turn on auto fix, we have like
0:14:15CI auto-fix,
0:14:16Claude can also see when one of those
0:14:18checks fails and then go back to fix it.
0:14:20So you're not the one spending your
0:14:22review on things that, you know, a check
0:14:24could have already caught.
0:14:27And for some of them, you can even turn
0:14:28on merge when ready. So then GitHub
0:14:30merges the PR once all of the checks
0:14:32have passed. So you're not in the loop
0:14:34to review. Um but again, that that's not
0:14:37the default and that really depends on
0:14:38how risky the change is. So for example,
0:14:40on our team, every PR gets reviewed by
0:14:43Claude first. So not by the session that
0:14:45wrote it. So it comes with a fresh
0:14:48context and also a separate group of
0:14:50agents to just double check every bug it
0:14:52finds.
0:14:53And then how much we review ourselves
0:14:55just depends on the risk. So smaller
0:14:57changes like docs or copy, they don't
0:14:59need a person. So once the automated
0:15:01checks are green, we're pretty good to
0:15:03go. Um maybe ordinary product changes
0:15:05just need one reviewer and then risky
0:15:07changes might need two.
0:15:10Now ultimately, kind of where we see
0:15:12things going um and we're already kind
0:15:14of doing more of that at Anthropic is
0:15:16that you're not really starting all of
0:15:18those sessions yourself either. Because
0:15:19if you're anything like me, I'm terrible
0:15:21at context switching. So having to
0:15:23manage so many sessions at the at the
0:15:25same time doesn't always go well.
0:15:27So instead, you just have one main
0:15:29session that's your orchestrator and you
0:15:31can already see this in things like
0:15:33Claude Code projects, which is still in
0:15:35beta, but
0:15:37you just tell it what it needs to be
0:15:39done and the orchestrator splits that up
0:15:41and then starts other sessions for each
0:15:43task and it keeps track of what comes
0:15:45back. So it automatically starts and
0:15:47verifies these threads which are just
0:15:49new sessions. So instead of talking to
0:15:52eight sessions, you're just talking to
0:15:53one.
0:15:55And the main session doesn't write the
0:15:56code itself. So there you want a model
0:15:58that's really good at deciding what to
0:16:00do and judging what comes back, a model
0:16:02like Fable. And if something comes back
0:16:05that isn't actually done, it just, you
0:16:06know, sends it back so until it's
0:16:08actually done and passes everything.
0:16:10And the threads need a model that's good
0:16:12at actually doing the work like Opus 55.
0:16:14And you can set those separately. So,
0:16:17you know, one model for the main session
0:16:18and then another one for the threads.
0:16:20And another benefit is that with this
0:16:22setup with CloakCode projects
0:16:24specifically, we don't manage context
0:16:26anymore. That's all done automatically
0:16:28for us.
0:16:29no longer have to think about, you know,
0:16:31when to compact or clear all of that
0:16:33stuff.
0:16:34Okay, so if you look back at those old
0:16:36best practices, they're all still there,
0:16:38but they've just moved.
0:16:40So, the model steers it now, um auto
0:16:42mode approves your deny rules and the
0:16:44environment restricts what's possible,
0:16:46um your tests, your hooks, and your CI
0:16:48checks it. And then ultimately, Cloud
0:16:50can just start them as well. So, almost
0:16:52everything we used to do ourselves is
0:16:54now either the model, the harness, or
0:16:57the environment. And we're really just
0:16:59here to start that first one and to just
0:17:00review what comes out.
0:17:02So, if you want to try this in your own
0:17:04setup, I would say just take one thing
0:17:06that's way bigger than you normally give
0:17:07Cloud. Make sure your CI actually
0:17:10catches the things you care about, and
0:17:12then spend a bit of time just
0:17:13configuring your environment, like its
0:17:14network access, its secret, and its, you
0:17:16know, setup script.
0:17:18And then start in the cloud and see
0:17:20where your sessions get stuck and where
0:17:21you're still in the loop because that's
0:17:23the next thing to hand over to Cloud. I
0:17:26also highly recommend checking out our
0:17:27best practices in the docs as well.
0:17:30Thank you so much.
