1
00:00:00,500 --> 00:00:02,990
I had some technical difficulties during

2
00:00:03,000 --> 00:00:05,690
my talk this morning, so I ran back home

3
00:00:05,700 --> 00:00:07,290
and I'm just going to record it from my

4
00:00:07,300 --> 00:00:09,290
computer instead. So, hi everyone. I'm

5
00:00:09,300 --> 00:00:11,390
Lydia. I work on the Cloud Code team at

6
00:00:11,400 --> 00:00:14,090
Anthropic. And the models have gotten

7
00:00:14,100 --> 00:00:16,590
really good lately. So we have Opus 55,

8
00:00:16,600 --> 00:00:18,990
which is my favorite daily driver. And

9
00:00:19,000 --> 00:00:20,390
earlier this week, we also launched

10
00:00:20,400 --> 00:00:23,790
Fable 52. And these models are extremely

11
00:00:23,800 --> 00:00:25,690
good at long-running tasks and

12
00:00:25,700 --> 00:00:27,290
understanding your codebase. And

13
00:00:27,300 --> 00:00:29,090
especially Fable is also really good at

14
00:00:29,100 --> 00:00:31,490
verifying its own output. But as these

15
00:00:31,500 --> 00:00:33,790
models get better, the way that we work

16
00:00:33,800 --> 00:00:36,690
with them in Cloud Code also changes.

17
00:00:36,700 --> 00:00:38,190
So earlier this year, I don't know if

18
00:00:38,200 --> 00:00:39,990
you remember, but sessions kind of used

19
00:00:40,000 --> 00:00:41,690
to look like this. So we'd use a model

20
00:00:41,700 --> 00:00:44,190
like Opus 45, which honestly for its

21
00:00:44,200 --> 00:00:45,890
time was really good. I really enjoyed

22
00:00:45,900 --> 00:00:47,990
working with Opus 45. But compared to

23
00:00:48,000 --> 00:00:50,190
today, it would just stop a lot earlier.

24
00:00:50,200 --> 00:00:52,290
So it would just run for shorter amounts

25
00:00:52,300 --> 00:00:53,390
of time.

26
00:00:53,400 --> 00:00:55,190
And we'd also keep correcting it. We'd

27
00:00:55,200 --> 00:00:56,990
keep reminding it to do things, add

28
00:00:57,000 --> 00:00:59,290
things like important, exclamation mark,

29
00:00:59,300 --> 00:01:01,290
all these things to our Cloud MD file.

30
00:01:01,300 --> 00:01:03,290
And also features like auto mode didn't

31
00:01:03,300 --> 00:01:05,790
exist yet. So we were either approving

32
00:01:05,800 --> 00:01:07,990
everything manually or taking the

33
00:01:08,000 --> 00:01:10,590
riskier approach with dangerously skip

34
00:01:10,600 --> 00:01:12,990
permission. So we were still very much

35
00:01:13,000 --> 00:01:15,190
in the loop as humans.

36
00:01:15,200 --> 00:01:17,490
But as these models got better at taking

37
00:01:17,500 --> 00:01:19,390
on bigger tasks, our sessions also

38
00:01:19,400 --> 00:01:21,390
changed. So nowadays, at least at

39
00:01:21,400 --> 00:01:23,090
Anthropic, our sessions look more like

40
00:01:23,100 --> 00:01:24,790
this. So we're running dozens of

41
00:01:24,800 --> 00:01:26,690
sessions at the same time, running for

42
00:01:26,700 --> 00:01:28,990
hours, verifying their own results along

43
00:01:29,000 --> 00:01:31,190
the way, and we even have sessions

44
00:01:31,200 --> 00:01:33,790
orchestrate other sessions. So whereas

45
00:01:33,800 --> 00:01:35,590
we used to be in the loop for almost

46
00:01:35,600 --> 00:01:37,590
every step and running one or just a

47
00:01:37,600 --> 00:01:39,490
couple sessions at a time, we've now

48
00:01:39,500 --> 00:01:42,090
essentially become managers for dozens

49
00:01:42,100 --> 00:01:43,790
of sessions.

50
00:01:43,800 --> 00:01:45,390
We also got a lot more done though. Now

51
00:01:45,400 --> 00:01:46,890
you may have seen our blog post in

52
00:01:46,900 --> 00:01:48,690
October about how Anthropic went from

53
00:01:48,700 --> 00:01:51,790
about 2,500 changes a day in May to more

54
00:01:51,800 --> 00:01:54,990
than 30,000 a day in September. But this

55
00:01:55,000 --> 00:01:57,390
also means that we as developers just

56
00:01:57,400 --> 00:01:59,190
have to rethink on what we're focused on

57
00:01:59,200 --> 00:02:00,282
instead.

58
00:02:00,292 --> 00:02:01,782
So, if you think about our best

59
00:02:01,792 --> 00:02:03,882
practices from earlier this year, it was

60
00:02:03,892 --> 00:02:05,782
mainly about steering sessions,

61
00:02:05,792 --> 00:02:08,182
approving commands, saying no to certain

62
00:02:08,192 --> 00:02:09,782
things or restricting what Claude could

63
00:02:09,792 --> 00:02:11,282
do,

64
00:02:11,292 --> 00:02:12,782
the output, and also starting the

65
00:02:12,792 --> 00:02:15,182
sessions. So, everywhere that we now

66
00:02:15,192 --> 00:02:17,182
step out of that loop, something else

67
00:02:17,192 --> 00:02:19,282
has to take over. And this is either the

68
00:02:19,292 --> 00:02:22,082
model, the harness, or the environment

69
00:02:22,092 --> 00:02:24,482
that it's running in. So, if we are not

70
00:02:24,492 --> 00:02:26,382
in the loop to approve every command and

71
00:02:26,392 --> 00:02:27,982
we're running so many sessions at the

72
00:02:27,992 --> 00:02:30,182
same time, how can we make sure that the

73
00:02:30,192 --> 00:02:32,882
code it's writing is still high quality?

74
00:02:32,892 --> 00:02:34,582
And most importantly, if these sessions

75
00:02:34,592 --> 00:02:36,182
are running for hours without our

76
00:02:36,192 --> 00:02:38,082
supervision, how can we make sure that

77
00:02:38,092 --> 00:02:40,382
they're running safely? So, let's go

78
00:02:40,392 --> 00:02:42,482
through that by taking this old session

79
00:02:42,492 --> 00:02:44,082
and seeing what we have to change to get

80
00:02:44,092 --> 00:02:45,182
to that end result that I showed

81
00:02:45,192 --> 00:02:46,182
earlier.

82
00:02:46,192 --> 00:02:47,682
Well, first of all, let's use the actual

83
00:02:47,692 --> 00:02:49,682
defaults we start sessions in nowadays.

84
00:02:49,692 --> 00:02:51,382
So, that's not really something you

85
00:02:51,392 --> 00:02:52,882
should think about. So, let's update the

86
00:02:52,892 --> 00:02:55,782
model here from Opus 4 5 to Opus 5 5 and

87
00:02:55,792 --> 00:02:57,682
change the manual permission to auto

88
00:02:57,692 --> 00:03:00,582
mode. And Opus 5 5 is a lot better at

89
00:03:00,592 --> 00:03:01,882
understanding your intent and

90
00:03:01,892 --> 00:03:03,282
understanding your code base. So, we

91
00:03:03,292 --> 00:03:04,882
really don't have to steer it as much

92
00:03:04,892 --> 00:03:06,882
anymore mid session. And you also no

93
00:03:06,892 --> 00:03:08,582
longer have to approve every command

94
00:03:08,592 --> 00:03:10,282
because we started making auto mode the

95
00:03:10,292 --> 00:03:12,582
default permission mode back in August.

96
00:03:12,592 --> 00:03:14,682
Now, being better at understanding your

97
00:03:14,692 --> 00:03:16,582
intent and your code base also means

98
00:03:16,592 --> 00:03:18,482
that we have to rethink the instructions

99
00:03:18,492 --> 00:03:20,482
we have in our code base. So,

100
00:03:20,492 --> 00:03:21,982
if you followed our best practices from

101
00:03:21,992 --> 00:03:23,382
earlier this year, you may still have

102
00:03:23,392 --> 00:03:25,782
things like important or never do this,

103
00:03:25,792 --> 00:03:27,182
exclamation mark, all these things in

104
00:03:27,192 --> 00:03:29,082
your cloud MD file. But with today's

105
00:03:29,092 --> 00:03:31,082
models, you really don't need that and

106
00:03:31,092 --> 00:03:32,682
it can actually make the model go off

107
00:03:32,692 --> 00:03:34,682
track. So, you end up with more tool

108
00:03:34,692 --> 00:03:37,282
calls and overall a worse results.

109
00:03:37,292 --> 00:03:39,082
So, nowadays your cloud MD file should

110
00:03:39,092 --> 00:03:40,582
only include the things the Claude

111
00:03:40,592 --> 00:03:42,582
cannot see from your code. So, for

112
00:03:42,592 --> 00:03:44,682
example, maybe you have a service that's

113
00:03:44,692 --> 00:03:46,082
still using some legacy source

114
00:03:46,092 --> 00:03:47,782
somewhere, but it's not directly

115
00:03:47,792 --> 00:03:49,382
referenced in the code. That's still

116
00:03:49,392 --> 00:03:51,582
good to keep in there. But everything

117
00:03:51,592 --> 00:03:53,282
else, like what stack you're on or where

118
00:03:53,292 --> 00:03:54,782
to look, you really don't need that

119
00:03:54,792 --> 00:03:56,782
anymore. And we actually have a command

120
00:03:56,792 --> 00:03:58,682
for this called doctor prompt audit,

121
00:03:58,692 --> 00:04:00,682
which makes it really easy to optimize

122
00:04:00,692 --> 00:04:02,782
your environment for a specific model.

123
00:04:02,792 --> 00:04:05,184
So, if on Opus 55 here and you run this

124
00:04:05,194 --> 00:04:06,884
command, it will automatically check

125
00:04:06,894 --> 00:04:08,884
your cloud MD and your skills to see

126
00:04:08,894 --> 00:04:11,484
what the new model no longer needs.

127
00:04:11,494 --> 00:04:13,384
Now, auto mode is a permission mode that

128
00:04:13,394 --> 00:04:15,384
runs a separate classifier on pretty

129
00:04:15,394 --> 00:04:17,584
much everything that isn't just reading

130
00:04:17,594 --> 00:04:20,084
or editing files in your project.

131
00:04:20,094 --> 00:04:22,084
So, it automatically allows commands

132
00:04:22,094 --> 00:04:23,784
that it knows are safe, like running

133
00:04:23,794 --> 00:04:26,284
tests or installing dependencies, but it

134
00:04:26,294 --> 00:04:28,484
still blocks commands where it's unsure

135
00:04:28,494 --> 00:04:29,684
whether they're safe.

136
00:04:29,694 --> 00:04:31,384
And normally that's pretty good. Uh

137
00:04:31,394 --> 00:04:32,784
normally that's kind of, you know, what

138
00:04:32,794 --> 00:04:35,984
we want. But for example here, if we ask

139
00:04:35,994 --> 00:04:37,884
it to ship something to staging, we

140
00:04:37,894 --> 00:04:39,184
would have just, you know, approved the

141
00:04:39,194 --> 00:04:42,084
push to our deploy repo. But by default,

142
00:04:42,094 --> 00:04:44,284
auto mode only trusts this folder and

143
00:04:44,294 --> 00:04:45,984
this repo, so it would block that

144
00:04:45,994 --> 00:04:48,084
command. So, we're back in the loop.

145
00:04:48,094 --> 00:04:50,384
Now, the defaults should be good in most

146
00:04:50,394 --> 00:04:51,984
cases, but if you still notice that you

147
00:04:51,994 --> 00:04:54,084
have to approve a lot of uh commands

148
00:04:54,094 --> 00:04:56,184
even in auto mode, the first thing to

149
00:04:56,194 --> 00:04:58,084
look at is settings here because you can

150
00:04:58,094 --> 00:05:00,384
configure auto mode in your own settings

151
00:05:00,394 --> 00:05:01,984
just in plain English, and this is what

152
00:05:01,994 --> 00:05:03,884
gets sent to the classifier, which then

153
00:05:03,894 --> 00:05:05,784
decides whether to reject or allow

154
00:05:05,794 --> 00:05:07,184
certain commands.

155
00:05:07,194 --> 00:05:09,084
So, here we can just write which repos,

156
00:05:09,094 --> 00:05:11,384
buckets, domains we trust um and what it

157
00:05:11,394 --> 00:05:13,384
should always allow or never do. Okay,

158
00:05:13,394 --> 00:05:15,684
so honestly this was pretty easy by just

159
00:05:15,694 --> 00:05:17,484
using today's defaults. We've already

160
00:05:17,494 --> 00:05:19,284
gotten rid of most of our in-the-loop

161
00:05:19,294 --> 00:05:20,184
stuff

162
00:05:20,194 --> 00:05:21,584
um because the model took over the

163
00:05:21,594 --> 00:05:23,484
steering and auto mode took over all the

164
00:05:23,494 --> 00:05:25,784
approving, so getting out of the loop is

165
00:05:25,794 --> 00:05:28,084
easy. But the harder question now is how

166
00:05:28,094 --> 00:05:29,884
do we make sure it's okay that we're not

167
00:05:29,894 --> 00:05:32,084
there anymore? So, first is keeping it

168
00:05:32,094 --> 00:05:34,684
safe and then also verification.

169
00:05:34,694 --> 00:05:35,884
So, let's start with keeping it safe

170
00:05:35,894 --> 00:05:37,684
because everything so far, so upgrading

171
00:05:37,694 --> 00:05:40,084
the model and using auto mode is just

172
00:05:40,094 --> 00:05:42,084
working with a model.

173
00:05:42,094 --> 00:05:44,284
And in most cases it does what you tell

174
00:05:44,294 --> 00:05:46,884
it to do, but that's not a guarantee.

175
00:05:46,894 --> 00:05:48,484
So, in a study we published back in

176
00:05:48,494 --> 00:05:50,584
August, we tested auto mode versus real

177
00:05:50,594 --> 00:05:53,084
people in the loop, and only about 14%

178
00:05:53,094 --> 00:05:55,284
of people caught a clearly dangerous

179
00:05:55,294 --> 00:05:58,684
command. Auto mode blocked 89% of them,

180
00:05:58,694 --> 00:06:00,584
so it is a lot better at it than we

181
00:06:00,594 --> 00:06:01,484
were.

182
00:06:01,494 --> 00:06:03,684
But about one in 10 of those commands

183
00:06:03,694 --> 00:06:05,885
still went through. So, if do want the

184
00:06:05,895 --> 00:06:08,285
guarantee, you can only enforce that in

185
00:06:08,295 --> 00:06:10,185
the harness. And this is where some of

186
00:06:10,195 --> 00:06:11,885
our older best practices still matters,

187
00:06:11,895 --> 00:06:13,885
and that's permissions. So, for example,

188
00:06:13,895 --> 00:06:16,185
if you have a deny rule for a specific

189
00:06:16,195 --> 00:06:18,385
tool called in your project, Cloud Code

190
00:06:18,395 --> 00:06:20,085
looks at this rule before the automo

191
00:06:20,095 --> 00:06:22,485
classifier even looks at it. In fact, if

192
00:06:22,495 --> 00:06:24,585
a tool called matches a deny rule, the

193
00:06:24,595 --> 00:06:26,585
classifier never even sees it. So,

194
00:06:26,595 --> 00:06:28,885
having deny rules is still the best way

195
00:06:28,895 --> 00:06:30,985
to make sure Cloud actually never does

196
00:06:30,995 --> 00:06:32,285
something.

197
00:06:32,295 --> 00:06:33,985
But then how do you know what needs to

198
00:06:33,995 --> 00:06:35,885
be a rule in your project versus what

199
00:06:35,895 --> 00:06:37,485
the model can just know because they've

200
00:06:37,495 --> 00:06:39,485
gotten so good at it? Um I think it kind

201
00:06:39,495 --> 00:06:41,585
of just comes down to if Cloud gets this

202
00:06:41,595 --> 00:06:44,585
wrong once, can I take it back? So, if

203
00:06:44,595 --> 00:06:46,185
it writes the wrong code, you can still

204
00:06:46,195 --> 00:06:48,385
revert that. Um so, you can still leave

205
00:06:48,395 --> 00:06:50,285
that to the model. But if it throws away

206
00:06:50,295 --> 00:06:52,185
our uncommitted changes or it reads a

207
00:06:52,195 --> 00:06:53,885
key that's now in your transcript and

208
00:06:53,895 --> 00:06:55,785
you don't don't want that, that should

209
00:06:55,795 --> 00:06:58,185
still be a rule. So, for example, let's

210
00:06:58,195 --> 00:06:59,985
say that we just never want Cloud

211
00:06:59,995 --> 00:07:02,485
reading our .env file. Now, by default,

212
00:07:02,495 --> 00:07:03,885
automo is fine with this, so it would

213
00:07:03,895 --> 00:07:05,885
have just approved it.

214
00:07:05,895 --> 00:07:07,285
So so, if you want to make sure that

215
00:07:07,295 --> 00:07:09,185
Cloud can never read this file, you

216
00:07:09,195 --> 00:07:11,185
should just add a deny rule. And then

217
00:07:11,195 --> 00:07:12,885
commands like cat cannot open this file

218
00:07:12,895 --> 00:07:14,885
either. But it's not always that

219
00:07:14,895 --> 00:07:16,685
straightforward cuz sometimes certain

220
00:07:16,695 --> 00:07:19,085
commands can have side effects. So, for

221
00:07:19,095 --> 00:07:21,385
example, let's say that npm test here, a

222
00:07:21,395 --> 00:07:23,385
very normal command, it's just a test,

223
00:07:23,395 --> 00:07:25,485
so automo runs it, that's fine.

224
00:07:25,495 --> 00:07:27,585
But when the script is running, it runs

225
00:07:27,595 --> 00:07:28,985
code that you didn't write, like your

226
00:07:28,995 --> 00:07:31,285
dependencies. Or maybe Cloud runs a

227
00:07:31,295 --> 00:07:33,185
script real quick that it wrote just to

228
00:07:33,195 --> 00:07:34,985
quickly check something.

229
00:07:34,995 --> 00:07:36,585
So, if one of those scripts is

230
00:07:36,595 --> 00:07:38,585
compromised, it could just open that

231
00:07:38,595 --> 00:07:41,585
same .env file or your SSH keys and send

232
00:07:41,595 --> 00:07:43,485
those to a server you've never heard of,

233
00:07:43,495 --> 00:07:46,485
right? And even if we were still there

234
00:07:46,495 --> 00:07:47,985
as humans approving everything,

235
00:07:47,995 --> 00:07:49,185
honestly, we would have just clicked

236
00:07:49,195 --> 00:07:50,785
allow cuz there's nothing to say no to.

237
00:07:50,795 --> 00:07:53,085
It's just part of the script that in

238
00:07:53,095 --> 00:07:56,685
this example case npm test runs. And

239
00:07:56,695 --> 00:07:59,185
permissions only decide if a tool call

240
00:07:59,195 --> 00:08:01,185
runs, but they cannot limit what it can

241
00:08:01,195 --> 00:08:03,485
reach once it's running. So, we want to

242
00:08:03,495 --> 00:08:05,085
make sure that these kinds of scripts

243
00:08:05,095 --> 00:08:07,285
can still run because this is just part

244
00:08:07,295 --> 00:08:08,787
of, you know, how Cloud can check its

245
00:08:08,797 --> 00:08:10,887
own work running tests.

246
00:08:10,897 --> 00:08:12,687
So we cannot just block npm tests, but

247
00:08:12,697 --> 00:08:14,887
we want to limit its side effects. So

248
00:08:14,897 --> 00:08:15,987
really, we just want to limit the

249
00:08:15,997 --> 00:08:18,087
environment that it's running in.

250
00:08:18,097 --> 00:08:20,187
So if you're running it locally or on

251
00:08:20,197 --> 00:08:22,187
your own machine, you can run commands

252
00:08:22,197 --> 00:08:24,787
like that in a sandbox. So then you have

253
00:08:24,797 --> 00:08:26,687
control over which files in a command it

254
00:08:26,697 --> 00:08:29,187
can read, which folders it can write to,

255
00:08:29,197 --> 00:08:31,287
which hosts it can reach. And the

256
00:08:31,297 --> 00:08:34,087
operating system enforces that.

257
00:08:34,097 --> 00:08:35,687
Also, this also applies to everything

258
00:08:35,697 --> 00:08:37,887
that command then starts, so any child

259
00:08:37,897 --> 00:08:39,387
processes.

260
00:08:39,397 --> 00:08:41,287
So on its own, that deny rule gets

261
00:08:41,297 --> 00:08:42,787
checked by Cloud before the command

262
00:08:42,797 --> 00:08:45,587
runs, but all it sees there is npm

263
00:08:45,597 --> 00:08:47,087
tests. So it doesn't see which files

264
00:08:47,097 --> 00:08:49,087
that script ends up opening. But with

265
00:08:49,097 --> 00:08:51,187
the sandbox on, that same rule gets

266
00:08:51,197 --> 00:08:53,787
passed on to the operating system. So

267
00:08:53,797 --> 00:08:56,087
now when when that dependency tries to

268
00:08:56,097 --> 00:08:57,887
open or end file, the operating system

269
00:08:57,897 --> 00:08:59,287
just doesn't let it.

270
00:08:59,297 --> 00:09:00,987
And by default, it can still read most

271
00:09:00,997 --> 00:09:02,487
of your machine, so you'd want to add

272
00:09:02,497 --> 00:09:04,487
another rule for your SSH keys if you

273
00:09:04,497 --> 00:09:06,487
care about that.

274
00:09:06,497 --> 00:09:08,687
And it also cannot reach a host that

275
00:09:08,697 --> 00:09:10,487
wasn't approved.

276
00:09:10,497 --> 00:09:12,487
But the sandbox only covers the shell

277
00:09:12,497 --> 00:09:14,087
commands that Cloud runs and whatever

278
00:09:14,097 --> 00:09:16,487
those start. But Cloud's own files, your

279
00:09:16,497 --> 00:09:18,887
MCP servers, and all of your hooks still

280
00:09:18,897 --> 00:09:21,187
run outside of it. So the rest of your

281
00:09:21,197 --> 00:09:22,787
session still runs on your own machine

282
00:09:22,797 --> 00:09:24,487
and everything that's on there. So

283
00:09:24,497 --> 00:09:26,587
ideally, you want this whole thing to

284
00:09:26,597 --> 00:09:28,787
run somewhere where none of that is even

285
00:09:28,797 --> 00:09:31,087
there in the first place. And that is a

286
00:09:31,097 --> 00:09:33,487
Cloud session because there it gets its

287
00:09:33,497 --> 00:09:36,087
own machine with a fresh copy of a repo.

288
00:09:36,097 --> 00:09:37,687
And this runs even when we close our

289
00:09:37,697 --> 00:09:39,387
laptop, which is definitely something

290
00:09:39,397 --> 00:09:40,587
you want for these longer running

291
00:09:40,597 --> 00:09:43,187
sessions that run for hours.

292
00:09:43,197 --> 00:09:45,587
So in this case, this automatically gets

293
00:09:45,597 --> 00:09:47,087
rid of the end file issue because it's

294
00:09:47,097 --> 00:09:49,787
not in Git, and neither are my SSH keys

295
00:09:49,797 --> 00:09:51,787
or our GitHub token.

296
00:09:51,797 --> 00:09:54,087
And if it still needs an API key, you

297
00:09:54,097 --> 00:09:55,887
can add that as a network secret where

298
00:09:55,897 --> 00:09:57,087
it will it will tell you like, hey, I'm

299
00:09:57,097 --> 00:10:00,087
missing an API key mid-session,

300
00:10:00,097 --> 00:10:01,887
so it can use it without actually ever

301
00:10:01,897 --> 00:10:03,987
sending it to the model. And by default,

302
00:10:03,997 --> 00:10:06,287
also in the Cloud session, network

303
00:10:06,297 --> 00:10:08,387
access is limited to things like package

304
00:10:08,397 --> 00:10:10,488
registries in GitHub. So you have to

305
00:10:10,498 --> 00:10:12,588
explicitly add those domains to your

306
00:10:12,598 --> 00:10:14,788
environment for Cloud to to reach other

307
00:10:14,798 --> 00:10:16,588
hosts.

308
00:10:16,598 --> 00:10:18,688
Okay, so with a Cloud session we can run

309
00:10:18,698 --> 00:10:20,488
these long-running sessions even with

310
00:10:20,498 --> 00:10:23,088
our laptop closed and it keeps us safe

311
00:10:23,098 --> 00:10:26,088
because it runs in its own environment.

312
00:10:26,098 --> 00:10:28,188
Now it's still good to remember that

313
00:10:28,198 --> 00:10:29,688
this machine is disposable, but the

314
00:10:29,698 --> 00:10:31,988
things that leave it aren't. So a commit

315
00:10:31,998 --> 00:10:33,988
pushed or a message it sent through a

316
00:10:33,998 --> 00:10:36,888
connector. So we still run it in auto

317
00:10:36,898 --> 00:10:38,788
mode with our deny rules just commit it

318
00:10:38,798 --> 00:10:40,988
into the repo so they come along. Uh so

319
00:10:40,998 --> 00:10:42,688
it it could still catch things like a

320
00:10:42,698 --> 00:10:44,788
forced push or a secret in a commit, all

321
00:10:44,798 --> 00:10:47,088
that kind of stuff.

322
00:10:47,098 --> 00:10:50,488
And beyond just safety um we also need

323
00:10:50,498 --> 00:10:51,888
to make sure that Cloud is actually

324
00:10:51,898 --> 00:10:54,188
working on the right thing. So next is

325
00:10:54,198 --> 00:10:56,188
verification. And the nice thing about

326
00:10:56,198 --> 00:10:58,888
being in a loop uh like we kind of knew

327
00:10:58,898 --> 00:11:00,588
the work was good because we'd already

328
00:11:00,598 --> 00:11:02,388
kind of seen every change, we'd approved

329
00:11:02,398 --> 00:11:03,688
everything.

330
00:11:03,698 --> 00:11:05,588
But now if Cloud works for hours in the

331
00:11:05,598 --> 00:11:07,688
background, we don't have that anymore.

332
00:11:07,698 --> 00:11:09,288
Um and we're not really going to read

333
00:11:09,298 --> 00:11:12,188
every file at the end either.

334
00:11:12,198 --> 00:11:14,788
So you know, say that we ask you to move

335
00:11:14,798 --> 00:11:17,088
our auth to a new API. Well Cloud is

336
00:11:17,098 --> 00:11:19,188
very good at understanding what we want,

337
00:11:19,198 --> 00:11:20,788
but there are still a lot of small

338
00:11:20,798 --> 00:11:22,488
decisions that it has to make along the

339
00:11:22,498 --> 00:11:24,988
way. So with the exact same prompt, you

340
00:11:24,998 --> 00:11:26,588
know, one run might move it over but

341
00:11:26,598 --> 00:11:29,088
leave the old API in there or one might

342
00:11:29,098 --> 00:11:31,488
move it but only for new logins and

343
00:11:31,498 --> 00:11:33,088
another one might move it but log

344
00:11:33,098 --> 00:11:35,588
everyone out. And these all look pretty

345
00:11:35,598 --> 00:11:38,088
reasonable, but it really helps to set

346
00:11:38,098 --> 00:11:39,988
up these verification points along the

347
00:11:39,998 --> 00:11:42,188
way. So Cloud keeps verifying its own

348
00:11:42,198 --> 00:11:44,588
results. And those can be anything that

349
00:11:44,598 --> 00:11:47,188
comes back as a pass or fail. So tests,

350
00:11:47,198 --> 00:11:48,888
a build, a screenshot, whatever you

351
00:11:48,898 --> 00:11:49,888
have.

352
00:11:49,898 --> 00:11:51,688
So now we just want to make sure that we

353
00:11:51,698 --> 00:11:53,988
review these verification points. Uh

354
00:11:53,998 --> 00:11:55,488
there are a lot shorter to review than

355
00:11:55,498 --> 00:11:57,188
if we just had to you know, review the

356
00:11:57,198 --> 00:11:58,888
whole change at the end. So so that's

357
00:11:58,898 --> 00:12:00,288
pretty reasonable.

358
00:12:00,298 --> 00:12:01,888
So if one of these runs would still log

359
00:12:01,898 --> 00:12:03,788
everyone out, it doesn't get past that

360
00:12:03,798 --> 00:12:05,688
first check and then it just has to find

361
00:12:05,698 --> 00:12:07,488
another way. And there's a couple ways

362
00:12:07,498 --> 00:12:09,688
to set this up. Um the easiest one is of

363
00:12:09,698 --> 00:12:11,388
course just in your prompt. So you know,

364
00:12:11,398 --> 00:12:13,289
you say what done looks like and ask it

365
00:12:13,299 --> 00:12:15,089
to split that into tasks so they have

366
00:12:15,099 --> 00:12:17,289
their own tests. Um, and I also like to

367
00:12:17,299 --> 00:12:19,489
ask it to show the actual output because

368
00:12:19,499 --> 00:12:21,589
you weren't there to see it. But again,

369
00:12:21,599 --> 00:12:23,389
prompting is not a guarantee that a

370
00:12:23,399 --> 00:12:25,689
model will do it. So for the ones that

371
00:12:25,699 --> 00:12:27,289
have to run

372
00:12:27,299 --> 00:12:29,489
guaranteed run, I would use hooks, um,

373
00:12:29,499 --> 00:12:31,089
which are just, you know, scripts that

374
00:12:31,099 --> 00:12:33,689
Cloud Code runs at a specific point.

375
00:12:33,699 --> 00:12:34,789
And

376
00:12:34,799 --> 00:12:36,289
those give you a lot of control because

377
00:12:36,299 --> 00:12:38,389
you decide when they run and what they

378
00:12:38,399 --> 00:12:39,889
run and what they check.

379
00:12:39,899 --> 00:12:41,689
So for example, every time it edits a

380
00:12:41,699 --> 00:12:43,689
file, we can run our linter. Every time

381
00:12:43,699 --> 00:12:45,789
it wants to mark a task as complete, we

382
00:12:45,799 --> 00:12:47,989
can run the tests for what it changed.

383
00:12:47,999 --> 00:12:49,589
Um, so you know, if it touched our auth

384
00:12:49,599 --> 00:12:51,089
code or something, it has to pass the

385
00:12:51,099 --> 00:12:52,989
auth auth tests.

386
00:12:52,999 --> 00:12:54,189
And when it wants to finish the whole

387
00:12:54,199 --> 00:12:55,789
session, we can just run the entire test

388
00:12:55,799 --> 00:12:56,989
suite.

389
00:12:56,999 --> 00:12:59,089
And if one of those fails, it goes, oh,

390
00:12:59,099 --> 00:13:01,489
that's still broken. It goes back until

391
00:13:01,499 --> 00:13:04,089
it passes, so it fixes it. Now, just one

392
00:13:04,099 --> 00:13:05,189
thing,

393
00:13:05,199 --> 00:13:07,489
um, when a test fails, by default, most

394
00:13:07,499 --> 00:13:10,289
tools that exit with a one exit code,

395
00:13:10,299 --> 00:13:12,889
um, but with a one, that task by Cloud

396
00:13:12,899 --> 00:13:14,689
Code just gets marked as completed

397
00:13:14,699 --> 00:13:17,289
anyway. So make sure that your scripts

398
00:13:17,299 --> 00:13:19,889
exit with a two exit code because that's

399
00:13:19,899 --> 00:13:21,789
what actually sends Cloud back to fix

400
00:13:21,799 --> 00:13:22,889
it.

401
00:13:22,899 --> 00:13:23,989
So

402
00:13:23,999 --> 00:13:25,589
okay, now we have one session that I'm

403
00:13:25,599 --> 00:13:28,189
pretty comfortable with leaving alone.

404
00:13:28,199 --> 00:13:30,289
So now we can start with a few more of

405
00:13:30,299 --> 00:13:32,089
those because they've all got their own

406
00:13:32,099 --> 00:13:33,889
machines, so they're, you know, not all

407
00:13:33,899 --> 00:13:35,589
working in the same folder and we don't

408
00:13:35,599 --> 00:13:36,989
really have to think about work trees

409
00:13:36,999 --> 00:13:39,389
too much ourselves anymore.

410
00:13:39,399 --> 00:13:41,789
But then we still start every session

411
00:13:41,799 --> 00:13:43,689
ourselves and we review every one of

412
00:13:43,699 --> 00:13:45,689
them. And honestly, this is already a

413
00:13:45,699 --> 00:13:47,489
pretty good place to be. We are still

414
00:13:47,499 --> 00:13:50,589
fully in control over what gets started

415
00:13:50,599 --> 00:13:53,589
and when we call a task done.

416
00:13:53,599 --> 00:13:55,389
But even there, um, you know, we're

417
00:13:55,399 --> 00:13:57,489
still the ones kicking off every session

418
00:13:57,499 --> 00:13:59,089
and we're still in the loop for all of

419
00:13:59,099 --> 00:14:01,189
them. So I think the most important

420
00:14:01,199 --> 00:14:02,689
thing to have in place here now is your

421
00:14:02,699 --> 00:14:04,289
CI because

422
00:14:04,299 --> 00:14:06,189
all of these sessions eventually open a

423
00:14:06,199 --> 00:14:08,789
PR and your CI runs the same checks on

424
00:14:08,799 --> 00:14:10,289
every one of them, no matter how that

425
00:14:10,299 --> 00:14:11,789
session was set up or, you know, what

426
00:14:11,799 --> 00:14:13,891
Cloud decided to do along the way. And

427
00:14:13,901 --> 00:14:15,391
And you turn on auto fix, we have like

428
00:14:15,401 --> 00:14:16,791
CI auto-fix,

429
00:14:16,801 --> 00:14:18,591
Claude can also see when one of those

430
00:14:18,601 --> 00:14:20,791
checks fails and then go back to fix it.

431
00:14:20,801 --> 00:14:22,291
So you're not the one spending your

432
00:14:22,301 --> 00:14:24,791
review on things that, you know, a check

433
00:14:24,801 --> 00:14:26,991
could have already caught.

434
00:14:27,001 --> 00:14:28,291
And for some of them, you can even turn

435
00:14:28,301 --> 00:14:30,391
on merge when ready. So then GitHub

436
00:14:30,401 --> 00:14:32,891
merges the PR once all of the checks

437
00:14:32,901 --> 00:14:34,391
have passed. So you're not in the loop

438
00:14:34,401 --> 00:14:37,091
to review. Um but again, that that's not

439
00:14:37,101 --> 00:14:38,691
the default and that really depends on

440
00:14:38,701 --> 00:14:40,891
how risky the change is. So for example,

441
00:14:40,901 --> 00:14:43,291
on our team, every PR gets reviewed by

442
00:14:43,301 --> 00:14:45,791
Claude first. So not by the session that

443
00:14:45,801 --> 00:14:47,991
wrote it. So it comes with a fresh

444
00:14:48,001 --> 00:14:50,491
context and also a separate group of

445
00:14:50,501 --> 00:14:52,291
agents to just double check every bug it

446
00:14:52,301 --> 00:14:53,691
finds.

447
00:14:53,701 --> 00:14:55,691
And then how much we review ourselves

448
00:14:55,701 --> 00:14:57,591
just depends on the risk. So smaller

449
00:14:57,601 --> 00:14:59,291
changes like docs or copy, they don't

450
00:14:59,301 --> 00:15:01,191
need a person. So once the automated

451
00:15:01,201 --> 00:15:02,991
checks are green, we're pretty good to

452
00:15:03,001 --> 00:15:05,691
go. Um maybe ordinary product changes

453
00:15:05,701 --> 00:15:07,891
just need one reviewer and then risky

454
00:15:07,901 --> 00:15:09,991
changes might need two.

455
00:15:10,001 --> 00:15:12,291
Now ultimately, kind of where we see

456
00:15:12,301 --> 00:15:14,591
things going um and we're already kind

457
00:15:14,601 --> 00:15:16,291
of doing more of that at Anthropic is

458
00:15:16,301 --> 00:15:18,091
that you're not really starting all of

459
00:15:18,101 --> 00:15:19,891
those sessions yourself either. Because

460
00:15:19,901 --> 00:15:21,491
if you're anything like me, I'm terrible

461
00:15:21,501 --> 00:15:23,191
at context switching. So having to

462
00:15:23,201 --> 00:15:25,091
manage so many sessions at the at the

463
00:15:25,101 --> 00:15:27,291
same time doesn't always go well.

464
00:15:27,301 --> 00:15:29,491
So instead, you just have one main

465
00:15:29,501 --> 00:15:31,891
session that's your orchestrator and you

466
00:15:31,901 --> 00:15:33,791
can already see this in things like

467
00:15:33,801 --> 00:15:35,491
Claude Code projects, which is still in

468
00:15:35,501 --> 00:15:37,291
beta, but

469
00:15:37,301 --> 00:15:39,091
you just tell it what it needs to be

470
00:15:39,101 --> 00:15:41,891
done and the orchestrator splits that up

471
00:15:41,901 --> 00:15:43,791
and then starts other sessions for each

472
00:15:43,801 --> 00:15:45,491
task and it keeps track of what comes

473
00:15:45,501 --> 00:15:47,191
back. So it automatically starts and

474
00:15:47,201 --> 00:15:49,491
verifies these threads which are just

475
00:15:49,501 --> 00:15:51,991
new sessions. So instead of talking to

476
00:15:52,001 --> 00:15:53,591
eight sessions, you're just talking to

477
00:15:53,601 --> 00:15:55,091
one.

478
00:15:55,101 --> 00:15:56,491
And the main session doesn't write the

479
00:15:56,501 --> 00:15:58,591
code itself. So there you want a model

480
00:15:58,601 --> 00:16:00,491
that's really good at deciding what to

481
00:16:00,501 --> 00:16:02,491
do and judging what comes back, a model

482
00:16:02,501 --> 00:16:04,991
like Fable. And if something comes back

483
00:16:05,001 --> 00:16:06,591
that isn't actually done, it just, you

484
00:16:06,601 --> 00:16:08,191
know, sends it back so until it's

485
00:16:08,201 --> 00:16:10,491
actually done and passes everything.

486
00:16:10,501 --> 00:16:11,991
And the threads need a model that's good

487
00:16:12,001 --> 00:16:14,891
at actually doing the work like Opus 55.

488
00:16:14,901 --> 00:16:17,091
And you can set those separately. So,

489
00:16:17,101 --> 00:16:18,692
you know, one model for the main session

490
00:16:18,702 --> 00:16:20,692
and then another one for the threads.

491
00:16:20,702 --> 00:16:22,692
And another benefit is that with this

492
00:16:22,702 --> 00:16:24,392
setup with CloakCode projects

493
00:16:24,402 --> 00:16:26,392
specifically, we don't manage context

494
00:16:26,402 --> 00:16:28,092
anymore. That's all done automatically

495
00:16:28,102 --> 00:16:29,592
for us.

496
00:16:29,602 --> 00:16:31,392
no longer have to think about, you know,

497
00:16:31,402 --> 00:16:33,392
when to compact or clear all of that

498
00:16:33,402 --> 00:16:34,492
stuff.

499
00:16:34,502 --> 00:16:36,692
Okay, so if you look back at those old

500
00:16:36,702 --> 00:16:38,692
best practices, they're all still there,

501
00:16:38,702 --> 00:16:40,392
but they've just moved.

502
00:16:40,402 --> 00:16:42,692
So, the model steers it now, um auto

503
00:16:42,702 --> 00:16:44,592
mode approves your deny rules and the

504
00:16:44,602 --> 00:16:46,892
environment restricts what's possible,

505
00:16:46,902 --> 00:16:48,592
um your tests, your hooks, and your CI

506
00:16:48,602 --> 00:16:50,692
checks it. And then ultimately, Cloud

507
00:16:50,702 --> 00:16:52,892
can just start them as well. So, almost

508
00:16:52,902 --> 00:16:54,892
everything we used to do ourselves is

509
00:16:54,902 --> 00:16:56,992
now either the model, the harness, or

510
00:16:57,002 --> 00:16:58,992
the environment. And we're really just

511
00:16:59,002 --> 00:17:00,792
here to start that first one and to just

512
00:17:00,802 --> 00:17:02,592
review what comes out.

513
00:17:02,602 --> 00:17:03,992
So, if you want to try this in your own

514
00:17:04,002 --> 00:17:06,292
setup, I would say just take one thing

515
00:17:06,302 --> 00:17:07,892
that's way bigger than you normally give

516
00:17:07,902 --> 00:17:10,192
Cloud. Make sure your CI actually

517
00:17:10,202 --> 00:17:12,192
catches the things you care about, and

518
00:17:12,202 --> 00:17:13,192
then spend a bit of time just

519
00:17:13,202 --> 00:17:14,792
configuring your environment, like its

520
00:17:14,802 --> 00:17:16,692
network access, its secret, and its, you

521
00:17:16,702 --> 00:17:18,292
know, setup script.

522
00:17:18,302 --> 00:17:20,092
And then start in the cloud and see

523
00:17:20,102 --> 00:17:21,892
where your sessions get stuck and where

524
00:17:21,902 --> 00:17:23,592
you're still in the loop because that's

525
00:17:23,602 --> 00:17:26,092
the next thing to hand over to Cloud. I

526
00:17:26,102 --> 00:17:27,792
also highly recommend checking out our

527
00:17:27,802 --> 00:17:30,392
best practices in the docs as well.

528
00:17:30,402 --> 00:17:31,000
Thank you so much.