1 01:37:50,639 --> 01:37:52,390 >> Let me start with one number from last 2 01:37:52,400 --> 01:37:55,750 month. On average, more than one agent 3 01:37:55,760 --> 01:37:58,149 has started on our platform every single 4 01:37:58,159 --> 01:38:01,350 minute, day and night. 5 01:38:01,360 --> 01:38:03,910 And honestly, the agents were the easy 6 01:38:03,920 --> 01:38:05,910 part. 7 01:38:05,920 --> 01:38:09,270 Hi, I'm Joey. I bu the cloud agent 8 01:38:09,280 --> 01:38:11,910 platform we built inside Harvey. 9 01:38:11,920 --> 01:38:15,350 After 10,000 PRs, the biggest lesson I 10 01:38:15,360 --> 01:38:17,910 can share is this. The thing that broke 11 01:38:17,920 --> 01:38:20,950 was an agent. It was everything around 12 01:38:20,960 --> 01:38:23,109 them. 13 01:38:23,119 --> 01:38:26,229 So what does one submit add up to? 14 01:38:26,239 --> 01:38:30,550 10,000 PRs that an agent wrote. And it 15 01:38:30,560 --> 01:38:34,310 isn't just Tax team playing with a toy. 16 01:38:34,320 --> 01:38:38,629 Last month about 460 different people at 17 01:38:38,639 --> 01:38:42,229 Harvey use it. And over the summer it 18 01:38:42,239 --> 01:38:44,950 stopped being only about code. People 19 01:38:44,960 --> 01:38:47,430 started asking a data question in Slack 20 01:38:47,440 --> 01:38:51,350 and getting answers back. My favorite 21 01:38:51,360 --> 01:38:53,350 recently someone on our marketing team 22 01:38:53,360 --> 01:38:55,510 drop a prototype and a talk track into 23 01:38:55,520 --> 01:38:58,470 Slack and ask for a story book for a 24 01:38:58,480 --> 01:39:02,149 product video. Once the harness exists, 25 01:39:02,159 --> 01:39:03,990 people bring the work that you never 26 01:39:04,000 --> 01:39:06,629 plan for. 27 01:39:06,639 --> 01:39:09,430 So a promise up front. This is not a 28 01:39:09,440 --> 01:39:12,310 talk about models, Evos or leaderboard. 29 01:39:12,320 --> 01:39:15,750 This is talk about hardness, 30 01:39:15,760 --> 01:39:18,950 odd things around the model, 31 01:39:18,960 --> 01:39:20,870 checking out the ripple, holding the 32 01:39:20,880 --> 01:39:24,229 key, opening the PR, getting the merged. 33 01:39:24,239 --> 01:39:26,229 When we first wrote about Spectre back 34 01:39:26,239 --> 01:39:29,990 in April, we put it like this. At some 35 01:39:30,000 --> 01:39:33,750 point, a harness become the product. 36 01:39:33,760 --> 01:39:35,990 Now, whenever I tell people this, I get 37 01:39:36,000 --> 01:39:38,390 the same question. Why on earth would 38 01:39:38,400 --> 01:39:41,030 you build this when clock or cursor 39 01:39:41,040 --> 01:39:43,350 already exist? 40 01:39:43,360 --> 01:39:46,470 My answer, we don't compete with it. We 41 01:39:46,480 --> 01:39:49,750 run them. And here's how I see it as my 42 01:39:49,760 --> 01:39:52,550 own view. Agents on your laptop are 43 01:39:52,560 --> 01:39:55,350 powerful, but they don't scale. You can 44 01:39:55,360 --> 01:39:58,390 realistically run five to 10 of them. 45 01:39:58,400 --> 01:40:01,109 Pass that you need a cloud. And cloud 46 01:40:01,119 --> 01:40:04,629 needs our secrets, our real stack, our 47 01:40:04,639 --> 01:40:08,149 alerts and tickets, one set of rules and 48 01:40:08,159 --> 01:40:10,470 enough capacity that nobody gets 49 01:40:10,480 --> 01:40:13,270 starved. Those single two owns that for 50 01:40:13,280 --> 01:40:16,310 us. So we build layer and that layer 51 01:40:16,320 --> 01:40:19,270 exactly what broke. So I want to tell 52 01:40:19,280 --> 01:40:22,149 three stories today. How we stop 53 01:40:22,159 --> 01:40:24,790 competing on writing code. How we let 54 01:40:24,800 --> 01:40:28,629 any agent drive and were broke once they 55 01:40:28,639 --> 01:40:31,590 did. The first story start with the 56 01:40:31,600 --> 01:40:34,470 question we keep hearing. 57 01:40:34,480 --> 01:40:37,430 More of our engineers were using cursor 58 01:40:37,440 --> 01:40:40,629 and they ask how do I get our secret 59 01:40:40,639 --> 01:40:42,950 there? How do I test against our real 60 01:40:42,960 --> 01:40:45,990 stack? We could have rebuilt all that 61 01:40:46,000 --> 01:40:48,470 inside cursor and then again for next 62 01:40:48,480 --> 01:40:51,830 two and the one after that. 63 01:40:51,840 --> 01:40:55,109 Instead answer became one sentence just 64 01:40:55,119 --> 01:40:58,229 ask cursor to use spectre 65 01:40:58,239 --> 01:41:00,629 and that's really the whole idea. People 66 01:41:00,639 --> 01:41:03,109 will always have a favorite agent and 67 01:41:03,119 --> 01:41:05,430 the favorite keeps changing. So pick 68 01:41:05,440 --> 01:41:09,430 yours clock codeex cursor dot slack are 69 01:41:09,440 --> 01:41:11,510 your own hands. 70 01:41:11,520 --> 01:41:13,990 Start on your laptop push the session to 71 01:41:14,000 --> 01:41:17,030 the cloud. Put it back. That's how 72 01:41:17,040 --> 01:41:20,070 spectra sits underneath all of that. 73 01:41:20,080 --> 01:41:22,149 Because once the code was written the 74 01:41:22,159 --> 01:41:24,310 real work starts 75 01:41:24,320 --> 01:41:28,310 picture appears day alert fires in Slack 76 01:41:28,320 --> 01:41:31,590 at 2 in the morning. An agent picks up 77 01:41:31,600 --> 01:41:35,510 nobody typing. It writes a fix, spins up 78 01:41:35,520 --> 01:41:38,310 real copy of our stack and post the 79 01:41:38,320 --> 01:41:40,390 proof on the PR. 80 01:41:40,400 --> 01:41:43,109 When C fails, the follow-up loop reads 81 01:41:43,119 --> 01:41:45,830 the failure and pushes another fix while 82 01:41:45,840 --> 01:41:49,270 everyone still sleep. By morning the PR 83 01:41:49,280 --> 01:41:52,310 sits in one review queue under the same 84 01:41:52,320 --> 01:41:56,070 rule as other PR whichever agents wrote 85 01:41:56,080 --> 01:41:59,830 it. The owner gets a notch and only once 86 01:41:59,840 --> 01:42:02,629 they are working day starts and once 87 01:42:02,639 --> 01:42:07,750 approved a agent can automerge it. 88 01:42:07,760 --> 01:42:10,310 If you only build one of those stages 89 01:42:10,320 --> 01:42:13,830 that I described build a follow-up loop. 90 01:42:13,840 --> 01:42:15,750 We switched it on for everyone in early 91 01:42:15,760 --> 01:42:18,550 October and that months. Four out of 92 01:42:18,560 --> 01:42:22,550 five agent PRs merge into our backend PR 93 01:42:22,560 --> 01:42:24,950 had been through it. 94 01:42:24,960 --> 01:42:26,709 That's the difference between agent that 95 01:42:26,719 --> 01:42:29,510 write code and an agent that finished 96 01:42:29,520 --> 01:42:31,510 the job. 97 01:42:31,520 --> 01:42:33,669 And then there's the knowledge. Every 98 01:42:33,679 --> 01:42:35,669 team has that one person who knows 99 01:42:35,679 --> 01:42:38,310 exactly how to test the tricky service 100 01:42:38,320 --> 01:42:41,270 which flag you never touch. And that 101 01:42:41,280 --> 01:42:43,990 knowledge leaves in their head. of their 102 01:42:44,000 --> 01:42:45,669 laptop. 103 01:42:45,679 --> 01:42:49,350 So we gave it a hope projects. 104 01:42:49,360 --> 01:42:51,910 People write down what matters. Adrian 105 01:42:51,920 --> 01:42:54,470 write down what they learn. Team pings 106 01:42:54,480 --> 01:42:56,310 the skills you trust and there's a 107 01:42:56,320 --> 01:42:59,910 record of who rent what. Any agent 108 01:42:59,920 --> 01:43:01,430 working in the project read the same 109 01:43:01,440 --> 01:43:04,870 page. My personal view whichever agent 110 01:43:04,880 --> 01:43:07,270 you pick the audit trail and the tracks 111 01:43:07,280 --> 01:43:09,910 should be the same. 112 01:43:09,920 --> 01:43:13,030 The second story is about rule. 113 01:43:13,040 --> 01:43:15,430 Back in May, if you want a spectctor to 114 01:43:15,440 --> 01:43:18,709 do something, you open a browser tab or 115 01:43:18,719 --> 01:43:22,310 tacking Slack. The whole mouse, exactly 116 01:43:22,320 --> 01:43:25,030 one task came out from terminal. 117 01:43:25,040 --> 01:43:27,030 So, we made a rule. If a person can 118 01:43:27,040 --> 01:43:30,950 click it, an agent can call it 119 01:43:30,960 --> 01:43:33,430 start a task, turn on the follow-up 120 01:43:33,440 --> 01:43:36,709 loop, arm, automerge, publish a page, 121 01:43:36,719 --> 01:43:39,109 all from CLI, 122 01:43:39,119 --> 01:43:41,109 which means your laptop agent can call 123 01:43:41,119 --> 01:43:44,310 it A agent inside the sandbox can also 124 01:43:44,320 --> 01:43:47,750 call it. Even CI can call it too. And 125 01:43:47,760 --> 01:43:51,189 then people ran with it. By October, the 126 01:43:51,199 --> 01:43:54,790 command line was starting about 6500 ts 127 01:43:54,800 --> 01:43:58,229 a month. The web app didn't shrink. 128 01:43:58,239 --> 01:44:01,830 Everyone else just grew around it. Task 129 01:44:01,840 --> 01:44:04,470 started from the terminal were merging 130 01:44:04,480 --> 01:44:06,950 and task started in browser. And the new 131 01:44:06,960 --> 01:44:10,470 kind of user showed up. people who start 132 01:44:10,480 --> 01:44:12,709 hundreds of tasks a month and never open 133 01:44:12,719 --> 01:44:15,030 the app at all. 134 01:44:15,040 --> 01:44:17,590 And this is where it gets fun. In 135 01:44:17,600 --> 01:44:21,189 October, I had a big task. So I bought 136 01:44:21,199 --> 01:44:22,790 code session and asked to be the 137 01:44:22,800 --> 01:44:24,709 coordinator. 138 01:44:24,719 --> 01:44:27,030 It split the tickets into 10 different 139 01:44:27,040 --> 01:44:30,709 milestones. Get one for me and the rest 140 01:44:30,719 --> 01:44:34,070 to spectre with hundreds of tasks. 141 01:44:34,080 --> 01:44:36,870 Each has its own sandbox to validate end 142 01:44:36,880 --> 01:44:40,070 to end. Then at the end of the October, 143 01:44:40,080 --> 01:44:43,430 our it give everyone tragic dots. Mind 144 01:44:43,440 --> 01:44:46,149 now watch Spectre for me. It list the 145 01:44:46,159 --> 01:44:48,550 question Spectre is waiting on. Draft my 146 01:44:48,560 --> 01:44:51,270 answers and once I say yes, send them 147 01:44:51,280 --> 01:44:53,109 back. 148 01:44:53,119 --> 01:44:56,470 I actually DM to my friend like that. I 149 01:44:56,480 --> 01:44:58,790 just use cursor clock code or whatever 150 01:44:58,800 --> 01:45:01,590 to spam spectre. Now 151 01:45:01,600 --> 01:45:03,590 well that's the dream, but it's also the 152 01:45:03,600 --> 01:45:07,750 problem. A quick detour. When we look at 153 01:45:07,760 --> 01:45:10,310 where our task time actually went, the 154 01:45:10,320 --> 01:45:13,030 slowest step wasn't model syncing. It 155 01:45:13,040 --> 01:45:15,669 was get just checking out every ripple 156 01:45:15,679 --> 01:45:20,550 we own took over a minute on average. 157 01:45:20,560 --> 01:45:23,030 The fix keep a copy of every ripple on 158 01:45:23,040 --> 01:45:26,229 every machine and let each test borrow 159 01:45:26,239 --> 01:45:27,910 it. 160 01:45:27,920 --> 01:45:30,229 The third story is the one where it 161 01:45:30,239 --> 01:45:34,390 broke in mid August. A release goes out 162 01:45:34,400 --> 01:45:37,430 with a bug. That bug triggers alert that 163 01:45:37,440 --> 01:45:40,390 fires once per workspace. So one bug 164 01:45:40,400 --> 01:45:43,350 become hundreds of alerts all landing in 165 01:45:43,360 --> 01:45:46,470 one single sack channel. Someone has set 166 01:45:46,480 --> 01:45:48,950 up workflow that tag our agent on every 167 01:45:48,960 --> 01:45:53,030 alert and every tag started task. On 168 01:45:53,040 --> 01:45:55,510 normal day those task started about 50 169 01:45:55,520 --> 01:45:59,109 task. That day in about 4 hours they 170 01:45:59,119 --> 01:46:02,070 started around 900 171 01:46:02,080 --> 01:46:04,310 900 agents all chasing the same bug 172 01:46:04,320 --> 01:46:06,950 until that wasn't room left for a single 173 01:46:06,960 --> 01:46:08,470 note. 174 01:46:08,480 --> 01:46:11,189 The fix was almost boring. A repeat 175 01:46:11,199 --> 01:46:13,350 alert now joins the investigation that's 176 01:46:13,360 --> 01:46:15,750 already open. 177 01:46:15,760 --> 01:46:18,550 Two months later it happened again and 178 01:46:18,560 --> 01:46:22,870 this time it wasn't a bot. On October, 179 01:46:22,880 --> 01:46:25,669 the engineer did exactly what we've been 180 01:46:25,679 --> 01:46:28,229 asking everyone to do. Hand work to 181 01:46:28,239 --> 01:46:30,149 Spectre. 182 01:46:30,159 --> 01:46:34,070 About 950 test from the command line in 183 01:46:34,080 --> 01:46:36,950 about 3 hours. Nothing in our platform 184 01:46:36,960 --> 01:46:40,470 can work. The cluster filled up and 185 01:46:40,480 --> 01:46:43,430 everyone else work set in line behind 186 01:46:43,440 --> 01:46:46,310 one laptop. That's not a user problem. 187 01:46:46,320 --> 01:46:49,910 That's our problem. 188 01:46:49,920 --> 01:46:52,709 Four days later, we had a real fix. A 189 01:46:52,719 --> 01:46:56,229 fail launch queue. Now starting a task 190 01:46:56,239 --> 01:46:58,629 gives you a place in line. And the 191 01:46:58,639 --> 01:47:01,669 dispatcher started when there's room. 192 01:47:01,679 --> 01:47:04,629 People sitting at a keyboard goes first. 193 01:47:04,639 --> 01:47:06,629 Everyone gets a fair share and can 194 01:47:06,639 --> 01:47:09,910 borrow more than the cluster is quite. 195 01:47:09,920 --> 01:47:12,390 And whenever you start a task, you can 196 01:47:12,400 --> 01:47:14,790 see our placing line. When we hit it 197 01:47:14,800 --> 01:47:18,310 with a 300 task bursting low test, the 198 01:47:18,320 --> 01:47:20,550 task lit went from almost two second to 199 01:47:20,560 --> 01:47:22,790 70 milliseconds. 200 01:47:22,800 --> 01:47:25,189 So did it work? 201 01:47:25,199 --> 01:47:28,790 When it's the busiest day of every week 202 01:47:28,800 --> 01:47:31,669 back in August, two certain task and 203 01:47:31,679 --> 01:47:35,030 about one in hand didn't finish. 204 01:47:35,040 --> 01:47:37,430 Then in November 4th, more than twice 205 01:47:37,440 --> 01:47:41,750 that and much better in the finish line, 206 01:47:41,760 --> 01:47:45,030 but we still have some room to grow. 207 01:47:45,040 --> 01:47:46,870 I will be modest, we change several 208 01:47:46,880 --> 01:47:49,030 things at once, so I cannot tell you 209 01:47:49,040 --> 01:47:51,270 which fix mattered the most, but the 210 01:47:51,280 --> 01:47:53,430 birds kept getting bigger and the 211 01:47:53,440 --> 01:47:56,310 platform kept up. 212 01:47:56,320 --> 01:47:58,470 And just when that was on the crunch, 213 01:47:58,480 --> 01:48:00,870 the model neck moves. 214 01:48:00,880 --> 01:48:03,990 At the end of October, our dev platform 215 01:48:04,000 --> 01:48:07,510 team raised an alarm. PRS across the 216 01:48:07,520 --> 01:48:10,550 company had jumped at around 60% in a 217 01:48:10,560 --> 01:48:13,669 single month. A merge that used to take 218 01:48:13,679 --> 01:48:16,470 15 to 30 minutes was taking over an hour 219 01:48:16,480 --> 01:48:19,109 now. And merge Q kept having to be 220 01:48:19,119 --> 01:48:22,950 switched off. So now a PR rebase onto 221 01:48:22,960 --> 01:48:27,109 main rerun CI and merge if it's green. 222 01:48:27,119 --> 01:48:29,830 If main breaks, everyone CI breaks with 223 01:48:29,840 --> 01:48:32,470 it. Alterately, 224 01:48:32,480 --> 01:48:35,270 we haven't cracked this one yet. 225 01:48:35,280 --> 01:48:39,030 But before I go, there's one more thing. 226 01:48:39,040 --> 01:48:42,070 While we were busy fixing all it, 227 01:48:42,080 --> 01:48:45,109 something else was quite growing. Most 228 01:48:45,119 --> 01:48:47,270 of what I've been showing you start with 229 01:48:47,280 --> 01:48:49,270 a person asking 230 01:48:49,280 --> 01:48:52,629 the fastest growing source of the work 231 01:48:52,639 --> 01:48:55,109 on Spectre doesn't. These are 232 01:48:55,119 --> 01:48:57,669 automations agent that wake up on 233 01:48:57,679 --> 01:49:00,629 schedule on an event and just do the 234 01:49:00,639 --> 01:49:04,550 work tri alert review the PR write the 235 01:49:04,560 --> 01:49:06,550 post. 236 01:49:06,560 --> 01:49:09,270 Last month about 110 people had 237 01:49:09,280 --> 01:49:11,430 automation running and we learned to 238 01:49:11,440 --> 01:49:14,870 give them a guardrail on day one. If one 239 01:49:14,880 --> 01:49:17,270 fails five times in a row, it switch 240 01:49:17,280 --> 01:49:21,430 itself off and it can carry a budget or 241 01:49:21,440 --> 01:49:23,910 expiry date because when no one's 242 01:49:23,920 --> 01:49:26,229 watching, an agent can start thousands 243 01:49:26,239 --> 01:49:28,390 of tasks, too. 244 01:49:28,400 --> 01:49:31,430 So, here's what I'm chasing next, and I 245 01:49:31,440 --> 01:49:34,390 don't have an answer yet, honestly. 246 01:49:34,400 --> 01:49:36,629 What if every Slack channel had its own 247 01:49:36,639 --> 01:49:39,910 agent that never locks off? One that 248 01:49:39,920 --> 01:49:41,669 follows competition and start work on 249 01:49:41,679 --> 01:49:44,709 its own. What if every pure proof it 250 01:49:44,719 --> 01:49:48,310 works before a human even look at it? 251 01:49:48,320 --> 01:49:51,350 Today only one merge in 20 carries that 252 01:49:51,360 --> 01:49:53,830 kind of proof. 253 01:49:53,840 --> 01:49:56,229 And how do you merge when agents never 254 01:49:56,239 --> 01:49:58,790 stop opening PRs? 255 01:49:58,800 --> 01:50:01,590 Ask me again six months. I will tell you 256 01:50:01,600 --> 01:50:04,149 what broke. If you're building something 257 01:50:04,159 --> 01:50:06,390 like this when you work, here's three 258 01:50:06,400 --> 01:50:09,189 things I do again. Oh, what happens 259 01:50:09,199 --> 01:50:12,950 after code is written. Make Eric action 260 01:50:12,960 --> 01:50:16,149 something an agent can call and cap what 261 01:50:16,159 --> 01:50:19,350 an agent, human or alert can start 262 01:50:19,360 --> 01:50:21,510 before they find out what they can do 263 01:50:21,520 --> 01:50:23,430 with it. 264 01:50:23,440 --> 01:50:25,990 And one last thought, that's my own 265 01:50:26,000 --> 01:50:29,109 view. Engineers can write code with 266 01:50:29,119 --> 01:50:32,070 whichever agent they like. The work 267 01:50:32,080 --> 01:50:34,790 that's lost is the harness that runs it, 268 01:50:34,800 --> 01:50:38,950 checks it, and get it merged. 269 01:50:38,960 --> 01:50:41,590 Thank you so much for watching. 270 01:50:41,600 --> 01:50:43,350 Come find me on linking if you are 271 01:50:43,360 --> 01:50:46,149 building similar things.