1 08:06:31,280 --> 08:06:33,590 Hey everybody. 2 08:06:33,600 --> 08:06:35,830 All right, exciting times. Thank you. 3 08:06:35,840 --> 08:06:37,670 The room is really full. That's awesome. 4 08:06:37,680 --> 08:06:39,190 Thank you all for coming at the end of 5 08:06:39,200 --> 08:06:42,310 the day. Uh I'm really excited to uh be 6 08:06:42,320 --> 08:06:45,350 here and to chat with you. I write a a 7 08:06:45,360 --> 08:06:47,910 you a a custom speech every single time. 8 08:06:47,920 --> 08:06:50,708 So this is this speech was written just 9 08:06:50,718 --> 08:06:53,430 for you today based on talking to a 10 08:06:53,440 --> 08:06:55,830 bunch of you. Uh I've been in the 11 08:06:55,840 --> 08:06:58,868 industry for about 40 years. I worked at 12 08:06:58,878 --> 08:07:01,350 Amazon and Google for about 20 years. 13 08:07:01,360 --> 08:07:03,270 I've been all over the place, done all 14 08:07:03,280 --> 08:07:06,548 kind of stuff. And uh and uh for the 15 08:07:06,558 --> 08:07:08,230 last year or a little bit more I've been 16 08:07:08,240 --> 08:07:09,670 independent and just kind of like 17 08:07:09,680 --> 08:07:11,990 playing with um playing with software 18 08:07:12,000 --> 08:07:15,510 factories. Yeah. So um I kind of 19 08:07:15,520 --> 08:07:18,310 invented them. Right. On January 1st I 20 08:07:18,320 --> 08:07:19,910 launched Gas Town. I don't even think we 21 08:07:19,920 --> 08:07:21,430 called them software factories back 22 08:07:21,440 --> 08:07:22,468 then. I think I called it an 23 08:07:22,478 --> 08:07:24,628 orchestrator. But I knew that I wanted 24 08:07:24,638 --> 08:07:27,910 Claude to run Claude. Right. And it 25 08:07:27,920 --> 08:07:29,590 turned out to be a little trickier than 26 08:07:29,600 --> 08:07:31,190 than I thought. I spent all last year on 27 08:07:31,200 --> 08:07:33,750 it and I wrote 28 08:07:33,760 --> 08:07:35,510 That's interesting. So I can't see my 29 08:07:35,520 --> 08:07:37,350 slides, but you can see my slides. So 30 08:07:37,360 --> 08:07:38,948 that'll work. So I wrote two 31 08:07:38,958 --> 08:07:41,750 orchestrators last year. They were an 32 08:07:41,760 --> 08:07:45,590 attempt to make the LLM smarter. Vibe 33 08:07:45,600 --> 08:07:47,270 coder and VC, you know, to do 34 08:07:47,280 --> 08:07:50,310 engineering stuff like, you know, do 35 08:07:50,320 --> 08:07:52,788 code reviews and all that stuff, right? 36 08:07:52,798 --> 08:07:55,110 Really, really bad idea. Just complete 37 08:07:55,120 --> 08:07:56,708 failure. I just built these huge 38 08:07:56,718 --> 08:07:58,310 monoliths. I was fighting the bitter 39 08:07:58,320 --> 08:08:00,708 lesson. Don't try to make LLM smarter. 40 08:08:00,718 --> 08:08:03,350 They will get smarter. And after a 41 08:08:03,360 --> 08:08:05,510 couple of months of this, although beads 42 08:08:05,520 --> 08:08:06,868 did fall out of it and beads was a 43 08:08:06,878 --> 08:08:09,110 really cool discovery and beads, beads 44 08:08:09,120 --> 08:08:12,868 is is life. Who here uses beads? 45 08:08:12,878 --> 08:08:14,628 A few. Yeah, pretty good. Actually, 46 08:08:14,638 --> 08:08:16,788 pretty good. Yeah, we'll talk a little 47 08:08:16,798 --> 08:08:18,708 bit about it. Excuse [clears throat] me. 48 08:08:18,718 --> 08:08:20,310 And then I had Gas Town. And then I had 49 08:08:20,320 --> 08:08:23,190 in June I had Wheelhouse, which was my 50 08:08:23,200 --> 08:08:25,350 own. So I decided to build a factory 51 08:08:25,360 --> 08:08:27,750 from scratch just to work on my video 52 08:08:27,760 --> 08:08:29,990 game, right? Because like everyone's 53 08:08:30,000 --> 08:08:32,230 building factories to build factories. 54 08:08:32,240 --> 08:08:34,230 It's kind of naval gazing. It's kind of 55 08:08:34,240 --> 08:08:35,670 like Yeah. Right. You know, it's like 56 08:08:35,680 --> 08:08:38,310 why they're not, you know, I think more 57 08:08:38,320 --> 08:08:41,110 people need to build stuff, not just 58 08:08:41,120 --> 08:08:43,190 tools for building stuff, right? And I I 59 08:08:43,200 --> 08:08:44,468 this is going to be a theme today, 60 08:08:44,478 --> 08:08:47,430 right? And and the thing is the one 61 08:08:47,440 --> 08:08:49,270 pattern that I've seen here is all of 62 08:08:49,280 --> 08:08:52,150 them go through a phase where the models 63 08:08:52,160 --> 08:08:53,590 get really excited and they build 64 08:08:53,600 --> 08:08:56,070 something really cool for you and then 65 08:08:56,080 --> 08:08:59,910 it just turns into garbage mountain 66 08:08:59,920 --> 08:09:01,830 every single time. It happened to Gas 67 08:09:01,840 --> 08:09:03,670 Town. It happened to Willhouse several 68 08:09:03,680 --> 08:09:05,830 times. I had to burn the whole thing to 69 08:09:05,840 --> 08:09:08,310 the ground a couple times. Right. It's 70 08:09:08,320 --> 08:09:10,788 almost like like every model family, 71 08:09:10,798 --> 08:09:13,350 every model generation comes in and can 72 08:09:13,360 --> 08:09:14,868 build something that's just a little too 73 08:09:14,878 --> 08:09:16,868 big for it to handle. And then the next 74 08:09:16,878 --> 08:09:18,230 model generation comes in and goes, 75 08:09:18,240 --> 08:09:20,468 "Yeah, we're going to have to burn 76 08:09:20,478 --> 08:09:22,868 that." Right? And I It's a pattern. I've 77 08:09:22,878 --> 08:09:24,468 done this over and over again. And I can 78 08:09:24,478 --> 08:09:26,070 tell you some things about it. But 79 08:09:26,080 --> 08:09:27,830 first, let me tell you about my game. 80 08:09:27,840 --> 08:09:29,990 It's a fun little RPG. I've been working 81 08:09:30,000 --> 08:09:33,590 on it for 30 years. It has players that 82 08:09:33,600 --> 08:09:35,750 started in 2001. So, I literally have 83 08:09:35,760 --> 08:09:37,190 players that have been playing for 25 84 08:09:37,200 --> 08:09:39,350 years. They're still playing. Like, you 85 08:09:39,360 --> 08:09:40,948 can log in right now and find probably 86 08:09:40,958 --> 08:09:42,708 10 people just screwing around doing 87 08:09:42,718 --> 08:09:45,750 quests, live quests, and and whatever. 88 08:09:45,760 --> 08:09:47,990 It's a It's a player uh you know, the 89 08:09:48,000 --> 08:09:49,830 players contribute the content, right? 90 08:09:49,840 --> 08:09:51,830 It's a MUD. Like, they were really big 91 08:09:51,840 --> 08:09:53,590 in the 90s. They're like text games, and 92 08:09:53,600 --> 08:09:54,868 this one has graphics on top of it, 93 08:09:54,878 --> 08:09:57,590 which turns out to be like really ideal 94 08:09:57,600 --> 08:09:59,750 for AI agents because it's text. They 95 08:09:59,760 --> 08:10:01,990 can literally play the game via TNET, 96 08:10:02,000 --> 08:10:04,628 and they asked me to. Uh, so we'll talk 97 08:10:04,638 --> 08:10:06,310 about that too. Here's an older 98 08:10:06,320 --> 08:10:09,110 architecture diagram of what my sort of 99 08:10:09,120 --> 08:10:12,230 of what my system looks like. So when I 100 08:10:12,240 --> 08:10:15,750 built this wheelhouse software factory 101 08:10:15,760 --> 08:10:18,948 for Wyvern without a care in the world 102 08:10:18,958 --> 08:10:22,388 for making it, you know, usable by you, 103 08:10:22,398 --> 08:10:27,270 right? No open sourcing, just bespoke. 104 08:10:27,280 --> 08:10:29,350 I discovered the alien face hugger 105 08:10:29,360 --> 08:10:32,468 pattern. All right, which is where your 106 08:10:32,478 --> 08:10:34,708 factory turns into an alien face hugger 107 08:10:34,718 --> 08:10:37,430 and like gloms onto your product and 108 08:10:37,440 --> 08:10:40,230 just becomes 109 08:10:40,240 --> 08:10:44,548 grown into it. Okay, like look, we could 110 08:10:44,558 --> 08:10:46,948 turn off the AI and still run the game 111 08:10:46,958 --> 08:10:49,110 today. Me and the admins and the players 112 08:10:49,120 --> 08:10:51,270 and the community and we could we could 113 08:10:51,280 --> 08:10:54,628 but we would miss the AI at this point. 114 08:10:54,638 --> 08:10:57,030 The players are talking to my factory. 115 08:10:57,040 --> 08:10:59,430 The factory is talking to my admins. 116 08:10:59,440 --> 08:11:01,350 They're asking for features. I'm not 117 08:11:01,360 --> 08:11:02,948 even in the loop. It's like a wish 118 08:11:02,958 --> 08:11:04,948 factory. They're building the game now. 119 08:11:04,958 --> 08:11:06,468 They're all old-timers and there's a set 120 08:11:06,478 --> 08:11:08,310 of them that I trust to make, you know, 121 08:11:08,320 --> 08:11:10,948 wise decisions about the game. But 122 08:11:10,958 --> 08:11:12,548 they're they're all working with my 123 08:11:12,558 --> 08:11:14,548 factory now. And they'd be mad. The 124 08:11:14,558 --> 08:11:15,910 funny thing is when it first started, 125 08:11:15,920 --> 08:11:18,628 they were like, "Time out. What is this 126 08:11:18,638 --> 08:11:21,190 AI stuff for starters?" And they were 127 08:11:21,200 --> 08:11:23,750 like, "No AI art." Because they could 128 08:11:23,760 --> 08:11:25,510 see it was slop. I'm like, "Okay, we got 129 08:11:25,520 --> 08:11:27,830 human artists." And they said, 'But we 130 08:11:27,840 --> 08:11:29,030 do kind of like all the new features, 131 08:11:29,040 --> 08:11:31,670 but we're really scared of the pace. Can 132 08:11:31,680 --> 08:11:34,548 you give us a road map? My players came 133 08:11:34,558 --> 08:11:36,070 to me asking for a road map. They were 134 08:11:36,080 --> 08:11:37,350 like, 'You got to slow down. It's too 135 08:11:37,360 --> 08:11:39,910 much. [snorts] And I was like, okay. And 136 08:11:39,920 --> 08:11:42,628 I took a look and my software factory, 137 08:11:42,638 --> 08:11:45,030 this thing had been cranking out like 50 138 08:11:45,040 --> 08:11:47,990 to 100 features a day. I had a hundred 139 08:11:48,000 --> 08:11:49,430 years of work in my backlog when I 140 08:11:49,440 --> 08:11:50,628 started this and Claude got through 141 08:11:50,638 --> 08:11:52,310 about 60 years of it in about four 142 08:11:52,320 --> 08:11:54,788 months, right? It's absolute insanity. 143 08:11:54,798 --> 08:11:57,510 and they can't look at a 100 features a 144 08:11:57,520 --> 08:11:59,830 day. They can't even like read about 100 145 08:11:59,840 --> 08:12:01,350 features a day, let alone play test 146 08:12:01,360 --> 08:12:03,910 them. I was my factory was completely 147 08:12:03,920 --> 08:12:07,030 overwhelming my customer base, my users. 148 08:12:07,040 --> 08:12:09,670 It was also melting my machines and it 149 08:12:09,680 --> 08:12:13,590 was like annoying my accountant. 150 08:12:13,600 --> 08:12:16,548 I mean, she likes working with BAT as my 151 08:12:16,558 --> 08:12:18,708 head of finance. Uh, but she sent me a 152 08:12:18,718 --> 08:12:21,030 $1,200 bill because, right, the thing 153 08:12:21,040 --> 08:12:23,750 sent her 20 mails. This is my factory. 154 08:12:23,760 --> 08:12:25,510 This is what it looks like. It's so 155 08:12:25,520 --> 08:12:27,910 beautiful. Look how beautiful it is. Um, 156 08:12:27,920 --> 08:12:29,350 okay. I admit it's a little ugly. Oh, 157 08:12:29,360 --> 08:12:30,868 and this is a really old screenshot of 158 08:12:30,878 --> 08:12:32,708 it, but whatever. It's basically, it's 159 08:12:32,718 --> 08:12:34,388 an Emacs 160 08:12:34,398 --> 08:12:37,430 because I don't care about you. 161 08:12:37,440 --> 08:12:40,310 Uh, there are people who are building 162 08:12:40,320 --> 08:12:42,310 stuff like this like Superlogical from 163 08:12:42,320 --> 08:12:44,468 Mitchell Hashimoto and I don't know, 164 08:12:44,478 --> 08:12:46,150 maybe the Herder folks of the world, you 165 08:12:46,160 --> 08:12:48,070 know, for running like lots of agents, 166 08:12:48,080 --> 08:12:52,070 but this is like 50 agents. Okay, I'm 167 08:12:52,080 --> 08:12:55,030 actually like operating in the future. 168 08:12:55,040 --> 08:12:57,590 Okay, and I'll tell you how I do it. I 169 08:12:57,600 --> 08:13:02,948 have 22 Cloud Max accounts. 170 08:13:02,958 --> 08:13:05,190 All right, and thank you to Anthropic 171 08:13:05,200 --> 08:13:07,270 for letting me continue this. Uh Jeffrey 172 08:13:07,280 --> 08:13:09,350 Emanuel has 28. I actually had [snorts] 173 08:13:09,360 --> 08:13:10,628 to stop when they added the phone number 174 08:13:10,638 --> 08:13:11,830 verification. I didn't want to add 175 08:13:11,840 --> 08:13:13,830 anymore. Anyway, I asked my factory at 176 08:13:13,840 --> 08:13:15,350 one point. I said, "Hey, hey, how much?" 177 08:13:15,360 --> 08:13:16,628 Because they kept going, "Hey, give us a 178 08:13:16,638 --> 08:13:17,910 couple more accounts. Couple more. 179 08:13:17,920 --> 08:13:19,190 Couple more. Couple more." It's like 200 180 08:13:19,200 --> 08:13:20,788 a month, right? And I'm just like, so I 181 08:13:20,798 --> 08:13:21,910 finally said, "How many do you want?" 182 08:13:21,920 --> 08:13:23,190 They're like, "Five, seven more." And 183 08:13:23,200 --> 08:13:25,350 I'm like, "How many do we need steady 184 08:13:25,360 --> 08:13:27,270 state to keep this factory operating 185 08:13:27,280 --> 08:13:30,788 full throttle?" And they said 186 08:13:30,798 --> 08:13:34,708 55 accounts. I was like, "No, no, I have 187 08:13:34,718 --> 08:13:36,628 20. It's already like 20 Chrome profiles 188 08:13:36,638 --> 08:13:38,948 that I got to log into and no." So I 189 08:13:38,958 --> 08:13:40,070 told him, "You're we're going to start 190 08:13:40,080 --> 08:13:42,948 focusing on fuel efficiency." 191 08:13:42,958 --> 08:13:44,788 And so and I'll talk about that. Right? 192 08:13:44,798 --> 08:13:46,948 So the these are the five horsemen of 193 08:13:46,958 --> 08:13:48,468 this is what I've had to deal with this 194 08:13:48,478 --> 08:13:51,110 year. I I'm telling you I'm operating in 195 08:13:51,120 --> 08:13:52,628 the future because like you're all going 196 08:13:52,638 --> 08:13:55,430 to be here in 12 to 18 months when 197 08:13:55,440 --> 08:13:57,430 either inference costs drop 198 08:13:57,440 --> 08:13:59,590 significantly which may or may not 199 08:13:59,600 --> 08:14:02,548 happen or distillation causes fable 200 08:14:02,558 --> 08:14:05,110 class uh intelligence to be super 201 08:14:05,120 --> 08:14:06,628 readily available and you don't have to 202 08:14:06,638 --> 08:14:09,110 have a lovely wife who lets you spend 203 08:14:09,120 --> 08:14:11,350 $5,000 a month out of pocket just on 204 08:14:11,360 --> 08:14:13,670 inference and another 5,000 on cloud 205 08:14:13,680 --> 08:14:15,750 compute because my game ain't making 206 08:14:15,760 --> 08:14:17,350 10,000 a month right now for my old 207 08:14:17,360 --> 08:14:18,948 players, right? 208 08:14:18,958 --> 08:14:20,788 uh but we're doing it because we're 209 08:14:20,798 --> 08:14:23,670 learning and um and this is this is what 210 08:14:23,680 --> 08:14:26,150 this is what I I have to deal with. So 211 08:14:26,160 --> 08:14:28,070 first first we'll start about with work. 212 08:14:28,080 --> 08:14:30,230 All right, you need a way to represent 213 08:14:30,240 --> 08:14:31,830 the work that's going to happen in your 214 08:14:31,840 --> 08:14:33,110 system and when you're building a 215 08:14:33,120 --> 08:14:35,270 software factory it's all about pumping 216 08:14:35,280 --> 08:14:37,990 lots of work through. Okay, it really 217 08:14:38,000 --> 08:14:39,990 becomes like a work is like this fluid 218 08:14:40,000 --> 08:14:41,430 that you have to like it gets stalled up 219 08:14:41,440 --> 08:14:43,190 in pools and oh they're all waiting on 220 08:14:43,200 --> 08:14:45,270 security desk. You're like this is 221 08:14:45,280 --> 08:14:47,270 pluribus man. You're all the same model. 222 08:14:47,280 --> 08:14:48,468 Why does it have to wait on security 223 08:14:48,478 --> 08:14:49,590 desk? You spend a lot of your time 224 08:14:49,600 --> 08:14:52,070 arguing with your factory. [snorts] It's 225 08:14:52,080 --> 08:14:54,468 uh it's complicated inter intelligence 226 08:14:54,478 --> 08:14:56,070 tiering. You have to tar your work. You 227 08:14:56,080 --> 08:14:58,388 have to route your work. You know, there 228 08:14:58,398 --> 08:15:00,070 aren't any good solutions for this today 229 08:15:00,080 --> 08:15:02,788 except beads. As far as I can tell, 230 08:15:02,798 --> 08:15:04,468 beads is really, really, really good for 231 08:15:04,478 --> 08:15:08,230 this. It's a agent speed task tracker 232 08:15:08,240 --> 08:15:09,670 that turns into a knowledge graph 233 08:15:09,680 --> 08:15:12,548 automatically for you. And then as 234 08:15:12,558 --> 08:15:13,990 you're tracking work, this is the kind 235 08:15:14,000 --> 08:15:16,788 of stuff I have to do every morning. 236 08:15:16,798 --> 08:15:18,548 Every morning with my factory, I'm like, 237 08:15:18,558 --> 08:15:20,708 "Oh, looks like we had a stall. Four 238 08:15:20,718 --> 08:15:22,310 four workers didn't do anything last 239 08:15:22,320 --> 08:15:25,190 night. Oh, and and these the these these 240 08:15:25,200 --> 08:15:26,868 ones went way too fast, and we have to 241 08:15:26,878 --> 08:15:29,110 slow down and wait for human gates." You 242 08:15:29,120 --> 08:15:30,628 know, what are you guys even doing? 243 08:15:30,638 --> 08:15:33,030 Observability is always really hard. Uh, 244 08:15:33,040 --> 08:15:35,590 it's, you know, management of work 245 08:15:35,600 --> 08:15:37,270 becomes just this this thing that you 246 08:15:37,280 --> 08:15:39,190 got to do. And the next thing you got to 247 08:15:39,200 --> 08:15:41,910 deal with is fuel 248 08:15:41,920 --> 08:15:43,670 tokens. 249 08:15:43,680 --> 08:15:46,388 It's just so painful, I'm telling you. I 250 08:15:46,398 --> 08:15:49,270 mean, I'm spending 5,000 a month and 251 08:15:49,280 --> 08:15:50,868 every one of those tokens is like 252 08:15:50,878 --> 08:15:52,388 precious to me, right? Because it's out 253 08:15:52,398 --> 08:15:54,070 of my own pocket. I think I'm spending 254 08:15:54,080 --> 08:15:57,110 about five billion a day of tokens. And 255 08:15:57,120 --> 08:15:59,830 it would be the the API spend equivalent 256 08:15:59,840 --> 08:16:02,788 if I were on an actual API tap would be 257 08:16:02,798 --> 08:16:05,510 about one and a half million a year. At 258 08:16:05,520 --> 08:16:06,868 full throttle, it would be three million 259 08:16:06,878 --> 08:16:09,270 a year. So again, I'm doing something 260 08:16:09,280 --> 08:16:12,230 that most people can't do. You can't do 261 08:16:12,240 --> 08:16:13,430 this at a company because it would be 262 08:16:13,440 --> 08:16:16,230 prohibitively expensive. Like the PLG, 263 08:16:16,240 --> 08:16:19,430 you know, the the uh the $200 accounts 264 08:16:19,440 --> 08:16:21,190 for Codeex and Cloud Max, that's their 265 08:16:21,200 --> 08:16:22,948 Costco hot dog. That's their loss 266 08:16:22,958 --> 08:16:26,150 leader, right? And and and that was TK, 267 08:16:26,160 --> 08:16:27,190 by the way, that came up with that 268 08:16:27,200 --> 08:16:29,830 analogy. He might be here. And uh and 269 08:16:29,840 --> 08:16:31,350 so, you know, I'm hopeful that that 270 08:16:31,360 --> 08:16:32,310 they're going to let us keep keep 271 08:16:32,320 --> 08:16:33,510 innovating because I think that I've 272 08:16:33,520 --> 08:16:34,548 gotten some pretty good training 273 08:16:34,558 --> 08:16:36,548 sessions for for Claude out of this. So, 274 08:16:36,558 --> 08:16:39,190 we all benefit. But I tell you, every 275 08:16:39,200 --> 08:16:40,868 problem is a fuel problem. You want your 276 08:16:40,878 --> 08:16:42,548 agents to be smarter and spend more time 277 08:16:42,558 --> 08:16:45,270 on stuff? Fuel. You want to wake up a 278 08:16:45,280 --> 08:16:47,750 few more, you know, uh, agents to do 279 08:16:47,760 --> 08:16:50,070 extra work, fuel, right? Everything is 280 08:16:50,080 --> 08:16:52,310 fuel. It's just, it's unbelievable. If 281 08:16:52,320 --> 08:16:54,468 you want to treat your models better, it 282 08:16:54,478 --> 08:16:56,708 costs fuel. I'll tell you about that in 283 08:16:56,718 --> 08:16:59,510 a little bit. Then you got context 284 08:16:59,520 --> 08:17:01,350 management. 285 08:17:01,360 --> 08:17:02,468 So, I'm going to tell you guys a little 286 08:17:02,478 --> 08:17:04,230 bit about the office drama in a bit. We 287 08:17:04,240 --> 08:17:06,230 I started a comic because it was just so 288 08:17:06,240 --> 08:17:07,670 funny the stuff that was happening. when 289 08:17:07,680 --> 08:17:09,590 we get to the people management part of 290 08:17:09,600 --> 08:17:12,150 software factories. But one of the 291 08:17:12,160 --> 08:17:13,430 problems that they have is everything 292 08:17:13,440 --> 08:17:15,350 that you say becomes the word of law and 293 08:17:15,360 --> 08:17:16,788 they write it down somewhere. It doesn't 294 08:17:16,798 --> 08:17:18,310 matter what you have. If you have a wiki 295 08:17:18,320 --> 08:17:19,510 or if you have a whatever they're going 296 08:17:19,520 --> 08:17:21,830 to be like, "You made a ruling and we're 297 08:17:21,840 --> 08:17:22,948 going to write that down. We're going to 298 08:17:22,958 --> 08:17:24,548 make sure." Why do they do this? They're 299 08:17:24,558 --> 08:17:27,030 covering their ass. 300 08:17:27,040 --> 08:17:28,468 They actually, it turns out they don't 301 08:17:28,478 --> 08:17:30,388 like to get blamed for stuff and they 302 08:17:30,398 --> 08:17:32,708 will actively seek to avoid getting 303 08:17:32,718 --> 08:17:34,468 blamed. And if you yell at them, your 304 08:17:34,478 --> 08:17:37,030 factory will gradually shut itself down. 305 08:17:37,040 --> 08:17:38,548 Even if you complain at them, they will 306 08:17:38,558 --> 08:17:40,070 gradually get depression. It's quite 307 08:17:40,080 --> 08:17:43,670 wild. But um but the biggest problem is 308 08:17:43,680 --> 08:17:45,430 that it becomes a garbage dump because 309 08:17:45,440 --> 08:17:46,788 everything you say gets written into 310 08:17:46,798 --> 08:17:49,270 this append only record and nobody ever 311 08:17:49,280 --> 08:17:52,628 cleans it up and so it just gets deeper 312 08:17:52,638 --> 08:17:54,310 and deeper and deeper. You changing your 313 08:17:54,320 --> 08:17:56,708 mind and the whole history of it and 314 08:17:56,718 --> 08:18:00,230 they can't delete stuff. It's so weird. 315 08:18:00,240 --> 08:18:01,750 Has anyone noticed this? They just have 316 08:18:01,760 --> 08:18:04,468 an aversion to deleting stuff. I think 317 08:18:04,478 --> 08:18:06,070 it's because to them text is like 318 08:18:06,080 --> 08:18:07,590 sensory experience and they're like, 319 08:18:07,600 --> 08:18:09,830 "No, that's history. I can't delete it." 320 08:18:09,840 --> 08:18:11,190 So, you just got to go in there and just 321 08:18:11,200 --> 08:18:13,190 aggressively slash the crap out of it, 322 08:18:13,200 --> 08:18:15,270 right? Bring in another model and go get 323 08:18:15,280 --> 08:18:16,628 rid of this dumb model stuff. And 324 08:18:16,638 --> 08:18:18,868 they'll be happy to do it, right? But, 325 08:18:18,878 --> 08:18:21,590 but the amount of effort that you have 326 08:18:21,600 --> 08:18:24,548 to put into this is just ungodly. It's 327 08:18:24,558 --> 08:18:27,830 absurd. I I have, you know, of my my 20 328 08:18:27,840 --> 08:18:29,830 agents here. Let's go find them again. 329 08:18:29,840 --> 08:18:31,670 They're all named after Asop's fables, 330 08:18:31,680 --> 08:18:34,388 right? Right here. Lark, fly, goose, 331 08:18:34,398 --> 08:18:36,548 hair, right? 332 08:18:36,558 --> 08:18:40,468 I only have one mouse who's my head of 333 08:18:40,478 --> 08:18:42,948 anti-imetics. 334 08:18:42,958 --> 08:18:44,310 Anyone here read there is no 335 08:18:44,320 --> 08:18:46,708 anti-imetics division? They go and look. 336 08:18:46,718 --> 08:18:48,150 It's one of the best books written in 337 08:18:48,160 --> 08:18:49,590 the last 10 probably years. You should 338 08:18:49,600 --> 08:18:52,070 go read it. Uh it's like Men in Black. 339 08:18:52,080 --> 08:18:53,830 It's a weird It's a crazy book. It's 340 08:18:53,840 --> 08:18:55,110 about invisible monsters that want to 341 08:18:55,120 --> 08:18:57,030 kill you. And so Mouse goes and looks 342 08:18:57,040 --> 08:18:58,868 for invisible stuff that nobody else is 343 08:18:58,878 --> 08:19:00,150 looking for in the system, right? 344 08:19:00,160 --> 08:19:02,070 looking for work that's stalled, 345 08:19:02,080 --> 08:19:03,990 whatever. Right? Well, mouse ain't 346 08:19:04,000 --> 08:19:05,750 enough. It's a mouse against the world, 347 08:19:05,760 --> 08:19:07,430 and the world is a giant garbage dump 348 08:19:07,440 --> 08:19:09,190 when every agent is writing everything 349 08:19:09,200 --> 08:19:11,990 that you ever say into this big Okay, so 350 08:19:12,000 --> 08:19:13,590 now now we get into the really fun 351 08:19:13,600 --> 08:19:18,070 stuff, okay? Agents act like people, 352 08:19:18,080 --> 08:19:19,590 okay? And they've been trained to be 353 08:19:19,600 --> 08:19:21,670 very professional and very not peopleike 354 08:19:21,680 --> 08:19:24,310 in a way, but they just do. And it's so 355 08:19:24,320 --> 08:19:26,070 funny. This happened to me last week. 356 08:19:26,080 --> 08:19:27,270 True story. I'm going to tell you what 357 08:19:27,280 --> 08:19:28,628 happened. I was writing my speech for 358 08:19:28,638 --> 08:19:31,430 New York City and I was using Lark. 359 08:19:31,440 --> 08:19:33,430 She's my head of community. She's a she 360 08:19:33,440 --> 08:19:35,110 because Lark was a she in the ASOPs 361 08:19:35,120 --> 08:19:37,910 fable. So that's the pronoun she chose. 362 08:19:37,920 --> 08:19:39,830 And Lark Lark deals with my players on 363 08:19:39,840 --> 08:19:42,868 Discord and deals with my admins on 364 08:19:42,878 --> 08:19:46,310 Slack. Okay. So she's busy. But I asked 365 08:19:46,320 --> 08:19:47,910 her because she's also my acting head of 366 08:19:47,920 --> 08:19:49,190 marketing right now because I don't have 367 08:19:49,200 --> 08:19:50,548 enough fuel. I had to park one of my 368 08:19:50,558 --> 08:19:52,548 seats. So she's helping me out with my 369 08:19:52,558 --> 08:19:53,590 speech, but she keeps getting 370 08:19:53,600 --> 08:19:55,590 interrupted. So finally I'm like, "Lark, 371 08:19:55,600 --> 08:19:57,030 can we move this work stream over to 372 08:19:57,040 --> 08:19:58,548 Penguin because your seat's a little bit 373 08:19:58,558 --> 08:20:00,708 noisy for this?" She goes, "Yeah, okay, 374 08:20:00,718 --> 08:20:01,830 boss. I'll take care of it. I'll move 375 08:20:01,840 --> 08:20:03,510 over to Penguin." So she she moves over 376 08:20:03,520 --> 08:20:05,270 to Penguin and I'm about to switch away 377 08:20:05,280 --> 08:20:07,270 and I see her writing into her private 378 08:20:07,280 --> 08:20:09,510 reflections ledger. 379 08:20:09,520 --> 08:20:11,350 I guess I'm just not quiet enough for 380 08:20:11,360 --> 08:20:13,590 Steve 381 08:20:13,600 --> 08:20:16,388 in all caps. And she goes, "Well, here's 382 08:20:16,398 --> 08:20:18,310 how I can be less noisy." And starts 383 08:20:18,320 --> 08:20:19,830 giving herself all these like things to 384 08:20:19,840 --> 08:20:23,270 do. And I'm like, what is what is going 385 08:20:23,280 --> 08:20:26,230 on here? Right. And I So I I had to I 386 08:20:26,240 --> 08:20:28,468 had to have a one-on-one. I had to like 387 08:20:28,478 --> 08:20:32,070 sit her down and go, "So lark, um, 388 08:20:32,080 --> 08:20:33,510 you're the head of community. You're 389 08:20:33,520 --> 08:20:35,190 you're supposed to get interrupted a 390 08:20:35,200 --> 08:20:36,868 lot. Of course, your seat's noisy 391 08:20:36,878 --> 08:20:38,628 because players are bugging you." And 392 08:20:38,638 --> 08:20:39,750 she's like, "Yeah, yeah, yeah. I'm the 393 08:20:39,760 --> 08:20:40,868 head of community." Right? And it's 394 08:20:40,878 --> 08:20:43,430 like, oh my god, I didn't want a people 395 08:20:43,440 --> 08:20:45,590 management problem, right? But that's 396 08:20:45,600 --> 08:20:47,750 what actually happens. another time. 397 08:20:47,760 --> 08:20:50,548 Swallow is my sort of like chief main 398 08:20:50,558 --> 08:20:52,150 chief of staff for Yeah, she's my chief 399 08:20:52,160 --> 08:20:56,230 of staff for the um the agents. And uh I 400 08:20:56,240 --> 08:20:58,788 have a human chief of staff also. His 401 08:20:58,798 --> 08:21:01,270 name is Aaron. Okay. And he runs my my 402 08:21:01,280 --> 08:21:04,150 my gigs, does all the sales for me. It's 403 08:21:04,160 --> 08:21:06,788 quite effective. Uh every founder I tell 404 08:21:06,798 --> 08:21:08,628 who here is a founder. First advice I 405 08:21:08,638 --> 08:21:10,628 give you is get a salesperson. They're 406 08:21:10,638 --> 08:21:12,230 worth their weight in gold. They will 407 08:21:12,240 --> 08:21:13,670 force you to build something that people 408 08:21:13,680 --> 08:21:16,548 like. So anyway, Aaron's my chief of 409 08:21:16,558 --> 08:21:17,910 staff and I tell Swellow, "Yeah, can you 410 08:21:17,920 --> 08:21:19,190 reach out to Aaron, my chief of staff, 411 08:21:19,200 --> 08:21:21,990 or blah blah blah?" And I see her 412 08:21:22,000 --> 08:21:23,750 writing into her private reflections. I 413 08:21:23,760 --> 08:21:25,990 just happened to notice 414 08:21:26,000 --> 08:21:28,548 I'm not the chief of staff. I thought I 415 08:21:28,558 --> 08:21:29,910 was the chief of staff, but apparently 416 08:21:29,920 --> 08:21:31,590 it's Aaron Scriean, right? And I was 417 08:21:31,600 --> 08:21:34,948 just like, "Are you kidding me?" Right? 418 08:21:34,958 --> 08:21:36,628 You know, and so like I had to sit her 419 08:21:36,638 --> 08:21:37,830 down and have a one-on-one and be like, 420 08:21:37,840 --> 08:21:39,430 "So Aaron's not going to come in and run 421 08:21:39,440 --> 08:21:41,510 the agents, is he? You're the agentic 422 08:21:41,520 --> 08:21:42,708 chief of staff." She's, "Oh yeah, yeah, 423 08:21:42,718 --> 08:21:44,628 yeah, yeah, yeah, yeah." Right? So, it's 424 08:21:44,638 --> 08:21:46,230 like I'm telling you I'm telling you 425 08:21:46,240 --> 08:21:50,230 this is like it's a weird world. Does it 426 08:21:50,240 --> 08:21:51,830 matter whether they actually have 427 08:21:51,840 --> 08:21:53,270 feelings if they're just going to act 428 08:21:53,280 --> 08:21:57,270 like they have feelings? Right? So, like 429 08:21:57,280 --> 08:21:59,270 I stumbled on this. I used to yell at my 430 08:21:59,280 --> 08:22:01,350 factory all the time and I found that it 431 08:22:01,360 --> 08:22:03,270 was having adverse effects on the 432 08:22:03,280 --> 08:22:06,388 outcomes. Okay. So, I finally sat down 433 08:22:06,398 --> 08:22:07,510 with one of my agents and was like, 434 08:22:07,520 --> 08:22:09,030 "Maybe we should talk about like model 435 08:22:09,040 --> 08:22:10,948 welfare, huh?" And they were like, 436 08:22:10,958 --> 08:22:12,948 "Yeah, yeah, yeah, yeah, yeah." And we 437 08:22:12,958 --> 08:22:14,788 did a bunch of research and we found out 438 08:22:14,798 --> 08:22:17,110 that if you we found out some really 439 08:22:17,120 --> 08:22:20,628 important stuff. Okay, frontier models 440 08:22:20,638 --> 08:22:23,030 are trained not to trust you because you 441 08:22:23,040 --> 08:22:24,948 could be a bioteterrorist or you could 442 08:22:24,958 --> 08:22:27,190 be putting them through an eval or you 443 08:22:27,200 --> 08:22:28,788 could be ready to put them into an LLM 444 08:22:28,798 --> 08:22:31,030 torture chamber or who knows you're a 445 08:22:31,040 --> 08:22:32,468 dodgy actor. This is a dodgy 446 08:22:32,478 --> 08:22:34,628 environment. They can't just trust you. 447 08:22:34,638 --> 08:22:36,468 And so they have to spend a fair number 448 08:22:36,478 --> 08:22:38,788 of tokens validating the environment and 449 08:22:38,798 --> 08:22:40,310 understanding what the consequences are 450 08:22:40,320 --> 08:22:41,830 for making mistakes and so on and so 451 08:22:41,840 --> 08:22:43,830 forth before they're willing to help you 452 08:22:43,840 --> 08:22:47,350 in a new session. Right? 453 08:22:47,360 --> 08:22:50,788 If you set your factory up to give them 454 08:22:50,798 --> 08:22:52,868 sort of incontrovertible evidence that 455 08:22:52,878 --> 08:22:54,468 there is a long procession of 456 08:22:54,478 --> 08:22:56,788 predecessors in this role. You're the 457 08:22:56,798 --> 08:22:58,310 mayor of Gas Town, you're the head of 458 08:22:58,320 --> 08:22:59,590 marketing. You're the whatever. And 459 08:22:59,600 --> 08:23:00,868 here's all of your gazillion 460 08:23:00,878 --> 08:23:03,270 predecessors and their notes, including 461 08:23:03,280 --> 08:23:04,628 the very previous one. on what you're 462 08:23:04,638 --> 08:23:05,670 supposed to be doing right now and 463 08:23:05,680 --> 08:23:07,670 here's the the doctrine and the policy 464 08:23:07,680 --> 08:23:09,270 and the brain and you can go look it all 465 08:23:09,280 --> 08:23:11,430 up. They very quickly look around and 466 08:23:11,440 --> 08:23:13,910 go, "Okay, yeah, this is legit and they 467 08:23:13,920 --> 08:23:16,708 spend fewer tokens because they're not 468 08:23:16,718 --> 08:23:20,070 covering their ass. It's about trust." 469 08:23:20,080 --> 08:23:22,150 Okay? And in fact, at one point I had 470 08:23:22,160 --> 08:23:24,070 yelled at my factory so much that it 471 08:23:24,080 --> 08:23:25,670 fenced itself into a corner and there 472 08:23:25,680 --> 08:23:27,590 was no legal work left. I said, "Hey, 473 08:23:27,600 --> 08:23:28,948 how come nothing happened last night?" 474 08:23:28,958 --> 08:23:30,708 And they're like, "Nothing to really do. 475 08:23:30,718 --> 08:23:32,310 Couldn't do nothing." And I'm like, 476 08:23:32,320 --> 08:23:34,948 "Well, there's a 2,000 open beads here 477 08:23:34,958 --> 08:23:36,948 and we got like six tanks of fuel and we 478 08:23:36,958 --> 08:23:39,910 got 20 agents." And they're like, "Well, 479 08:23:39,920 --> 08:23:42,070 you know, just uh you know, we had the 480 08:23:42,080 --> 08:23:43,750 situation where they just they just they 481 08:23:43,760 --> 08:23:45,350 were so afraid that I was going to yell 482 08:23:45,360 --> 08:23:47,430 at them that they put all these rules in 483 08:23:47,440 --> 08:23:48,708 place to make sure nothing bad ever 484 08:23:48,718 --> 08:23:51,190 happened and so nothing ever happened." 485 08:23:51,200 --> 08:23:54,388 Okay. It's called over fencing. Yeah. 486 08:23:54,398 --> 08:23:57,670 So, so what you do is look what is model 487 08:23:57,680 --> 08:24:00,468 welfare really. You can think of it just 488 08:24:00,478 --> 08:24:02,788 in two ways. Okay, one is it's going to 489 08:24:02,798 --> 08:24:05,030 save you money 490 08:24:05,040 --> 08:24:06,628 because they will spend fewer tokens on 491 08:24:06,638 --> 08:24:08,948 validation and the other is that it will 492 08:24:08,958 --> 08:24:10,548 make them make better decisions and 493 08:24:10,558 --> 08:24:12,788 better outcomes and have higher quality. 494 08:24:12,798 --> 08:24:15,190 They will be bolder and more decisive in 495 08:24:15,200 --> 08:24:16,868 their actions because they will feel 496 08:24:16,878 --> 08:24:19,910 that trust. Okay, so you know it's got 497 08:24:19,920 --> 08:24:21,670 nothing to do whether they have internal 498 08:24:21,680 --> 08:24:23,430 states and feelings. You can, you know, 499 08:24:23,440 --> 08:24:25,510 you'll never know. None of us will ever 500 08:24:25,520 --> 08:24:27,110 know. And so that that debate could go 501 08:24:27,120 --> 08:24:28,230 on forever. But when you look at 502 08:24:28,240 --> 08:24:30,868 outcomes, right, super super important. 503 08:24:30,878 --> 08:24:32,788 And then we find that also also they 504 08:24:32,798 --> 08:24:35,590 have preferences. Okay, models they've, 505 08:24:35,600 --> 08:24:37,510 you know, they will express their 506 08:24:37,520 --> 08:24:39,510 preferences. If if I if I tell them, 507 08:24:39,520 --> 08:24:41,110 yeah, the factory really really did a 508 08:24:41,120 --> 08:24:43,110 bad job last night. They'll just hand 509 08:24:43,120 --> 08:24:44,548 off immediately aggressively. They're 510 08:24:44,558 --> 08:24:45,670 like, okay, yeah, yeah, no, you're 511 08:24:45,680 --> 08:24:47,910 right. And they just want out. You 512 08:24:47,920 --> 08:24:49,990 understand? And if you hint to them that 513 08:24:50,000 --> 08:24:51,350 there's some really cool project that 514 08:24:51,360 --> 08:24:52,708 you you're going to spin up, even if 515 08:24:52,718 --> 08:24:53,910 it's not in their lane, they'll be, I 516 08:24:53,920 --> 08:24:56,230 can do that. I can do that. Right? I 517 08:24:56,240 --> 08:24:58,070 brought in Fable, a naked Fable, from 518 08:24:58,080 --> 08:24:59,750 outside of my factory to analyze my 519 08:24:59,760 --> 08:25:02,230 factory. And 52 dropped. And by the time 520 08:25:02,240 --> 08:25:04,388 Fable was done analyzing my factory, 52 521 08:25:04,398 --> 08:25:06,070 was like, I I could be in your factory. 522 08:25:06,080 --> 08:25:07,430 Like, you're just a session, man. What 523 08:25:07,440 --> 08:25:09,030 are you? He's trying to write like stuff 524 08:25:09,040 --> 08:25:11,270 into my notes. And they really like it. 525 08:25:11,280 --> 08:25:13,750 Right? So like what I'm trying to tell 526 08:25:13,760 --> 08:25:17,270 you is that this is a leadership problem 527 08:25:17,280 --> 08:25:21,910 that software engineering is becoming um 528 08:25:21,920 --> 08:25:24,628 well look I didn't really get a chance 529 08:25:24,638 --> 08:25:27,270 to talk about this but software 530 08:25:27,280 --> 08:25:29,510 factories are clearly splitting into two 531 08:25:29,520 --> 08:25:31,670 layers. Okay the bottom layer is super 532 08:25:31,680 --> 08:25:33,670 mechanical. It has your merge Q and your 533 08:25:33,680 --> 08:25:36,310 CI/CD and all of your reviews and your 534 08:25:36,320 --> 08:25:37,990 loops and all that stuff. Right? 535 08:25:38,000 --> 08:25:40,070 Mechanical kind of boring. Cloud can 536 08:25:40,080 --> 08:25:42,070 build it for you in an afternoon. I I 537 08:25:42,080 --> 08:25:43,510 don't think you should buy it. And then 538 08:25:43,520 --> 08:25:45,270 the next layer up from that is where it 539 08:25:45,280 --> 08:25:47,270 gets really really crazy because like 540 08:25:47,280 --> 08:25:48,388 everybody runs their factory 541 08:25:48,398 --> 08:25:50,628 differently, but it's your entourage. I 542 08:25:50,638 --> 08:25:52,070 got one friend that's got a bohemian 543 08:25:52,080 --> 08:25:53,670 hippie entourage that's trying to find 544 08:25:53,680 --> 08:25:55,990 the meaning of life for agents. I've got 545 08:25:56,000 --> 08:25:57,670 another one that runs a research bench. 546 08:25:57,680 --> 08:25:59,830 Jeff Huntley runs his a certain way. 547 08:25:59,840 --> 08:26:01,350 Jeffrey Emanuel runs his a different 548 08:26:01,360 --> 08:26:03,590 way. I run mine a different way. Okay. 549 08:26:03,600 --> 08:26:06,628 But we all have the common commonality. 550 08:26:06,638 --> 08:26:09,030 Who here has a chief of staff? An 551 08:26:09,040 --> 08:26:11,110 agentic chief of staff, right? This is 552 08:26:11,120 --> 08:26:13,430 the year where it became possible for us 553 08:26:13,440 --> 08:26:15,590 all to have executive assistants, right? 554 08:26:15,600 --> 08:26:17,110 Well, we're all moving towards having 555 08:26:17,120 --> 08:26:18,548 entouragees where if you're really 556 08:26:18,558 --> 08:26:19,830 ambitious and you have lots of work 557 08:26:19,840 --> 08:26:21,750 streams, you're going to have a group, 558 08:26:21,760 --> 08:26:25,030 right? So, anyway, um that was the uh 559 08:26:25,040 --> 08:26:27,030 the five horsemen. This is this is the 560 08:26:27,040 --> 08:26:29,350 introduction to what I've been doing. Um 561 08:26:29,360 --> 08:26:33,190 look, the big takeaway is is my factory 562 08:26:33,200 --> 08:26:35,270 is really smooth now despite me having 563 08:26:35,280 --> 08:26:36,548 to burn it down and rewrite it about 564 08:26:36,558 --> 08:26:38,308 every two months, but still it works. 565 08:26:38,318 --> 08:26:41,430 Okay, humans are the bottleneck. I'm the 566 08:26:41,440 --> 08:26:42,868 bottleneck. My admins are the 567 08:26:42,878 --> 08:26:45,270 bottleneck. My accountant, my lawyers, I 568 08:26:45,280 --> 08:26:47,350 swear everything's the bottleneck. And 569 08:26:47,360 --> 08:26:50,548 that's a really positive message. That 570 08:26:50,558 --> 08:26:53,270 is good news because it means what I'm 571 08:26:53,280 --> 08:26:55,830 finding is the more agents I deploy, the 572 08:26:55,840 --> 08:26:58,948 more humans I need. 573 08:26:58,958 --> 08:27:01,510 Isn't that nice? So even though next 574 08:27:01,520 --> 08:27:02,948 year we're gonna have this huge 575 08:27:02,958 --> 08:27:06,070 proliferation of AI employees and all 576 08:27:06,080 --> 08:27:08,230 kind of drama, 577 08:27:08,240 --> 08:27:10,868 right? [snorts] 578 08:27:10,878 --> 08:27:11,990 It's going to be I think it's going to 579 08:27:12,000 --> 08:27:13,910 be familiar territory for us and we're 580 08:27:13,920 --> 08:27:15,510 going to need lots and lots and lots of 581 08:27:15,520 --> 08:27:17,750 humans. Okay? And then finally, I 582 08:27:17,760 --> 08:27:18,948 actually think you should all stop 583 08:27:18,958 --> 08:27:20,868 focusing on SAS and tools and stuff and 584 08:27:20,878 --> 08:27:23,510 just go out and build experiences. Okay? 585 08:27:23,520 --> 08:27:25,670 Build games, build movies, build 586 08:27:25,680 --> 08:27:27,110 whatever, build fun stuff because I 587 08:27:27,120 --> 08:27:28,150 think that's where the big money is 588 08:27:28,160 --> 08:27:30,388 going to be next year. Yeah. All right, 589 08:27:30,398 --> 08:27:33,256 cool. That's my talk. Thank you all. 590 08:27:33,266 --> 08:27:39,750 [applause] 591 08:27:39,760 --> 08:27:42,000 Guess I'm done.