1 05:06:58,400 --> 05:06:59,750 >> Give me a sec to make sure this is 2 05:06:59,760 --> 05:07:03,590 working. Heck yeah. I'm Diego Almeida 3 05:07:03,600 --> 05:07:06,628 and this talk is I love you so jeving 4 05:07:06,638 --> 05:07:09,830 much. Um glass animals reference. We're 5 05:07:09,840 --> 05:07:12,708 not just weird. Um like I said, I'm 6 05:07:12,718 --> 05:07:15,430 Diego Almeida. I'm known as the Jev guy, 7 05:07:15,440 --> 05:07:18,230 sometimes Jevy Pinkman. Um, also 8 05:07:18,240 --> 05:07:21,270 co-created chat GPT, RLHF, and Instruct 9 05:07:21,280 --> 05:07:23,830 GPT, but I assume that for the crowd 10 05:07:23,840 --> 05:07:25,830 today, you all know who I am, at least 11 05:07:25,840 --> 05:07:30,230 mostly. Um, I have like a bunch of AI 12 05:07:30,240 --> 05:07:31,670 specific stuff I want to get to, but I 13 05:07:31,680 --> 05:07:33,590 like going into my favorite topics in 14 05:07:33,600 --> 05:07:36,628 AI. First one is what the is 15 05:07:36,638 --> 05:07:38,948 actually going on and where is all the 16 05:07:38,958 --> 05:07:40,628 automation. If you've heard me say this 17 05:07:40,638 --> 05:07:42,708 before, I'm sorry, but it's such an 18 05:07:42,718 --> 05:07:44,550 important thing to beat into the field 19 05:07:44,560 --> 05:07:46,470 because people still do not get it and I 20 05:07:46,480 --> 05:07:48,788 need to yell this until everyone really 21 05:07:48,798 --> 05:07:51,030 understands what the is going on. 22 05:07:51,040 --> 05:07:52,628 And if you're an AI, you should have a 23 05:07:52,638 --> 05:07:55,190 really good answer about why AI can do 24 05:07:55,200 --> 05:07:57,190 so many awesome things and yet still be 25 05:07:57,200 --> 05:07:59,990 so useless for a bunch of other ones. My 26 05:08:00,000 --> 05:08:02,310 relationship to chat GPT is somewhat 27 05:08:02,320 --> 05:08:04,390 unique. One of the few people who hates 28 05:08:04,400 --> 05:08:07,510 on chat GPT who made it. I don't hate 29 05:08:07,520 --> 05:08:10,230 chat GPT to be clear. I love using 30 05:08:10,240 --> 05:08:11,990 agents. I love using coding agents. I 31 05:08:12,000 --> 05:08:13,590 love chat bots. They're just quite 32 05:08:13,600 --> 05:08:15,510 limited. And in particular, I think it's 33 05:08:15,520 --> 05:08:17,750 a little tragic how much they've 34 05:08:17,760 --> 05:08:19,830 bifurcated the field into overpromise, 35 05:08:19,840 --> 05:08:21,910 underdeliver. Really, really terrible 36 05:08:21,920 --> 05:08:24,628 stuff for the world in my opinion. And I 37 05:08:24,638 --> 05:08:26,550 like to talk about like the two cults in 38 05:08:26,560 --> 05:08:29,270 AI and how we can resolve the cognitive 39 05:08:29,280 --> 05:08:31,990 dissonance that everyone I think should 40 05:08:32,000 --> 05:08:33,590 be having unless they have a very good 41 05:08:33,600 --> 05:08:36,070 answer. There's cult one, which is, you 42 05:08:36,080 --> 05:08:38,550 know, the cult of like EAs. They look at 43 05:08:38,560 --> 05:08:40,230 all the benchmarks and they say like not 44 05:08:40,240 --> 05:08:43,110 only are LM surpassing human level on 45 05:08:43,120 --> 05:08:45,190 basically every benchmark they have been 46 05:08:45,200 --> 05:08:47,510 for two years we've been announcing AGI 47 05:08:47,520 --> 05:08:49,910 three months away for two years now even 48 05:08:49,920 --> 05:08:51,350 since like the 03 days do people 49 05:08:51,360 --> 05:08:52,948 remember that no one even car cares 50 05:08:52,958 --> 05:08:55,110 about that model anymore and it was that 51 05:08:55,120 --> 05:08:57,030 that was like peak internal to openai 52 05:08:57,040 --> 05:08:59,430 AGI hype and on the other hand you look 53 05:08:59,440 --> 05:09:02,390 at the nonbenchmarks and you look at the 54 05:09:02,400 --> 05:09:04,070 reality and you see the rest of the 55 05:09:04,080 --> 05:09:05,590 world especially if you guys go outside 56 05:09:05,600 --> 05:09:07,510 and leave San Francisco a lot of the 57 05:09:07,520 --> 05:09:09,990 world is like really unchanged despite 58 05:09:10,000 --> 05:09:12,868 AI being so cool. And I think AI 59 05:09:12,878 --> 05:09:14,868 is transformative. It's a rare point of 60 05:09:14,878 --> 05:09:17,030 view to actually think AI is cool and 61 05:09:17,040 --> 05:09:19,430 transformative and also like 62 05:09:19,440 --> 05:09:21,670 underwhelming the world right now. And 63 05:09:21,680 --> 05:09:23,270 the tricky part is this is the same 64 05:09:23,280 --> 05:09:25,190 field and we are all seeing the same 65 05:09:25,200 --> 05:09:28,550 facts. So what gives you know why is AI 66 05:09:28,560 --> 05:09:30,628 jagged so to speak like why can it be so 67 05:09:30,638 --> 05:09:33,270 good at some things and so not good at 68 05:09:33,280 --> 05:09:36,310 other things and what is the aams razor 69 05:09:36,320 --> 05:09:38,550 explanation for that um I will be 70 05:09:38,560 --> 05:09:40,470 blasting through my slides fast I'm 71 05:09:40,480 --> 05:09:42,230 sorry maybe I could slow down for people 72 05:09:42,240 --> 05:09:45,670 taking pictures um but uh I'm noticing 73 05:09:45,680 --> 05:09:47,910 that uh this is going to go fast is all 74 05:09:47,920 --> 05:09:50,550 I'm saying so I think that we as a field 75 05:09:50,560 --> 05:09:52,708 should have an aams razor explanation 76 05:09:52,718 --> 05:09:55,910 for why are there some things that are 77 05:09:55,920 --> 05:09:57,590 Not just good, they're unbelievably 78 05:09:57,600 --> 05:09:59,750 good, superhuman, like millennium price 79 05:09:59,760 --> 05:10:02,070 problems in math. But on the other hand, 80 05:10:02,080 --> 05:10:03,590 they're not just bad, they're so 81 05:10:03,600 --> 05:10:05,430 useless, they can't even be used to 82 05:10:05,440 --> 05:10:07,270 touch accounting. That sounds way easier 83 05:10:07,280 --> 05:10:08,550 than million prize problems in math, 84 05:10:08,560 --> 05:10:09,910 right? I could do the latter. I 85 05:10:09,920 --> 05:10:11,110 definitely could not do the former. And 86 05:10:11,120 --> 05:10:13,750 I was a mathlete. So what gives here? 87 05:10:13,760 --> 05:10:16,230 What is the reason for this? And I 88 05:10:16,240 --> 05:10:17,830 normally love this as an interactive 89 05:10:17,840 --> 05:10:19,430 thing, but I will instead just tell you 90 05:10:19,440 --> 05:10:21,510 my answer. Sorry for the power dynamic. 91 05:10:21,520 --> 05:10:24,150 Um, everything on the left side is 92 05:10:24,160 --> 05:10:26,390 intrinsically a human in the loop task. 93 05:10:26,400 --> 05:10:28,230 Like deep research, I could not do 94 05:10:28,240 --> 05:10:29,590 better than deep research. The models 95 05:10:29,600 --> 05:10:32,470 are so good at that. But if gun 96 05:10:32,480 --> 05:10:34,550 to my head or gun to your head, you had 97 05:10:34,560 --> 05:10:36,230 to pick between me and deep research. 98 05:10:36,240 --> 05:10:38,070 And the answer had to be correct, not 99 05:10:38,080 --> 05:10:39,590 well written, you'd probably pick me 100 05:10:39,600 --> 05:10:41,910 every time. And that's the distinction 101 05:10:41,920 --> 05:10:44,470 between like optimizing for automation, 102 05:10:44,480 --> 05:10:46,310 which tends to be a lot easier and a lot 103 05:10:46,320 --> 05:10:49,430 simpler. Yet m AI keeps failing at this. 104 05:10:49,440 --> 05:10:51,110 Every year there's like an infamous 105 05:10:51,120 --> 05:10:53,110 failure of AI failing to even automate 106 05:10:53,120 --> 05:10:55,430 drive-throughs. And that should be kind 107 05:10:55,440 --> 05:10:58,628 of shocking to people. And my claim is 108 05:10:58,638 --> 05:11:00,470 actually for these tasks pleasing a 109 05:11:00,480 --> 05:11:02,310 human is nothing to do with doing the 110 05:11:02,320 --> 05:11:04,310 task. What you want to do is basically 111 05:11:04,320 --> 05:11:06,470 be an abstraction away. You want the 112 05:11:06,480 --> 05:11:08,310 decision to be made and you want to get 113 05:11:08,320 --> 05:11:10,868 back into doing automation and code is 114 05:11:10,878 --> 05:11:12,070 the language of automation. Therefore, 115 05:11:12,080 --> 05:11:13,830 you need to integrate with code. This is 116 05:11:13,840 --> 05:11:15,910 a lot but I will summarize it again. I 117 05:11:15,920 --> 05:11:18,628 get the irony that uh you know I've 118 05:11:18,638 --> 05:11:20,948 co-created RHF and I do believe RLHF is 119 05:11:20,958 --> 05:11:23,110 part of the problem and RHF in 120 05:11:23,120 --> 05:11:26,470 particular is why all AI today does 121 05:11:26,480 --> 05:11:28,390 require a human in the loop with an 122 05:11:28,400 --> 05:11:30,230 asterisk unless the task is like very 123 05:11:30,240 --> 05:11:32,150 trivially verifiable and can run on its 124 05:11:32,160 --> 05:11:33,750 own but still requires a human in the 125 05:11:33,760 --> 05:11:35,750 loop to verify it afterwards. It is 126 05:11:35,760 --> 05:11:37,750 because we literally put it in that is 127 05:11:37,760 --> 05:11:40,868 what the HF is in RHF. And my claim is 128 05:11:40,878 --> 05:11:43,430 that there's a better way. You know, my 129 05:11:43,440 --> 05:11:45,670 summary of the state is 100% of LMS 130 05:11:45,680 --> 05:11:47,750 today are optimized for assistance. You 131 05:11:47,760 --> 05:11:49,510 might talk about RLVR, but the RLVR 132 05:11:49,520 --> 05:11:51,590 models are very bad to interact with if 133 05:11:51,600 --> 05:11:53,990 they've not been RHFed. The result is 134 05:11:54,000 --> 05:11:56,310 LMS become too good to be true at 135 05:11:56,320 --> 05:11:57,990 assistance. They are just so 136 05:11:58,000 --> 05:11:59,350 good. Like, you should never pick me 137 05:11:59,360 --> 05:12:02,310 over chat GPT for your chat GPT queries. 138 05:12:02,320 --> 05:12:05,590 But for actually doing like you know run 139 05:12:05,600 --> 05:12:07,350 in the background cognitive work you 140 05:12:07,360 --> 05:12:09,110 should not trust LMS at all because they 141 05:12:09,120 --> 05:12:10,788 have not been optimized for that which 142 05:12:10,798 --> 05:12:12,708 is why they're so bad at the tasks of 143 05:12:12,718 --> 05:12:14,628 automation and I would also guess that 144 05:12:14,638 --> 05:12:16,310 automation is a lot more valuable than 145 05:12:16,320 --> 05:12:18,470 assistants because you can do way cooler 146 05:12:18,480 --> 05:12:19,910 deeper I'm going to get into this later 147 05:12:19,920 --> 05:12:21,110 but I think there's going to be so much 148 05:12:21,120 --> 05:12:22,868 stuff that we can do and we are just 149 05:12:22,878 --> 05:12:24,550 scratching the surface of that right now 150 05:12:24,560 --> 05:12:26,868 and that's the tragic thing about AI uh 151 05:12:26,878 --> 05:12:30,150 the state of AI for me let's talk about 152 05:12:30,160 --> 05:12:33,590 the evolution Um, I I can't tell if this 153 05:12:33,600 --> 05:12:36,470 is appropriate, but it's too late. Um, 154 05:12:36,480 --> 05:12:38,550 oh god, don't take pictures of this one. 155 05:12:38,560 --> 05:12:42,390 Um, cool. Oh, I'm getting a thumbs up. 156 05:12:42,400 --> 05:12:44,470 Yeah. Uh, excellent. So, we're we 157 05:12:44,480 --> 05:12:46,390 instead of doing RHF, we just have to do 158 05:12:46,400 --> 05:12:47,510 something else. We need to think from 159 05:12:47,520 --> 05:12:49,350 first principles. How do we optimize for 160 05:12:49,360 --> 05:12:51,030 machine native AI? You guys hopefully 161 05:12:51,040 --> 05:12:52,470 know the interface of Jev and that is 162 05:12:52,480 --> 05:12:54,310 what we are doing. more important than 163 05:12:54,320 --> 05:12:57,350 optimizing for its interface, we are uh 164 05:12:57,360 --> 05:12:59,910 rethinking how AI should be done because 165 05:12:59,920 --> 05:13:02,948 this is what we did again for Arlf for 166 05:13:02,958 --> 05:13:05,110 Sutton's bitter lesson. I'm going to 167 05:13:05,120 --> 05:13:06,868 loosely paraphrase to make my point 168 05:13:06,878 --> 05:13:10,230 easier to say is that scaling um and 169 05:13:10,240 --> 05:13:12,230 compute matters more than algorithms. 170 05:13:12,240 --> 05:13:13,910 This is why people become like scaling 171 05:13:13,920 --> 05:13:16,470 pill jironies. And I would argue that 172 05:13:16,480 --> 05:13:18,628 there's an even bitter lesson than that 173 05:13:18,638 --> 05:13:20,708 which is data always matters more than 174 05:13:20,718 --> 05:13:22,310 compute. We are learning that quite a 175 05:13:22,320 --> 05:13:23,910 bit in the field now, but it has been 176 05:13:23,920 --> 05:13:25,990 obvious since the beginning of time. 177 05:13:26,000 --> 05:13:27,590 I'll tell you when the this is the one 178 05:13:27,600 --> 05:13:29,110 you should be photographing. I would 179 05:13:29,120 --> 05:13:31,830 argue that the the bitterest lesson is 180 05:13:31,840 --> 05:13:33,830 doing the right task. And doing the 181 05:13:33,840 --> 05:13:35,350 right task is like a weird thing to 182 05:13:35,360 --> 05:13:37,430 think about especially for ML people 183 05:13:37,440 --> 05:13:39,750 because they like algorithms. They want 184 05:13:39,760 --> 05:13:41,590 number goes down or up or whichever is 185 05:13:41,600 --> 05:13:42,550 good direction. They don't want to think 186 05:13:42,560 --> 05:13:44,708 about the real world. But this is always 187 05:13:44,718 --> 05:13:46,628 how you make gigantic gains in 188 05:13:46,638 --> 05:13:47,990 performance. This is how the big 189 05:13:48,000 --> 05:13:49,910 breakthroughs happen. 100% of the time 190 05:13:49,920 --> 05:13:52,708 it comes from this and this is true for 191 05:13:52,718 --> 05:13:54,628 classical RL games. This is why Sutton 192 05:13:54,638 --> 05:13:57,190 had his bitter lesson it which is data 193 05:13:57,200 --> 05:13:59,110 is derived by compute when you're doing 194 05:13:59,120 --> 05:14:00,788 rollouts like there's there's why would 195 05:14:00,798 --> 05:14:02,628 you need off policy data when playing 196 05:14:02,638 --> 05:14:04,708 chess or something like that and doing 197 05:14:04,718 --> 05:14:06,470 the right task is defined by the rules. 198 05:14:06,480 --> 05:14:07,990 So actually when you're doing RL you 199 05:14:08,000 --> 05:14:09,270 don't even need to think about the right 200 05:14:09,280 --> 05:14:12,390 task for pre-training we actually as a 201 05:14:12,400 --> 05:14:14,070 field had to do a lot of thinking about 202 05:14:14,080 --> 05:14:15,750 the right task. I don't know if people 203 05:14:15,760 --> 05:14:18,070 remember the birth days with um masked 204 05:14:18,080 --> 05:14:20,550 language modeling, but it took a while 205 05:14:20,560 --> 05:14:22,310 and it was very unclear to figure out 206 05:14:22,320 --> 05:14:24,708 that next pro token prediction was super 207 05:14:24,718 --> 05:14:26,708 cool. Not because it was better 208 05:14:26,718 --> 05:14:28,310 at masked language modeling. You know, 209 05:14:28,320 --> 05:14:30,150 the objectives on their own aren't 210 05:14:30,160 --> 05:14:31,670 special. It's about what you can do with 211 05:14:31,680 --> 05:14:33,510 them afterwards. And it turns out next 212 05:14:33,520 --> 05:14:34,868 token prediction is incredible at 213 05:14:34,878 --> 05:14:36,628 generating strings, which is what these 214 05:14:36,638 --> 05:14:38,708 pre-trained models were so good at. So 215 05:14:38,718 --> 05:14:40,550 no level of mass language modeling even 216 05:14:40,560 --> 05:14:41,750 if you had today's compute and 217 05:14:41,760 --> 05:14:43,430 algorithms would make it as interesting 218 05:14:43,440 --> 05:14:46,150 as GPT3 was because it was just doing a 219 05:14:46,160 --> 05:14:48,948 different task and data you know the 220 05:14:48,958 --> 05:14:50,230 internet is obviously a giant 221 05:14:50,240 --> 05:14:51,430 bottleneck. Now we're like you know 222 05:14:51,440 --> 05:14:52,868 burning books or whatever in order to 223 05:14:52,878 --> 05:14:54,948 get more data but data obviously matters 224 05:14:54,958 --> 05:14:56,628 a lot more than compute and people keep 225 05:14:56,638 --> 05:14:58,070 like rediscovering this and figuring 226 05:14:58,080 --> 05:15:00,550 this out. It's extra true for making 227 05:15:00,560 --> 05:15:04,310 LLMs useful. um before RHF even existed 228 05:15:04,320 --> 05:15:06,788 the data for RHF didn't exist and you 229 05:15:06,798 --> 05:15:09,270 could like it was night and day I'll 230 05:15:09,280 --> 05:15:10,310 just show you the next slide this is the 231 05:15:10,320 --> 05:15:11,910 probably the relevant part this is what 232 05:15:11,920 --> 05:15:14,642 this was the work in the RHF paper and 233 05:15:14,652 --> 05:15:16,390 [snorts] the high level was that even 234 05:15:16,400 --> 05:15:18,390 with the dumbest algorithms in the world 235 05:15:18,400 --> 05:15:21,270 even with a GPT2 sized model you got 236 05:15:21,280 --> 05:15:23,670 something like GPT9 level performance 237 05:15:23,680 --> 05:15:26,230 like real GPT9 not today's like fake GPT 238 05:15:26,240 --> 05:15:28,150 numbers like the numbers used to mean 239 05:15:28,160 --> 05:15:29,990 like a 100x scaling and effective 240 05:15:30,000 --> 05:15:31,750 compute for pre-training and they've 241 05:15:31,760 --> 05:15:33,110 given up on that because it just can't 242 05:15:33,120 --> 05:15:35,270 scale anymore. And still even with that, 243 05:15:35,280 --> 05:15:37,830 it would take something like GPT7 to9 to 244 05:15:37,840 --> 05:15:41,430 match GPT to the model from like 2018 or 245 05:15:41,440 --> 05:15:43,430 whatever it was at the task of 246 05:15:43,440 --> 05:15:45,350 instruction following. That is kind of 247 05:15:45,360 --> 05:15:47,590 wild because you think big LMS are so 248 05:15:47,600 --> 05:15:49,910 important for this and they are, but 249 05:15:49,920 --> 05:15:51,510 that is not the most important thing 250 05:15:51,520 --> 05:15:54,150 compared to even the gigantic jump of 251 05:15:54,160 --> 05:15:56,150 doing the right task. And spoilers, this 252 05:15:56,160 --> 05:15:57,750 is what we did for Jeff, right? like we 253 05:15:57,760 --> 05:15:59,270 made this gigantic leap in the field 254 05:15:59,280 --> 05:16:01,350 that no one realized could be coming 255 05:16:01,360 --> 05:16:03,270 because we were optimizing for a task 256 05:16:03,280 --> 05:16:04,788 and very few people think about the 257 05:16:04,798 --> 05:16:09,510 right task. Um this is why Jev has the 258 05:16:09,520 --> 05:16:12,550 Jeb Nissiqua. Um that's a really smart 259 05:16:12,560 --> 05:16:16,150 bun I think. Um 260 05:16:16,160 --> 05:16:17,430 little bit of a pity laugh, but thank 261 05:16:17,440 --> 05:16:20,470 you. Um you get what you optimize for. I 262 05:16:20,480 --> 05:16:21,910 I'm looking for a better word than 263 05:16:21,920 --> 05:16:23,110 reliable. So if you guys have 264 05:16:23,120 --> 05:16:25,190 suggestions, let me know. But there's 265 05:16:25,200 --> 05:16:27,030 there's this reliability of intelligence 266 05:16:27,040 --> 05:16:29,350 that's really missing in LLMs. I often 267 05:16:29,360 --> 05:16:32,070 call it you get Einstein 95% of the 268 05:16:32,080 --> 05:16:34,708 time, but Mr. Bean 5% of the time. And 269 05:16:34,718 --> 05:16:36,390 if you can't sell which is which, how 270 05:16:36,400 --> 05:16:37,430 the are you going to automate 271 05:16:37,440 --> 05:16:39,510 anything? And instead, instead of like 272 05:16:39,520 --> 05:16:41,430 focusing on like sometimes occasional 273 05:16:41,440 --> 05:16:43,270 smarts that you can then verify, what if 274 05:16:43,280 --> 05:16:45,910 you focused on extreme extreme narrow 275 05:16:45,920 --> 05:16:48,070 focused reliability so you can get that 276 05:16:48,080 --> 05:16:49,990 smarts every time? And this doesn't show 277 05:16:50,000 --> 05:16:51,830 up on benchmarks. It is the reason why 278 05:16:51,840 --> 05:16:53,670 people love Jev. And this is also I 279 05:16:53,680 --> 05:16:54,948 realized in hindsight we obviously 280 05:16:54,958 --> 05:16:57,270 didn't prepare for a chat GPT moment for 281 05:16:57,280 --> 05:16:59,350 developers. But it happened because 282 05:16:59,360 --> 05:17:02,150 people in the AI space just overpromise 283 05:17:02,160 --> 05:17:04,708 underdel so much that having a 284 05:17:04,718 --> 05:17:06,550 model that actually don't do that was a 285 05:17:06,560 --> 05:17:08,788 surprise. And it is not the speed cost 286 05:17:08,798 --> 05:17:10,788 or even interface. This is something 287 05:17:10,798 --> 05:17:12,628 that you know my team tells me I 288 05:17:12,638 --> 05:17:13,990 shouldn't be telling people so that we 289 05:17:14,000 --> 05:17:15,430 can have a long-term advantage and I 290 05:17:15,440 --> 05:17:17,430 just care too much. Like it's reliable 291 05:17:17,440 --> 05:17:19,830 intelligence. Make sick models please. 292 05:17:19,840 --> 05:17:21,350 even if we, you know, if it doesn't 293 05:17:21,360 --> 05:17:24,708 involve our models. Um, we see that Jev 294 05:17:24,718 --> 05:17:26,470 as like a foundation for smart software. 295 05:17:26,480 --> 05:17:27,830 I'm going to skip all over this, but I 296 05:17:27,840 --> 05:17:29,750 think software is going to be so 297 05:17:29,760 --> 05:17:31,430 back. I think the narrative of SAS 298 05:17:31,440 --> 05:17:33,110 apocalypse is obviously not going to 299 05:17:33,120 --> 05:17:36,788 happen, but I think that even calling it 300 05:17:36,798 --> 05:17:38,788 a SAS Palooa is like an understatement 301 05:17:38,798 --> 05:17:40,390 and I will get into that a bit and you 302 05:17:40,400 --> 05:17:41,750 guys ideally are the perfect crowd for 303 05:17:41,760 --> 05:17:43,110 this. I'm so excited for this part 304 05:17:43,120 --> 05:17:44,868 because that's the part that's new to 305 05:17:44,878 --> 05:17:47,350 me. Um, I want to make the distinction 306 05:17:47,360 --> 05:17:49,750 between smart software and just in time 307 05:17:49,760 --> 05:17:52,628 software. The the crazy part is AI is so 308 05:17:52,638 --> 05:17:55,910 transformative and yet somehow despite 309 05:17:55,920 --> 05:17:58,150 AI being so awesome, software has 310 05:17:58,160 --> 05:18:00,390 essentially unchanged in the last decade 311 05:18:00,400 --> 05:18:02,470 except being cheaper to write. Isn't 312 05:18:02,480 --> 05:18:04,708 that weird? Isn't the it weird that the 313 05:18:04,718 --> 05:18:06,550 best we can do in software is having a 314 05:18:06,560 --> 05:18:08,628 chatbot to the side that sometimes 315 05:18:08,638 --> 05:18:10,230 interacts with a software but cannot 316 05:18:10,240 --> 05:18:12,230 take expensive decisions for you. That 317 05:18:12,240 --> 05:18:14,310 is kind of like a a weird thing if AI 318 05:18:14,320 --> 05:18:15,990 was really that transformative. And I 319 05:18:16,000 --> 05:18:17,590 think the problem is we're focusing too 320 05:18:17,600 --> 05:18:19,990 much on coding agents and not enough 321 05:18:20,000 --> 05:18:21,590 about putting the intelligence into the 322 05:18:21,600 --> 05:18:22,948 inner loop, which I'm going to get more 323 05:18:22,958 --> 05:18:25,430 into a bit. And people are jeving. 324 05:18:25,440 --> 05:18:27,350 There's been tons of successes. We don't 325 05:18:27,360 --> 05:18:28,948 need to flex this. There's so many cool 326 05:18:28,958 --> 05:18:32,550 things, but my AIE secret is going to be 327 05:18:32,560 --> 05:18:34,788 that the field isn't jeving hard enough. 328 05:18:34,798 --> 05:18:36,310 And this is what Swix said that I can 329 05:18:36,320 --> 05:18:37,670 make a call to action for you guys 330 05:18:37,680 --> 05:18:38,788 that's unique. And hopefully I have 331 05:18:38,798 --> 05:18:40,868 enough time for this. I think the world 332 05:18:40,878 --> 05:18:43,190 still has LLM Stockholm syndrome. We are 333 05:18:43,200 --> 05:18:45,350 still thinking from a really LLM pill 334 05:18:45,360 --> 05:18:48,470 place which is I think tragic you know 335 05:18:48,480 --> 05:18:50,868 like I I I love the software saying the 336 05:18:50,878 --> 05:18:52,708 tools you use change the way you think 337 05:18:52,718 --> 05:18:54,310 and I think that it's changed the way we 338 05:18:54,320 --> 05:18:56,708 think way too much and we're not really 339 05:18:56,718 --> 05:18:58,788 rethinking software like I think the 340 05:18:58,798 --> 05:19:00,150 field needs. I think that we could just 341 05:19:00,160 --> 05:19:02,070 do be doing so much more. Hopefully I 342 05:19:02,080 --> 05:19:03,830 can try to do my best for my remaining 343 05:19:03,840 --> 05:19:06,470 time to convince you guys of that. Um so 344 05:19:06,480 --> 05:19:10,070 let's Jev to the future. Um 345 05:19:10,080 --> 05:19:13,030 I an analogy I love bear with me is 346 05:19:13,040 --> 05:19:16,070 about LEDs. Before there was the blue 347 05:19:16,080 --> 05:19:18,070 LED there was red and green LEDs. These 348 05:19:18,080 --> 05:19:20,150 were really limited in capability and 349 05:19:20,160 --> 05:19:21,350 they were used for like simple 350 05:19:21,360 --> 05:19:23,270 indicators and you know like clocks and 351 05:19:23,280 --> 05:19:25,990 stuff like that. And when the blue LED 352 05:19:26,000 --> 05:19:27,990 was made discovered I actually don't 353 05:19:28,000 --> 05:19:29,350 really know how things outside of 354 05:19:29,360 --> 05:19:31,510 computer science work. Um, but when it 355 05:19:31,520 --> 05:19:34,868 existed, I guess, um, it enabled not 356 05:19:34,878 --> 05:19:37,190 just like 50% more use cases, which what 357 05:19:37,200 --> 05:19:38,628 you'd expect about like changing colors 358 05:19:38,638 --> 05:19:40,788 from two to three, it enabled so many 359 05:19:40,798 --> 05:19:42,708 more. Like now LEDs could be used for 360 05:19:42,718 --> 05:19:43,990 white light. It could be used for like 361 05:19:44,000 --> 05:19:45,750 the displays that we all use right now. 362 05:19:45,760 --> 05:19:48,070 And it created a revolution in lighting. 363 05:19:48,080 --> 05:19:50,628 And it's it's it's unfathomable how much 364 05:19:50,638 --> 05:19:53,670 bigger the capabilities of LEDs were. 365 05:19:53,680 --> 05:19:55,430 Just ex, you know, adding a new 366 05:19:55,440 --> 05:19:57,510 primitive that composes together with a 367 05:19:57,520 --> 05:19:59,030 old primitive. So you can see where I'm 368 05:19:59,040 --> 05:20:01,430 going here. maybe. And I think it's a 369 05:20:01,440 --> 05:20:04,230 similar story for software. Software is 370 05:20:04,240 --> 05:20:06,390 so incredible. Ignore AI for the 371 05:20:06,400 --> 05:20:07,750 last, you know, half decade. But 372 05:20:07,760 --> 05:20:10,550 software has done so many insanely cool 373 05:20:10,560 --> 05:20:12,628 things. We've done a lot of stuff, you 374 05:20:12,638 --> 05:20:16,150 know, like do I love using SAS? Not 375 05:20:16,160 --> 05:20:18,470 really. But does it solve problems? Abso 376 05:20:18,480 --> 05:20:20,550 lutely. And it's incredible how 377 05:20:20,560 --> 05:20:22,390 much we've done with three fundamental 378 05:20:22,400 --> 05:20:24,230 logic gates. You know, like we are doing 379 05:20:24,240 --> 05:20:26,550 this with like simple binary operations 380 05:20:26,560 --> 05:20:28,230 that can do things like greater than, 381 05:20:28,240 --> 05:20:30,708 less than, equal. It's kind of nuts how 382 05:20:30,718 --> 05:20:32,868 much software has been built from such 383 05:20:32,878 --> 05:20:36,230 simple primitives. And my claim is that 384 05:20:36,240 --> 05:20:38,150 what we are doing with AI now when we're 385 05:20:38,160 --> 05:20:40,708 making a intelligence machine native 386 05:20:40,718 --> 05:20:42,948 this is kind of conceptually adding a 387 05:20:42,958 --> 05:20:45,670 fourth semantic logic gate. What if we 388 05:20:45,680 --> 05:20:47,750 can program with common sense with 389 05:20:47,760 --> 05:20:49,990 intention with human understanding? Like 390 05:20:50,000 --> 05:20:51,990 do we still need reexes with like crazy 391 05:20:52,000 --> 05:20:53,590 things or can we just say like we want a 392 05:20:53,600 --> 05:20:55,350 Gmail email address or a real looking 393 05:20:55,360 --> 05:20:57,190 one? There's so much that we can do that 394 05:20:57,200 --> 05:20:58,550 we're going to need to push the f, you 395 05:20:58,560 --> 05:21:00,868 know, push the frontiers of cheap, 396 05:21:00,878 --> 05:21:03,110 smart, fast in order to get done. But I 397 05:21:03,120 --> 05:21:04,310 think that there's going to be like so 398 05:21:04,320 --> 05:21:06,070 many things here. And I think just like 399 05:21:06,080 --> 05:21:08,070 the blue LED, it's going to be way 400 05:21:08,080 --> 05:21:10,070 bigger than just increasing the 401 05:21:10,080 --> 05:21:12,150 representational ability of software by 402 05:21:12,160 --> 05:21:14,550 33%. Because it's adding effectively 403 05:21:14,560 --> 05:21:17,030 another kind of sort of logic gate. And 404 05:21:17,040 --> 05:21:18,868 this sci-fi is the stuff that I think 405 05:21:18,878 --> 05:21:20,868 will be impossible to predict. Just like 406 05:21:20,878 --> 05:21:22,628 who could predict how powerful the 407 05:21:22,638 --> 05:21:24,070 internet would have been because it 408 05:21:24,080 --> 05:21:25,830 requires it's like much more grassroots 409 05:21:25,840 --> 05:21:27,510 and decentralized and it requires people 410 05:21:27,520 --> 05:21:30,948 to be empowered to do insane stuff. Um I 411 05:21:30,958 --> 05:21:34,628 call this software research. Um that's 412 05:21:34,638 --> 05:21:37,430 not a joke. Um but uh I I think that 413 05:21:37,440 --> 05:21:39,510 there's going to be like an explosion of 414 05:21:39,520 --> 05:21:42,230 what software can do and we will want 415 05:21:42,240 --> 05:21:44,150 people to be exploring those boundaries 416 05:21:44,160 --> 05:21:46,230 of software. Not necessarily because 417 05:21:46,240 --> 05:21:48,310 it's valuable, but because they can. 418 05:21:48,320 --> 05:21:50,310 Because we have like new frontiers to 419 05:21:50,320 --> 05:21:51,830 explore and this is what the early 420 05:21:51,840 --> 05:21:53,590 internet was like. Should all of those 421 05:21:53,600 --> 05:21:56,470 websites existed? Probably not. But like 422 05:21:56,480 --> 05:21:58,628 it was very cool and we found a lot of 423 05:21:58,638 --> 05:21:59,990 awesome stuff from that. And I think we 424 05:22:00,000 --> 05:22:01,430 we, you know, we have another 425 05:22:01,440 --> 05:22:03,030 opportunity for that. So we should not 426 05:22:03,040 --> 05:22:05,030 just keep living in this little, you 427 05:22:05,040 --> 05:22:07,030 know, fish tank that we are the victims 428 05:22:07,040 --> 05:22:09,270 of open anthropic on whatever they ship 429 05:22:09,280 --> 05:22:11,190 and we just have to be like LM rappers. 430 05:22:11,200 --> 05:22:13,590 So I'm going to pitch software research 431 05:22:13,600 --> 05:22:15,590 as a direction here. The theme is going 432 05:22:15,600 --> 05:22:18,390 to be real software engineering, not 433 05:22:18,400 --> 05:22:20,390 just vibe coding. There's it's going to 434 05:22:20,400 --> 05:22:21,670 be like the kind of stuff that everyone 435 05:22:21,680 --> 05:22:23,030 has known if you've been a software 436 05:22:23,040 --> 05:22:25,430 engineer. Decomposition, breaking tasks 437 05:22:25,440 --> 05:22:27,350 down. We need to think about stages, 438 05:22:27,360 --> 05:22:29,190 ideally ones that can be verified so we 439 05:22:29,200 --> 05:22:31,670 have real encapsulation. And we need to 440 05:22:31,680 --> 05:22:33,670 actually do state management instead of 441 05:22:33,680 --> 05:22:35,350 assuming everything is global state 442 05:22:35,360 --> 05:22:37,190 within a KV cache. And the theme is just 443 05:22:37,200 --> 05:22:38,788 going to be like real 444 05:22:38,798 --> 05:22:40,868 engineering by people who care a lot 445 05:22:40,878 --> 05:22:42,788 about the end result. It doesn't have to 446 05:22:42,798 --> 05:22:44,948 be valuable. Ideally, it's sick as 447 05:22:44,958 --> 05:22:46,550 Um, and you know, we should be pushing 448 05:22:46,560 --> 05:22:49,110 the limits. I'm going to go into a 449 05:22:49,120 --> 05:22:51,190 couple of like my favorite ideas and I 450 05:22:51,200 --> 05:22:53,190 want to figure out how to encourage 451 05:22:53,200 --> 05:22:55,190 these in the world. So, this is my first 452 05:22:55,200 --> 05:22:56,708 step at this. I'm just going to talk 453 05:22:56,718 --> 05:22:58,550 about it. We have raised a bunch of 454 05:22:58,560 --> 05:22:59,590 money. So, I would like to figure out 455 05:22:59,600 --> 05:23:00,948 how to fund this, but we don't have the 456 05:23:00,958 --> 05:23:02,868 operations help to do that. But here are 457 05:23:02,878 --> 05:23:05,670 just some ideas out there. Um, I think 458 05:23:05,680 --> 05:23:09,510 agents are they're very cool, but like 459 05:23:09,520 --> 05:23:11,590 they're they're so inflexible in their 460 05:23:11,600 --> 05:23:13,590 design because of the tyranny of the KV 461 05:23:13,600 --> 05:23:15,270 cache. I don't have the time for this if 462 05:23:15,280 --> 05:23:17,030 this timer is to be believed, but 463 05:23:17,040 --> 05:23:18,868 believe me that there's so much that 464 05:23:18,878 --> 05:23:20,628 people have tried to do but failed at 465 05:23:20,638 --> 05:23:22,390 because it ends up becoming inefficient 466 05:23:22,400 --> 05:23:24,390 because it fights against the KV cache. 467 05:23:24,400 --> 05:23:26,230 But I think there's a lot more to do. 468 05:23:26,240 --> 05:23:28,390 Why is memory o of nan, right? Like why 469 05:23:28,400 --> 05:23:30,390 can't we have all of our history of 470 05:23:30,400 --> 05:23:32,550 coding agents always included in every 471 05:23:32,560 --> 05:23:35,590 new call? That is how continual learning 472 05:23:35,600 --> 05:23:37,670 works, right? And we have the memory 473 05:23:37,680 --> 05:23:39,030 available. It's just that we don't have 474 05:23:39,040 --> 05:23:40,948 the smarts to map produce over the 475 05:23:40,958 --> 05:23:43,110 memory to find out like what you should 476 05:23:43,120 --> 05:23:44,310 what you should remember, what should 477 05:23:44,320 --> 05:23:46,390 you shouldn't. Um I think there's many 478 05:23:46,400 --> 05:23:48,788 ways to do login or even better lookups. 479 05:23:48,798 --> 05:23:50,550 Why when you're making many agents at 480 05:23:50,560 --> 05:23:52,470 once, why can't they not share state? 481 05:23:52,480 --> 05:23:54,230 Why do they need to rediscover the same 482 05:23:54,240 --> 05:23:55,990 patterns of the codebase over and over 483 05:23:56,000 --> 05:23:58,310 and over again? And why can't non-naive 484 05:23:58,320 --> 05:23:59,830 routing and sub agents work? Like even 485 05:23:59,840 --> 05:24:01,670 naive routing doesn't work, but like why 486 05:24:01,680 --> 05:24:03,190 can't we do these things? And why does 487 05:24:03,200 --> 05:24:04,708 like the you know the big labs keep 488 05:24:04,718 --> 05:24:05,910 going back and forth on whether or not 489 05:24:05,920 --> 05:24:08,868 sub agents work? Um I think that like 490 05:24:08,878 --> 05:24:10,708 from an engineering perspective there's 491 05:24:10,718 --> 05:24:12,230 elegance to it but they fight against 492 05:24:12,240 --> 05:24:13,990 the KV cache and I think that if we 493 05:24:14,000 --> 05:24:16,150 design agents around that we could do so 494 05:24:16,160 --> 05:24:19,110 much more. This one's the the probably 495 05:24:19,120 --> 05:24:20,868 the most vague of my directions but I 496 05:24:20,878 --> 05:24:23,350 feel very strongly about it. Um the 497 05:24:23,360 --> 05:24:24,868 world is full of principal agent 498 05:24:24,878 --> 05:24:26,550 problems. This is when you have like you 499 05:24:26,560 --> 05:24:28,150 know ulterior motives between this is 500 05:24:28,160 --> 05:24:30,070 not a coding agent but like any agent. 501 05:24:30,080 --> 05:24:32,150 The the classic example is citizens 502 05:24:32,160 --> 05:24:34,230 versus elected officials. I don't want 503 05:24:34,240 --> 05:24:36,230 to make a too political claim here, but 504 05:24:36,240 --> 05:24:38,550 like there's misalignment in all levels 505 05:24:38,560 --> 05:24:40,628 of society. And this results in like 506 05:24:40,638 --> 05:24:43,990 waste, fraud, bad things, and it it 507 05:24:44,000 --> 05:24:45,910 results in for a society that wants 508 05:24:45,920 --> 05:24:47,990 fairness, you need to overemphasize the 509 05:24:48,000 --> 05:24:49,670 letter over the spirit, which then 510 05:24:49,680 --> 05:24:52,390 results in loopholes and play uh pay to- 511 05:24:52,400 --> 05:24:54,390 win dynamics in the world. And I think 512 05:24:54,400 --> 05:24:55,750 the root of that problem is we can't 513 05:24:55,760 --> 05:24:57,590 encode the spirit. Just like with 514 05:24:57,600 --> 05:24:59,750 software, we're constrained to encoding 515 05:24:59,760 --> 05:25:01,510 bits instead of what we actually meant 516 05:25:01,520 --> 05:25:04,150 in that situation. And I think so many 517 05:25:04,160 --> 05:25:05,670 organization, society could be 518 05:25:05,680 --> 05:25:07,190 transformed by really encoding the 519 05:25:07,200 --> 05:25:09,830 spirit all over the place. Obviously, 520 05:25:09,840 --> 05:25:12,550 real-time intelligence security, can we 521 05:25:12,560 --> 05:25:14,628 prevent the OpenAI hugging face sandbox, 522 05:25:14,638 --> 05:25:17,190 blah blah blah. Um, why did why couldn't 523 05:25:17,200 --> 05:25:19,190 they prevent it? It is not because they 524 05:25:19,200 --> 05:25:20,550 didn't have the intelligence. I would 525 05:25:20,560 --> 05:25:22,070 argue it's a lot easier to figure out 526 05:25:22,080 --> 05:25:23,590 they're doing a naughty thing than to 527 05:25:23,600 --> 05:25:25,350 actually do the naughty thing. It is 528 05:25:25,360 --> 05:25:27,270 because it was not fast, cheap, and 529 05:25:27,280 --> 05:25:29,670 reliable enough to actually run real 530 05:25:29,680 --> 05:25:32,150 time. And what happens when every 531 05:25:32,160 --> 05:25:34,150 corporation has their own real-time 532 05:25:34,160 --> 05:25:36,150 security that's actually smart? How cool 533 05:25:36,160 --> 05:25:38,470 would that be? I can't talk 534 05:25:38,480 --> 05:25:40,310 about generative UIs because people are 535 05:25:40,320 --> 05:25:41,830 going to be talking about it afterwards. 536 05:25:41,840 --> 05:25:43,190 I don't want to steal their thunder, but 537 05:25:43,200 --> 05:25:46,310 it's cool as Trust me. Um, 538 05:25:46,320 --> 05:25:48,948 like semantic dependent typing. Why 539 05:25:48,958 --> 05:25:51,990 can't we have like a type system that 540 05:25:52,000 --> 05:25:54,230 actually is semantic, right? like why 541 05:25:54,240 --> 05:25:57,190 can't we like declare what local 542 05:25:57,200 --> 05:25:59,030 properties we want so we don't have to 543 05:25:59,040 --> 05:26:01,350 read a chain of dependencies deep down 544 05:26:01,360 --> 05:26:03,350 these all should be possible and imagine 545 05:26:03,360 --> 05:26:05,750 if you could like run a jeike model in 546 05:26:05,760 --> 05:26:07,350 your lint checker to check if you're 547 05:26:07,360 --> 05:26:09,670 it's all type safe and it could you know 548 05:26:09,680 --> 05:26:11,910 uphold semantic properties it would be 549 05:26:11,920 --> 05:26:15,110 so cool there's there was a 550 05:26:15,120 --> 05:26:17,750 great Twitter video on um like a voice 551 05:26:17,760 --> 05:26:19,830 interface in general I think new 552 05:26:19,840 --> 05:26:22,310 interfaces are so exciting this is to me 553 05:26:22,320 --> 05:26:23,670 the return of like this is going to show 554 05:26:23,680 --> 05:26:25,670 by agent software like do what I mean. 555 05:26:25,680 --> 05:26:27,350 This used to be like a popular thing but 556 05:26:27,360 --> 05:26:29,430 it never really worked because you it 557 05:26:29,440 --> 05:26:31,190 was it was too many hard-coded rules to 558 05:26:31,200 --> 05:26:33,750 really figure out what I mean. Um 559 05:26:33,760 --> 05:26:36,150 there's stopping slop obviously a big 560 05:26:36,160 --> 05:26:38,070 theme of here but like we definitely 561 05:26:38,080 --> 05:26:39,510 believe this. This is my props. I have 562 05:26:39,520 --> 05:26:42,550 to we have a stop geni baseball bat for 563 05:26:42,560 --> 05:26:45,750 type safe. It is very very awesome. I 564 05:26:45,760 --> 05:26:47,910 want to talk about the root of slop. Um 565 05:26:47,920 --> 05:26:49,670 the the root of slop is that AI is 566 05:26:49,680 --> 05:26:52,390 unreliable. Therefore, you as a company 567 05:26:52,400 --> 05:26:54,070 or you as you know like someone who's 568 05:26:54,080 --> 05:26:55,590 doing stuff, it can't be used for 569 05:26:55,600 --> 05:26:58,070 decisions that costs you stuff. But it's 570 05:26:58,080 --> 05:26:59,670 actually totally fine to push the costs 571 05:26:59,680 --> 05:27:01,350 into other people, which is horrible. 572 05:27:01,360 --> 05:27:02,948 You know, externalized costs and then 573 05:27:02,958 --> 05:27:05,030 this is what ends up becoming stop. You 574 05:27:05,040 --> 05:27:06,470 know, if people actually read the stuff 575 05:27:06,480 --> 05:27:08,150 and filter it and like care about the 576 05:27:08,160 --> 05:27:10,550 quality, it becomes not slop, but now 577 05:27:10,560 --> 05:27:12,070 the world is full of this because 578 05:27:12,080 --> 05:27:13,830 there's an asymmetric advantage for 579 05:27:13,840 --> 05:27:15,510 people who are like bad actors and don't 580 05:27:15,520 --> 05:27:17,510 give a So, I think that we can 581 05:27:17,520 --> 05:27:19,270 save the world from at least inbound 582 05:27:19,280 --> 05:27:20,948 hell among many other things and that 583 05:27:20,958 --> 05:27:23,910 would be very cool. Um, and even 584 05:27:23,920 --> 05:27:25,830 something I think is super sick is 585 05:27:25,840 --> 05:27:28,470 cursed software. Like why if you guys 586 05:27:28,480 --> 05:27:29,990 have ever seen the subreddit bad UI 587 05:27:30,000 --> 05:27:32,628 battles, it is so cool. Even 588 05:27:32,638 --> 05:27:34,470 though they make the horrible interfaces 589 05:27:34,480 --> 05:27:35,830 like this whole game of like would you 590 05:27:35,840 --> 05:27:37,510 like to delete your account becomes you 591 05:27:37,520 --> 05:27:40,310 know a random thing. And and I actually 592 05:27:40,320 --> 05:27:42,390 think doing things because we can is 593 05:27:42,400 --> 05:27:44,230 what software engineering is about 594 05:27:44,240 --> 05:27:46,628 because who knows maybe 1% of these 595 05:27:46,638 --> 05:27:48,948 ideas actually lead to something super 596 05:27:48,958 --> 05:27:50,868 awesome. And I think like really 597 05:27:50,878 --> 05:27:53,030 cultivating even bad ideas is kind of 598 05:27:53,040 --> 05:27:54,788 what being like this you know playful 599 05:27:54,798 --> 05:27:58,628 hacker is all about and generally just 600 05:27:58,638 --> 05:28:01,030 automating everything. This is what the 601 05:28:01,040 --> 05:28:02,868 followup this is this is a slop image 602 05:28:02,878 --> 05:28:04,548 but uh sorry logo is a little bit wrong 603 05:28:04,558 --> 05:28:05,990 but don't stare at that. That's a little 604 05:28:06,000 --> 05:28:08,788 weird. Um, this is what a lot of the 605 05:28:08,798 --> 05:28:10,628 follow-up talks are going to be about. 606 05:28:10,638 --> 05:28:13,350 Thank you. Um, and and if you're a 607 05:28:13,360 --> 05:28:15,830 software engineer, you almost certainly 608 05:28:15,840 --> 05:28:17,190 think in the world of bits. If you know 609 05:28:17,200 --> 05:28:19,430 anything about software, uh, and you've 610 05:28:19,440 --> 05:28:21,910 interacted with software, you know that 611 05:28:21,920 --> 05:28:23,910 there are things software can do and 612 05:28:23,920 --> 05:28:25,910 software cannot do because that is just 613 05:28:25,920 --> 05:28:28,390 only obvious, right? Like there's some 614 05:28:28,400 --> 05:28:30,070 things that you know you need to do 615 05:28:30,080 --> 05:28:31,590 because that is not compatible with 616 05:28:31,600 --> 05:28:33,910 software. But what if those constraints 617 05:28:33,920 --> 05:28:35,750 were no longer true? Maybe not today, 618 05:28:35,760 --> 05:28:37,830 but maybe in like six months or a year. 619 05:28:37,840 --> 05:28:39,830 What couldn't we automate? And that's 620 05:28:39,840 --> 05:28:41,190 kind of where I want to leave this whole 621 05:28:41,200 --> 05:28:43,510 thing because I think there is very 622 05:28:43,520 --> 05:28:45,590 likely to be a future where it's not a 623 05:28:45,600 --> 05:28:46,868 question of whether or not we can 624 05:28:46,878 --> 05:28:48,310 automate something. It's going to be a 625 05:28:48,320 --> 05:28:49,670 question of like how much work is it, 626 05:28:49,680 --> 05:28:51,110 which is how it is right now for the 627 05:28:51,120 --> 05:28:52,788 stuff that can be automated. And that 628 05:28:52,798 --> 05:28:55,510 would be cool as So, hell yeah. 629 05:28:55,520 --> 05:28:57,986 Thank you. Um, 630 05:28:57,996 --> 05:28:59,030 [applause] 631 05:28:59,040 --> 05:29:02,310 I'm super overtime, but thank you to the 632 05:29:02,320 --> 05:29:04,230 early believers. You know, we had no 633 05:29:04,240 --> 05:29:05,910 idea so many people would resonate with 634 05:29:05,920 --> 05:29:07,510 this. We thought we were the insane 635 05:29:07,520 --> 05:29:09,190 ones, but it turns out like we were 636 05:29:09,200 --> 05:29:10,708 actually the majority of the world who 637 05:29:10,718 --> 05:29:13,430 realized AI could be so much more. Thank 638 05:29:13,440 --> 05:29:14,708 you to like the people who are like 639 05:29:14,718 --> 05:29:16,628 really realistic about reliability. 640 05:29:16,638 --> 05:29:18,390 Everyone knows I hate benchmarks and a 641 05:29:18,400 --> 05:29:19,750 lot of other people are really screaming 642 05:29:19,760 --> 05:29:21,510 out like benchmarks suck. 643 05:29:21,520 --> 05:29:23,830 They're actually they hurt the people 644 05:29:23,840 --> 05:29:25,430 who do not have time for anything other 645 05:29:25,440 --> 05:29:27,430 than benchmarks most. It's tragic. And 646 05:29:27,440 --> 05:29:28,708 for the people who are pushing the 647 05:29:28,718 --> 05:29:31,670 limits of software, like yes. And 648 05:29:31,680 --> 05:29:33,430 we want to be there with you helping you 649 05:29:33,440 --> 05:29:35,430 all do that. The the dreamers and the 650 05:29:35,440 --> 05:29:37,750 people who don't want to be limited by 651 05:29:37,760 --> 05:29:40,548 everything that has happened so far. We 652 05:29:40,558 --> 05:29:42,470 want to really fight for you and we want 653 05:29:42,480 --> 05:29:45,510 this to be as reliable as a fourth logic 654 05:29:45,520 --> 05:29:48,070 gate and that is a very high bar and we 655 05:29:48,080 --> 05:29:50,230 have like a lot of work to do. So thank 656 05:29:50,240 --> 05:29:53,000 you. Thank you.