1 05:19:18,000 --> 05:19:29,000 My name is Jared. I'm the creator of Bun, and I work at Anthropic. Bun is the JavaScript runtime that powers cloud code. 2 05:19:30,240 --> 05:19:39,640 And recently I merged a several million line of code PR to add an ahead-of-time compiler to Bun, 3 05:19:39,640 --> 05:19:46,600 which is basically the idea is instead of paying the cost of a JIT at runtime, 4 05:19:47,000 --> 05:19:51,760 you can compile all the code ahead of time. 5 05:19:55,180 --> 05:19:57,620 And it's like a really big project. 6 05:19:58,940 --> 05:19:59,800 Over the summer, 7 05:20:00,100 --> 05:20:02,700 I rewrote Bun & Rust. 8 05:20:05,040 --> 05:20:07,420 And this is a project on a larger scale. 9 05:20:08,100 --> 05:20:10,720 This ahead-of-time compiler is like a larger project than that. 10 05:20:13,600 --> 05:20:17,760 And I think one of the challenges right now with large projects is like, 11 05:20:17,820 --> 05:20:22,800 how do you actually confidently ship millions of lines of code written by agents? 12 05:20:25,000 --> 05:20:26,040 So that's what I'm going to talk about. 13 05:20:27,580 --> 05:20:30,540 This was like the opening prompt after some discussion. 14 05:20:31,220 --> 05:20:39,680 It was like, basically, we need to make Cloud Code use a lot less memory and faster. 15 05:20:39,900 --> 05:20:42,960 And we can modify either WebKit or Bun. 16 05:20:43,920 --> 05:20:47,900 Bun uses JavaScript core, which is WebKit, under the hood. 17 05:20:49,340 --> 05:20:51,180 And it's been a little bit over a month. 18 05:20:55,980 --> 05:21:03,680 this is this compiler has gone through five rewrites you can see the like the first couple 19 05:21:03,680 --> 05:21:10,440 ones were like pretty short and then I like almost shipped the blue one and then I was like 20 05:21:10,440 --> 05:21:16,100 this is building way too long like it's taking way too long to compile the code so rewrote it again 21 05:21:17,240 --> 05:21:19,240 and this next one is shipping next week. 22 05:21:24,800 --> 05:21:28,360 This is kind of the way I think about a project like this 23 05:21:28,360 --> 05:21:30,460 is like you're really constrained 24 05:21:30,460 --> 05:21:33,060 by how you test that the code works. 25 05:21:34,300 --> 05:21:36,160 So that's like even more important 26 05:21:36,160 --> 05:21:37,220 than what you're specifically, 27 05:21:37,600 --> 05:21:39,500 like the approach you go with for how to build it 28 05:21:39,500 --> 05:21:41,680 is entirely driven by how do you verify 29 05:21:41,680 --> 05:21:43,340 that it actually is going to like work 30 05:21:43,340 --> 05:21:45,880 and not be buggy and that you can trust it. 31 05:21:48,100 --> 05:21:50,420 And so just a little more background on like, 32 05:21:51,680 --> 05:21:53,920 like why this particular project, 33 05:21:54,060 --> 05:21:55,080 why the ahead of time compiler, 34 05:21:55,220 --> 05:21:59,820 and it's because for a typical JavaScript based application, 35 05:22:00,920 --> 05:22:03,400 roughly half of like the CPU time 36 05:22:03,400 --> 05:22:05,320 is just spent compiling code on demand. 37 05:22:07,040 --> 05:22:10,960 And so a lot of the general like thing you've seen 38 05:22:10,960 --> 05:22:13,060 in a lot of applications where people are rewriting 39 05:22:13,060 --> 05:22:14,840 their code bases in a language like Rust, 40 05:22:14,840 --> 05:22:15,860 or sometimes go. 41 05:22:17,940 --> 05:22:19,700 It's like when coming from JavaScript, 42 05:22:20,020 --> 05:22:25,640 it's because it's really the combination of the JIT 43 05:22:25,640 --> 05:22:28,440 and that it's a single-threaded application. 44 05:22:31,240 --> 05:22:33,620 So that's sort of the difference between a JIT and 45 05:22:34,360 --> 05:22:37,440 is that you compile before, 46 05:22:37,640 --> 05:22:40,080 whereas with a JIT you compile as you go along. 47 05:22:41,900 --> 05:22:46,940 This is particularly important for something like Cloud Code when using the Cloud Code 48 05:22:46,940 --> 05:22:55,600 like iOS app, where the CPU time is a lot if you have to have every session of Cloud 49 05:22:55,600 --> 05:22:56,740 Code also compiling. 50 05:23:01,020 --> 05:23:06,260 So we talked a lot about, should we rewrite Cloud Code in Rust? 51 05:23:07,620 --> 05:23:11,720 because the Bun team, us, rewrote Bun and Rust. 52 05:23:12,000 --> 05:23:14,380 So we're pretty good at rewrites. 53 05:23:17,640 --> 05:23:19,240 And we decided not to. 54 05:23:21,660 --> 05:23:25,380 And the reason why is because in Bun's case, 55 05:23:25,500 --> 05:23:26,860 it's actually pretty easy to do. 56 05:23:27,500 --> 05:23:29,420 Bun is an easy product to do a Rust rewrite 57 05:23:29,420 --> 05:23:32,320 because the tests were written in a language 58 05:23:32,320 --> 05:23:34,500 that's independent of the one it runs in. 59 05:23:34,500 --> 05:23:36,900 So because all the tests were written in TypeScript, 60 05:23:38,460 --> 05:23:40,860 we could run the tests on both the Zig version 61 05:23:40,860 --> 05:23:44,200 and the Rust version, whereas for Cloud Code, 62 05:23:46,180 --> 05:23:48,540 we, like Cloud Code's tests are written in TypeScript. 63 05:23:48,800 --> 05:23:51,820 So we would have to port Cloud Code's test suite. 64 05:23:53,520 --> 05:23:56,080 And as soon as you're like rewriting the tests, 65 05:23:56,340 --> 05:23:58,400 then you can't really trust the tests anymore. 66 05:24:00,400 --> 05:24:02,120 So that seemed like it'd be too risky. 67 05:24:03,300 --> 05:24:09,180 So instead, what if we just make JavaScript faster so we don't have to rewrite it in Rust? 68 05:24:12,140 --> 05:24:14,280 This was another prompt a little bit later. 69 05:24:18,100 --> 05:24:26,640 And the result is that it actually did have JavaScript's memory, like the memory usage. 70 05:24:27,640 --> 05:24:29,660 It uses about 4x less memory. 71 05:24:31,500 --> 05:24:33,600 it starts about two times faster 72 05:24:37,840 --> 05:24:40,360 and we're dogfooding this internally 73 05:24:40,360 --> 05:24:42,520 and it'll be external soon 74 05:24:45,020 --> 05:24:46,160 some more benchmarks 75 05:24:46,160 --> 05:24:48,060 and then I'll also go into more detail 76 05:24:48,060 --> 05:24:49,420 about how we actually did this 77 05:24:50,980 --> 05:24:54,120 so like it's possible that 78 05:24:54,820 --> 05:24:56,480 an interesting thing is like 79 05:24:56,480 --> 05:25:00,460 like TypeScript 7 rewrote their code by hand 80 05:25:00,460 --> 05:25:04,980 from JavaScript slash TypeScript to Go, 81 05:25:05,160 --> 05:25:08,880 and the single-threaded version, 82 05:25:10,440 --> 05:25:12,320 when compiled via Bun compile, 83 05:25:12,440 --> 05:25:16,420 is now faster than the single-threaded TypeScript Go port. 84 05:25:19,040 --> 05:25:21,060 And we will add threads also 85 05:25:21,060 --> 05:25:23,700 to the ahead-of-time compiled version of Bun, 86 05:25:23,860 --> 05:25:25,640 but we haven't done that yet. 87 05:25:26,480 --> 05:25:27,380 so there is a PR. 88 05:25:29,540 --> 05:25:33,440 And so a little bit about the actual ahead of time compiler. 89 05:25:35,600 --> 05:25:40,100 TypeScript is not, types don't necessarily need to be correct, 90 05:25:40,360 --> 05:25:42,260 so it treats types as predictions 91 05:25:43,080 --> 05:25:45,400 instead of trusting them. 92 05:25:48,240 --> 05:25:51,140 And this kind of problem works really well for models 93 05:25:51,140 --> 05:25:55,360 because they're just really good at writing compilers. 94 05:25:59,800 --> 05:26:03,220 So this is sort of like more about like how, 95 05:26:05,100 --> 05:26:06,300 how like should we, 96 05:26:06,760 --> 05:26:08,780 like how we were thinking about this decision. 97 05:26:09,860 --> 05:26:11,520 And a big thing here is like, 98 05:26:11,620 --> 05:26:14,340 there's just so many tests you can run when you're, 99 05:26:14,400 --> 05:26:16,740 when it's like in ahead of time, 100 05:26:16,920 --> 05:26:17,900 like for something like this, 101 05:26:18,000 --> 05:26:20,460 because we don't need to write any tests pretty much ourselves. 102 05:26:20,640 --> 05:26:22,000 Like we actually do write a lot of tests, 103 05:26:22,040 --> 05:26:26,360 but we can largely trust that the, 104 05:26:26,360 --> 05:26:32,960 tests that predate the project work. And this was the same thing with the Bunn Rust rewrite, where 105 05:26:32,960 --> 05:26:38,940 we had Bunn's existing test suite we could run. And now with the compiler, we have Bunn's existing 106 05:26:38,940 --> 05:26:47,280 test suite plus WebKids tests plus test262 plus CloudCodes tests. And then we also made it so that 107 05:26:47,280 --> 05:26:49,920 That way, BUN itself, BUN's own test runner 108 05:26:49,920 --> 05:26:51,680 could be run in the AOT mode. 109 05:26:57,380 --> 05:27:00,160 And we have one breaking change, 110 05:27:00,480 --> 05:27:01,780 or really it's two breaking changes 111 05:27:01,780 --> 05:27:03,560 with the at-of-time compiler, 112 05:27:03,720 --> 05:27:06,420 and that's pretty much why these particular tests 113 05:27:06,420 --> 05:27:07,220 are failing. 114 05:27:08,280 --> 05:27:12,160 But the two breaking changes are actually 115 05:27:12,160 --> 05:27:16,880 mostly security things to prevent prototype pollution 116 05:27:17,280 --> 05:27:19,960 tax, but they also end up benefiting for performance. 117 05:27:23,180 --> 05:27:30,100 So what makes this particular kind of problem pretty good for agents is you can run all 118 05:27:30,100 --> 05:27:34,280 three versions of an application, one with a JIT, one with ahead of time compiled, and 119 05:27:34,280 --> 05:27:35,440 one also without the types. 120 05:27:35,680 --> 05:27:41,700 And as long as the three of them behave the same, then you know that it works, at least 121 05:27:41,700 --> 05:27:42,960 for that specific application. 122 05:27:42,960 --> 05:27:46,000 but then you can take that kind of problem 123 05:27:46,000 --> 05:27:47,700 and fuzz it a lot. 124 05:27:50,980 --> 05:27:54,220 And just have clogged like churn out code generators 125 05:27:54,220 --> 05:27:56,120 to like fuzz different patterns. 126 05:27:58,020 --> 05:27:59,720 And you can catch a lot of bugs that way. 127 05:28:02,760 --> 05:28:04,620 This is a very large project. 128 05:28:04,800 --> 05:28:07,360 This is the biggest I think I've worked on 129 05:28:08,300 --> 05:28:10,200 in terms of like code volume. 130 05:28:11,860 --> 05:28:13,020 most of this is tests 131 05:28:14,960 --> 05:28:17,240 and also because we rewrote it five times 132 05:28:19,260 --> 05:28:21,460 when you're like doing this kind of project 133 05:28:22,840 --> 05:28:25,220 how I think about it is like you have like 134 05:28:25,220 --> 05:28:29,120 batches of like okay we need to write a lot of code at once 135 05:28:29,120 --> 05:28:32,940 and then after that initial batch then it's okay 136 05:28:32,940 --> 05:28:37,180 let's have let's like fix everything make sure it compiles make sure 137 05:28:37,180 --> 05:28:45,100 it passes a bunch of tests and make sure it like really truly works and the actual logistics of 138 05:28:45,100 --> 05:28:52,260 that are like when you're in the big batch kind of project stage then you don't want to run builds 139 05:28:52,260 --> 05:28:57,280 you want to run one build because otherwise you're going to spend all your time compiling and like 140 05:28:57,280 --> 05:29:01,080 waiting for build errors and it's going to like not be a good use of anyone's time 141 05:29:03,000 --> 05:29:08,860 and then separately for like longer running like once you have once you've 142 05:29:08,860 --> 05:29:16,160 written all the code you want to have it like have lots of individual agents. 143 05:29:18,600 --> 05:29:22,940 This is like a little bit of the timeline of this project. 144 05:29:26,900 --> 05:29:29,640 the first one was not very good 145 05:29:31,620 --> 05:29:33,360 like cloud code was a gigabyte 146 05:29:34,620 --> 05:29:35,580 which is terrible 147 05:29:39,200 --> 05:29:42,060 and like this used as much as possible 148 05:29:42,060 --> 05:29:44,040 from JavaScript core's existing code 149 05:29:45,300 --> 05:29:47,760 so it was like basically the ceiling was 150 05:29:47,760 --> 05:29:50,980 what is like only as good as like 151 05:29:50,980 --> 05:29:52,480 what it could be from that 152 05:29:55,720 --> 05:29:59,560 So then a little bit after this, we added a TypeScript type checker into Bunn itself. 153 05:30:03,360 --> 05:30:23,600 These were prompts. And so we added BunCheck and then we're like, hey, Claude, please use BunCheck. And then after that, startup time got two times faster in Claude code and started using about 3x less memory. 154 05:30:26,260 --> 05:30:27,820 But the code is slop. 155 05:30:31,680 --> 05:30:34,360 And like, this is not good, it should never, 156 05:30:35,220 --> 05:30:37,240 basically this meant that any object 157 05:30:37,240 --> 05:30:40,000 that accessed a property called archive, 158 05:30:40,380 --> 05:30:43,060 it would then assume that the bun.archive API 159 05:30:43,060 --> 05:30:44,160 was potentially used. 160 05:30:44,880 --> 05:30:46,360 And I looked at this and I was like, 161 05:30:46,420 --> 05:30:48,380 we cannot ship this, this is not good. 162 05:30:49,420 --> 05:30:50,220 So then, 163 05:30:58,300 --> 05:31:00,860 This is like, yeah, and this is another thing too, 164 05:31:00,980 --> 05:31:03,160 is like, it just like, 165 05:31:04,760 --> 05:31:06,540 like when you're doing like a large planning, 166 05:31:06,900 --> 05:31:08,240 like when you're starting to like have it 167 05:31:08,240 --> 05:31:09,600 do a bunch of design docs and stuff, 168 05:31:10,220 --> 05:31:12,640 it just loves writing markdown docs 169 05:31:12,640 --> 05:31:13,560 and it doesn't make sense. 170 05:31:14,000 --> 05:31:18,660 Cause who's gonna like 1500 markdown files 171 05:31:18,660 --> 05:31:21,580 is just not reasonable. 172 05:31:28,680 --> 05:31:33,520 So for this it was like, this was like I think the third compiler version, maybe the fourth, 173 05:31:33,980 --> 05:31:34,700 no this is the third. 174 05:31:38,960 --> 05:31:45,830 It, to like, just to kind of batch get it all out, 140 agents at the same time, write 175 05:31:45,830 --> 05:31:49,550 all the code all at the same time, then get Cloud Code to run. 176 05:31:54,890 --> 05:31:58,690 And then this is the fourth one. 177 05:31:59,010 --> 05:32:06,690 And this one, the code was not slop, pretty good, with basically the same performance 178 05:32:06,690 --> 05:32:07,210 as before. 179 05:32:07,730 --> 05:32:11,230 However, the compilation time was way too slow. 180 05:32:11,230 --> 05:32:14,970 because people coming from a JavaScript background 181 05:32:14,970 --> 05:32:16,790 don't expect any compilation time. 182 05:32:20,410 --> 05:32:22,470 So 50 seconds for... 183 05:32:22,470 --> 05:32:24,290 Cloud Code is a pretty large code base, 184 05:32:24,290 --> 05:32:27,510 but this meant that other projects 185 05:32:27,510 --> 05:32:29,130 would take single-digit seconds, 186 05:32:29,210 --> 05:32:31,010 and people don't want to wait that long. 187 05:32:35,970 --> 05:32:38,530 So we're going to rewrite it again 188 05:32:40,790 --> 05:32:42,870 because in older versions of this compiler 189 05:32:42,870 --> 05:32:45,150 it was taking like 10 to 20 seconds. 190 05:32:47,050 --> 05:32:49,430 And then Cloud does this thing where like 191 05:32:50,150 --> 05:32:53,630 it'll try to like size up everything 192 05:32:53,630 --> 05:32:55,870 to judge whether or not it's worth working on 193 05:32:55,870 --> 05:32:58,470 but it's really easy to convince yourself 194 05:32:58,470 --> 05:33:00,410 that like your measurements are good 195 05:33:00,410 --> 05:33:01,910 when they're not good. 196 05:33:08,950 --> 05:33:10,570 and part of it was also like, 197 05:33:10,690 --> 05:33:12,210 why is this 900,000 lines of code? 198 05:33:12,350 --> 05:33:13,870 This should be a lot smaller. 199 05:33:19,530 --> 05:33:21,070 So then we did it one more time 200 05:33:22,970 --> 05:33:24,310 and the code was good 201 05:33:25,290 --> 05:33:28,590 and the build time was under 20 seconds. 202 05:33:36,410 --> 05:33:40,810 It's possible that we end up with a sixth time sometime later, but the fifth one is 203 05:33:40,810 --> 05:33:45,750 like very shippable, and I've worked on this for enough time that it's like time to ship 204 05:33:45,750 --> 05:33:45,910 it. 205 05:33:48,530 --> 05:33:55,090 A lot of this was like, just like, hey, Claude, try this idea, and then see what happens, and 206 05:33:55,090 --> 05:33:56,530 then it would like size up the idea. 207 05:33:57,910 --> 05:34:00,050 and I think this is like still kind of not right. 208 05:34:01,230 --> 05:34:04,950 Like I think a lot of this are like things 209 05:34:04,950 --> 05:34:07,610 that we should try again and not let it, 210 05:34:07,730 --> 05:34:12,190 like kind of what I said earlier about like sizing up 211 05:34:12,190 --> 05:34:16,170 the impact of a change is like it's really easy 212 05:34:16,170 --> 05:34:19,830 to size it up incorrectly and then talk yourself out of it. 213 05:34:21,230 --> 05:34:23,690 So I think this probably needs to be revisited more. 214 05:34:26,530 --> 05:34:30,350 My old approach, particularly with the Bunn-Russ rewrite, 215 05:34:30,810 --> 05:34:34,330 was very much leaning on cloud code workflows to do things. 216 05:34:35,390 --> 05:34:38,250 And pretty much now, with the most recent models, 217 05:34:39,390 --> 05:34:42,950 I mostly just rely on sub-agents that are very long-lived, 218 05:34:45,390 --> 05:34:49,070 except for that initial part where you're writing all the code all at once. 219 05:34:51,850 --> 05:34:56,330 With the sub-agents, it works pretty well because they keep all the context 220 05:34:56,330 --> 05:34:58,310 of the relevant code in their head, 221 05:34:58,530 --> 05:35:00,030 or like in the context window. 222 05:35:03,830 --> 05:35:05,730 And that's like less wasteful. 223 05:35:06,830 --> 05:35:09,030 And like they can just keep going. 224 05:35:14,510 --> 05:35:15,890 Here's a little more about like 225 05:35:15,890 --> 05:35:17,910 how the different parts of the compiler works. 226 05:35:19,790 --> 05:35:21,470 Generally it's like there's, 227 05:35:22,010 --> 05:35:24,830 if we can confidently prove that 228 05:35:26,330 --> 05:35:28,150 the value is, like the types are the same, 229 05:35:28,230 --> 05:35:28,990 that they're monomorphic, 230 05:35:29,150 --> 05:35:30,870 then it can become a field access. 231 05:35:31,390 --> 05:35:33,770 Otherwise, it has to check every single time. 232 05:35:36,130 --> 05:35:37,830 And it has like some inline caching, 233 05:35:37,870 --> 05:35:39,010 but not like a lot. 234 05:35:42,610 --> 05:35:44,470 As with a project of this size, 235 05:35:44,990 --> 05:35:46,450 you really do need to, 236 05:35:46,690 --> 05:35:48,150 it's kind of hard to follow sometimes, 237 05:35:48,370 --> 05:35:50,090 like what everything's happening. 238 05:35:50,490 --> 05:35:54,790 So I lean a lot on like just having an hourly update. 239 05:35:56,730 --> 05:35:58,330 and where it would post a scorecard 240 05:35:58,810 --> 05:36:01,190 comparing a bunch of metrics that we were prioritizing. 241 05:36:04,950 --> 05:36:06,550 Usually though, when it was posting in Slack, 242 05:36:06,570 --> 05:36:09,110 it would show like a previous and this version. 243 05:36:09,250 --> 05:36:12,190 So that way it was easy to see if numbers went up or down, 244 05:36:12,910 --> 05:36:14,930 but it's like not helpful in a slide 245 05:36:14,930 --> 05:36:17,470 because the previous one is just the JIT. 246 05:36:23,570 --> 05:36:24,910 The important thing is like, 247 05:36:24,910 --> 05:36:28,350 none of these tests, for the most part, are things that are made up. 248 05:36:31,090 --> 05:36:34,770 They're things that predated the project, 249 05:36:34,770 --> 05:36:37,690 so that way we know that the agent couldn't have messed with it. 250 05:36:40,390 --> 05:36:44,190 Fusili is like a JavaScript engine fuzzer that V8 uses. 251 05:36:46,710 --> 05:36:48,550 And we were already using that in Bunn, 252 05:36:48,570 --> 05:36:54,790 but we also extended it to test for TypeScript types as well. 253 05:36:59,650 --> 05:37:03,250 and it's also important to check that the tests actually catch things 254 05:37:03,250 --> 05:37:08,010 because it's really easy to fool yourself that it's finding bugs when it's not finding bugs 255 05:37:11,410 --> 05:37:20,710 some more numbers on the ahead of time compiler it generally for most projects outperforms the JIT 256 05:37:22,190 --> 05:37:23,030 not always. 257 05:37:23,410 --> 05:37:25,150 You can see in the uglify.js case 258 05:37:25,150 --> 05:37:27,270 and that's basically if it has to use 259 05:37:27,270 --> 05:37:29,050 the interpreter a lot, so if there's a lot of 260 05:37:29,050 --> 05:37:30,570 eval or new function 261 05:37:31,370 --> 05:37:32,630 then it's not faster 262 05:37:33,830 --> 05:37:35,310 but most applications 263 05:37:35,310 --> 05:37:35,910 don't do that 264 05:37:37,730 --> 05:37:38,250 and 265 05:37:39,030 --> 05:37:39,830 it's like 266 05:37:42,330 --> 05:37:43,510 it's as close as possible 267 05:37:43,510 --> 05:37:44,790 as you can get to a 268 05:37:44,790 --> 05:37:46,770 one-liner to make an application 269 05:37:46,770 --> 05:37:47,750 run faster. 270 05:37:49,150 --> 05:37:50,690 this is not 100% true 271 05:37:50,690 --> 05:37:52,050 because as you can see 272 05:37:52,050 --> 05:37:54,810 a GlifyJS and there are other cases as well 273 05:37:55,410 --> 05:37:56,710 but it is 274 05:37:56,710 --> 05:37:57,330 often true 275 05:38:00,490 --> 05:38:02,570 so like yeah the other thing is object 276 05:38:02,570 --> 05:38:04,510 prototype this is like 277 05:38:04,510 --> 05:38:06,450 a rare edge case but basically 278 05:38:07,570 --> 05:38:08,630 there's a lot of security 279 05:38:08,630 --> 05:38:10,510 vulnerabilities in JavaScript 280 05:38:10,510 --> 05:38:12,510 applications called like prototype pollution 281 05:38:12,510 --> 05:38:14,870 and this usually works by assigning properties 282 05:38:14,870 --> 05:38:16,290 onto object.prototype 283 05:38:17,030 --> 05:38:19,150 and the compiler bans that. 284 05:38:22,050 --> 05:38:24,450 So like for a large project like this, 285 05:38:24,530 --> 05:38:25,430 what do you actually do all day? 286 05:38:25,690 --> 05:38:28,450 These are like the top messages that I sent 287 05:38:29,410 --> 05:38:31,190 saying continue a lot. 288 05:38:34,330 --> 05:38:36,390 Also like my laptop kept dying. 289 05:38:42,010 --> 05:38:44,910 And then like, but like more concretely, 290 05:38:44,910 --> 05:38:49,870 what you actually do is like a bunch of leading questions like why are we doing 291 05:38:49,870 --> 05:38:56,170 this can we do this also I end up saying like can you do this faster or like 292 05:38:56,170 --> 05:39:00,550 what's taking so long because it kind of like talks itself into making up rules 293 05:39:07,310 --> 05:39:13,150 and also it tends to not be it tends to like underestimate what it can do 294 05:39:14,930 --> 05:39:17,930 so I often end up saying things like this to Claude 295 05:39:17,930 --> 05:39:22,510 to get it to try harder. 296 05:39:27,490 --> 05:39:31,030 And the result is that Jetstream 3 297 05:39:31,030 --> 05:39:33,090 which is one of the main benchmarks 298 05:39:33,090 --> 05:39:34,510 that JavaScript engines use 299 05:39:37,350 --> 05:39:41,310 the bun outperforms both JavaScript cores JIT 300 05:39:41,310 --> 05:39:46,490 and Node.js in like V8s JIT in most of them. 301 05:39:52,890 --> 05:39:56,230 And the whole time it's been using Cloud Code 302 05:39:56,230 --> 05:39:59,750 to build the Bun compile, to build Bun compile itself 303 05:39:59,750 --> 05:40:00,000 and 304 05:40:00,060 --> 05:40:03,640 then using the build of Claude Code that was compiled with Bun Compile. 305 05:40:09,340 --> 05:40:21,380 So sort of how I think about this is like the testability thing is super important when you're working on a large project with Claude and to make sure that the tests actually test things. 306 05:40:23,760 --> 05:40:26,140 and if you're not actively remeasuring 307 05:40:26,140 --> 05:40:28,320 then it's going to get worse 308 05:40:28,320 --> 05:40:30,460 because if there's so much stuff going on 309 05:40:30,460 --> 05:40:31,560 it's really easy to lose track 310 05:40:35,560 --> 05:40:39,060 and it's easier for large projects like this 311 05:40:39,060 --> 05:40:40,180 that haven't shipped yet 312 05:40:40,180 --> 05:40:43,720 to rewrite than it is to refactor 313 05:40:45,500 --> 05:40:48,000 and if you don't act ambitiously 314 05:40:48,000 --> 05:40:53,080 then the models will start having 0.1% improvements 315 05:40:53,080 --> 05:40:54,920 instead of 10% improvements. 316 05:40:55,320 --> 05:40:58,120 So you kind of have to constantly nudge it to like, 317 05:40:58,580 --> 05:41:01,140 hey, you can do really cool things if you try harder. 318 05:41:06,720 --> 05:41:08,280 You can try this now if you want 319 05:41:08,280 --> 05:41:10,420 by running bun upgrade dash dash canary 320 05:41:10,420 --> 05:41:14,480 if you have bun installed, and then just bun compile. 321 05:41:20,860 --> 05:41:22,560 Thanks, everyone.