Choose the project you can verify

5:19:18My name is Jared. I'm the creator of Bun, and I work at Anthropic. Bun is the JavaScript runtime that powers cloud code.

5:19:30And recently I merged a several million line of code PR to add an ahead-of-time compiler to Bun,

5:19:39which is basically the idea is instead of paying the cost of a JIT at runtime,

5:19:47you can compile all the code ahead of time.

5:19:55And it's like a really big project.

5:19:58Over the summer,

5:20:00I rewrote Bun & Rust.

5:20:05And this is a project on a larger scale.

5:20:08This ahead-of-time compiler is like a larger project than that.

5:20:13And I think one of the challenges right now with large projects is like,

5:20:17how do you actually confidently ship millions of lines of code written by agents?

5:20:25So that's what I'm going to talk about.

5:20:27This was like the opening prompt after some discussion.

5:20:31It was like, basically, we need to make Cloud Code use a lot less memory and faster.

5:20:39And we can modify either WebKit or Bun.

5:20:43Bun uses JavaScript core, which is WebKit, under the hood.

5:20:49And it's been a little bit over a month.

5:20:55this is this compiler has gone through five rewrites you can see the like the first couple

5:21:03ones were like pretty short and then I like almost shipped the blue one and then I was like

5:21:10this is building way too long like it's taking way too long to compile the code so rewrote it again

5:21:17and this next one is shipping next week.

5:21:24This is kind of the way I think about a project like this

5:21:28is like you're really constrained

5:21:30by how you test that the code works.

5:21:34So that's like even more important

5:21:36than what you're specifically,

5:21:37like the approach you go with for how to build it

5:21:39is entirely driven by how do you verify

5:21:41that it actually is going to like work

5:21:43and not be buggy and that you can trust it.

5:21:48And so just a little more background on like,

5:21:51like why this particular project,

5:21:54why the ahead of time compiler,

5:21:55and it's because for a typical JavaScript based application,

5:22:00roughly half of like the CPU time

5:22:03is just spent compiling code on demand.

5:22:07And so a lot of the general like thing you've seen

5:22:10in a lot of applications where people are rewriting

5:22:13their code bases in a language like Rust,

5:22:14or sometimes go.

5:22:17It's like when coming from JavaScript,

5:22:20it's because it's really the combination of the JIT

5:22:25and that it's a single-threaded application.

5:22:31So that's sort of the difference between a JIT and

5:22:34is that you compile before,

5:22:37whereas with a JIT you compile as you go along.

5:22:41This is particularly important for something like Cloud Code when using the Cloud Code

5:22:46like iOS app, where the CPU time is a lot if you have to have every session of Cloud

5:22:55Code also compiling.

5:23:01So we talked a lot about, should we rewrite Cloud Code in Rust?

5:23:07because the Bun team, us, rewrote Bun and Rust.

5:23:12So we're pretty good at rewrites.

5:23:17And we decided not to.

5:23:21And the reason why is because in Bun's case,

5:23:25it's actually pretty easy to do.

5:23:27Bun is an easy product to do a Rust rewrite

5:23:29because the tests were written in a language

5:23:32that's independent of the one it runs in.

5:23:34So because all the tests were written in TypeScript,

5:23:38we could run the tests on both the Zig version

5:23:40and the Rust version, whereas for Cloud Code,

5:23:46we, like Cloud Code's tests are written in TypeScript.

5:23:48So we would have to port Cloud Code's test suite.

5:23:53And as soon as you're like rewriting the tests,

5:23:56then you can't really trust the tests anymore.

5:24:00So that seemed like it'd be too risky.

5:24:03So instead, what if we just make JavaScript faster so we don't have to rewrite it in Rust?

5:24:12This was another prompt a little bit later.

5:24:18And the result is that it actually did have JavaScript's memory, like the memory usage.

5:24:27It uses about 4x less memory.

5:24:31it starts about two times faster

5:24:37and we're dogfooding this internally

5:24:40and it'll be external soon

5:24:45some more benchmarks

5:24:46and then I'll also go into more detail

5:24:48about how we actually did this

5:24:50so like it's possible that

5:24:54an interesting thing is like

5:24:56like TypeScript 7 rewrote their code by hand

5:25:00from JavaScript slash TypeScript to Go,

5:25:05and the single-threaded version,

5:25:10when compiled via Bun compile,

5:25:12is now faster than the single-threaded TypeScript Go port.

5:25:19And we will add threads also

5:25:21to the ahead-of-time compiled version of Bun,

5:25:23but we haven't done that yet.

5:25:26so there is a PR.

5:25:29And so a little bit about the actual ahead of time compiler.

5:25:35TypeScript is not, types don't necessarily need to be correct,

5:25:40so it treats types as predictions

5:25:43instead of trusting them.

5:25:48And this kind of problem works really well for models

5:25:51because they're just really good at writing compilers.

5:25:59So this is sort of like more about like how,

5:26:05how like should we,

5:26:06like how we were thinking about this decision.

5:26:09And a big thing here is like,

5:26:11there's just so many tests you can run when you're,

5:26:14when it's like in ahead of time,

5:26:16like for something like this,

5:26:18because we don't need to write any tests pretty much ourselves.

5:26:20Like we actually do write a lot of tests,

5:26:22but we can largely trust that the,

5:26:26tests that predate the project work. And this was the same thing with the Bunn Rust rewrite, where

5:26:32we had Bunn's existing test suite we could run. And now with the compiler, we have Bunn's existing

5:26:38test suite plus WebKids tests plus test262 plus CloudCodes tests. And then we also made it so that

5:26:47That way, BUN itself, BUN's own test runner

5:26:49could be run in the AOT mode.

5:26:57And we have one breaking change,

5:27:00or really it's two breaking changes

5:27:01with the at-of-time compiler,

5:27:03and that's pretty much why these particular tests

5:27:06are failing.

5:27:08But the two breaking changes are actually

5:27:12mostly security things to prevent prototype pollution

5:27:17tax, but they also end up benefiting for performance.

5:27:23So what makes this particular kind of problem pretty good for agents is you can run all

5:27:30three versions of an application, one with a JIT, one with ahead of time compiled, and

5:27:34one also without the types.

5:27:35And as long as the three of them behave the same, then you know that it works, at least

5:27:41for that specific application.

5:27:42but then you can take that kind of problem

5:27:46and fuzz it a lot.

5:27:50And just have clogged like churn out code generators

5:27:54to like fuzz different patterns.

5:27:58And you can catch a lot of bugs that way.

5:28:02This is a very large project.

5:28:04This is the biggest I think I've worked on

5:28:08in terms of like code volume.

5:28:11most of this is tests

5:28:14and also because we rewrote it five times

5:28:19when you're like doing this kind of project

5:28:22how I think about it is like you have like

5:28:25batches of like okay we need to write a lot of code at once

5:28:29and then after that initial batch then it's okay

5:28:32let's have let's like fix everything make sure it compiles make sure

5:28:37it passes a bunch of tests and make sure it like really truly works and the actual logistics of

5:28:45that are like when you're in the big batch kind of project stage then you don't want to run builds

5:28:52you want to run one build because otherwise you're going to spend all your time compiling and like

5:28:57waiting for build errors and it's going to like not be a good use of anyone's time

5:29:03and then separately for like longer running like once you have once you've

5:29:08written all the code you want to have it like have lots of individual agents.

5:29:18This is like a little bit of the timeline of this project.

5:29:26the first one was not very good

5:29:31like cloud code was a gigabyte

5:29:34which is terrible

5:29:39and like this used as much as possible

5:29:42from JavaScript core's existing code

5:29:45so it was like basically the ceiling was

5:29:47what is like only as good as like

5:29:50what it could be from that

5:29:55So then a little bit after this, we added a TypeScript type checker into Bunn itself.

5:30:03These were prompts. And so we added BunCheck and then we're like, hey, Claude, please use BunCheck. And then after that, startup time got two times faster in Claude code and started using about 3x less memory.

5:30:26But the code is slop.

5:30:31And like, this is not good, it should never,

5:30:35basically this meant that any object

5:30:37that accessed a property called archive,

5:30:40it would then assume that the bun.archive API

5:30:43was potentially used.

5:30:44And I looked at this and I was like,

5:30:46we cannot ship this, this is not good.

5:30:49So then,

5:30:58This is like, yeah, and this is another thing too,

5:31:00is like, it just like,

5:31:04like when you're doing like a large planning,

5:31:06like when you're starting to like have it

5:31:08do a bunch of design docs and stuff,

5:31:10it just loves writing markdown docs

5:31:12and it doesn't make sense.

5:31:14Cause who's gonna like 1500 markdown files

5:31:18is just not reasonable.

5:31:28So for this it was like, this was like I think the third compiler version, maybe the fourth,

5:31:33no this is the third.

5:31:38It, to like, just to kind of batch get it all out, 140 agents at the same time, write

5:31:45all the code all at the same time, then get Cloud Code to run.

5:31:54And then this is the fourth one.

5:31:59And this one, the code was not slop, pretty good, with basically the same performance

5:32:06as before.

5:32:07However, the compilation time was way too slow.

5:32:11because people coming from a JavaScript background

5:32:14don't expect any compilation time.

5:32:20So 50 seconds for...

5:32:22Cloud Code is a pretty large code base,

5:32:24but this meant that other projects

5:32:27would take single-digit seconds,

5:32:29and people don't want to wait that long.

5:32:35So we're going to rewrite it again

5:32:40because in older versions of this compiler

5:32:42it was taking like 10 to 20 seconds.

5:32:47And then Cloud does this thing where like

5:32:50it'll try to like size up everything

5:32:53to judge whether or not it's worth working on

5:32:55but it's really easy to convince yourself

5:32:58that like your measurements are good

5:33:00when they're not good.

5:33:08and part of it was also like,

5:33:10why is this 900,000 lines of code?

5:33:12This should be a lot smaller.

5:33:19So then we did it one more time

5:33:22and the code was good

5:33:25and the build time was under 20 seconds.

5:33:36It's possible that we end up with a sixth time sometime later, but the fifth one is

5:33:40like very shippable, and I've worked on this for enough time that it's like time to ship

5:33:45it.

5:33:48A lot of this was like, just like, hey, Claude, try this idea, and then see what happens, and

5:33:55then it would like size up the idea.

5:33:57and I think this is like still kind of not right.

5:34:01Like I think a lot of this are like things

5:34:04that we should try again and not let it,

5:34:07like kind of what I said earlier about like sizing up

5:34:12the impact of a change is like it's really easy

5:34:16to size it up incorrectly and then talk yourself out of it.

5:34:21So I think this probably needs to be revisited more.

5:34:26My old approach, particularly with the Bunn-Russ rewrite,

5:34:30was very much leaning on cloud code workflows to do things.

5:34:35And pretty much now, with the most recent models,

5:34:39I mostly just rely on sub-agents that are very long-lived,

5:34:45except for that initial part where you're writing all the code all at once.

5:34:51With the sub-agents, it works pretty well because they keep all the context

5:34:56of the relevant code in their head,

5:34:58or like in the context window.

5:35:03And that's like less wasteful.

5:35:06And like they can just keep going.

5:35:14Here's a little more about like

5:35:15how the different parts of the compiler works.

5:35:19Generally it's like there's,

5:35:22if we can confidently prove that

5:35:26the value is, like the types are the same,

5:35:28that they're monomorphic,

5:35:29then it can become a field access.

5:35:31Otherwise, it has to check every single time.

5:35:36And it has like some inline caching,

5:35:37but not like a lot.

5:35:42As with a project of this size,

5:35:44you really do need to,

5:35:46it's kind of hard to follow sometimes,

5:35:48like what everything's happening.

5:35:50So I lean a lot on like just having an hourly update.

5:35:56and where it would post a scorecard

5:35:58comparing a bunch of metrics that we were prioritizing.

5:36:04Usually though, when it was posting in Slack,

5:36:06it would show like a previous and this version.

5:36:09So that way it was easy to see if numbers went up or down,

5:36:12but it's like not helpful in a slide

5:36:14because the previous one is just the JIT.

5:36:23The important thing is like,

5:36:24none of these tests, for the most part, are things that are made up.

5:36:31They're things that predated the project,

5:36:34so that way we know that the agent couldn't have messed with it.

5:36:40Fusili is like a JavaScript engine fuzzer that V8 uses.

5:36:46And we were already using that in Bunn,

5:36:48but we also extended it to test for TypeScript types as well.

5:36:59and it's also important to check that the tests actually catch things

5:37:03because it's really easy to fool yourself that it's finding bugs when it's not finding bugs

5:37:11some more numbers on the ahead of time compiler it generally for most projects outperforms the JIT

5:37:22not always.

5:37:23You can see in the uglify.js case

5:37:25and that's basically if it has to use

5:37:27the interpreter a lot, so if there's a lot of

5:37:29eval or new function

5:37:31then it's not faster

5:37:33but most applications

5:37:35don't do that

5:37:37and

5:37:39it's like

5:37:42it's as close as possible

5:37:43as you can get to a

5:37:44one-liner to make an application

5:37:46run faster.

5:37:49this is not 100% true

5:37:50because as you can see

5:37:52a GlifyJS and there are other cases as well

5:37:55but it is

5:37:56often true

5:38:00so like yeah the other thing is object

5:38:02prototype this is like

5:38:04a rare edge case but basically

5:38:07there's a lot of security

5:38:08vulnerabilities in JavaScript

5:38:10applications called like prototype pollution

5:38:12and this usually works by assigning properties

5:38:14onto object.prototype

5:38:17and the compiler bans that.

5:38:22So like for a large project like this,

5:38:24what do you actually do all day?

5:38:25These are like the top messages that I sent

5:38:29saying continue a lot.

5:38:34Also like my laptop kept dying.

5:38:42And then like, but like more concretely,

5:38:44what you actually do is like a bunch of leading questions like why are we doing

5:38:49this can we do this also I end up saying like can you do this faster or like

5:38:56what's taking so long because it kind of like talks itself into making up rules

5:39:07and also it tends to not be it tends to like underestimate what it can do

5:39:14so I often end up saying things like this to Claude

5:39:17to get it to try harder.

5:39:27And the result is that Jetstream 3

5:39:31which is one of the main benchmarks

5:39:33that JavaScript engines use

5:39:37the bun outperforms both JavaScript cores JIT

5:39:41and Node.js in like V8s JIT in most of them.

5:39:52And the whole time it's been using Cloud Code

5:39:56to build the Bun compile, to build Bun compile itself

5:39:59and

5:40:00then using the build of Claude Code that was compiled with Bun Compile.

5:40:09So sort of how I think about this is like the testability thing is super important when you're working on a large project with Claude and to make sure that the tests actually test things.

5:40:23and if you're not actively remeasuring

5:40:26then it's going to get worse

5:40:28because if there's so much stuff going on

5:40:30it's really easy to lose track

5:40:35and it's easier for large projects like this

5:40:39that haven't shipped yet

5:40:40to rewrite than it is to refactor

5:40:45and if you don't act ambitiously

5:40:48then the models will start having 0.1% improvements

5:40:53instead of 10% improvements.

5:40:55So you kind of have to constantly nudge it to like,

5:40:58hey, you can do really cool things if you try harder.

5:41:06You can try this now if you want

5:41:08by running bun upgrade dash dash canary

5:41:10if you have bun installed, and then just bun compile.

5:41:20Thanks, everyone.