Code Generation and Maintenance at Scale

[on hold music] So I'm Morgante.

I'm the founder of Grit. I'm gonna be talking about code generation and maintenance at scale, uh, or CPUs still matter and what it takes to actually make one of these agentic workflows work in production.

Quinn was talking about how most people have not merged them. That's a hundred percent true. Uh, Grit has probably merged more PRs than any other company in the world at this point because we focused very narrowly and have done a lot of work above the model layer, and we're gonna talk about how we did that.

Uh, it's helpful to know why I started Grit. My background is all developer tools. Uh, I've been working at Google Cloud for five years and built a lot of stuff, uh, on the DevOps layer, right?

Thinking about Kubernetes and how do you orchestrate very large scale systems, uh, working on tools like Customize or Terraform templates. Uh, and one of the biggest things I learned from this was how rare it was for a customer to come to us and ask for a brand-new application, right?

People didn't come and say, "I wanna build a new app on Google Cloud." It sounds cool. Ninety percent of the time customers came and said, "I have this line of business application that is doing a hundred million dollars in revenue.

How do I run that on Kubernetes?" Right? And that's what all of our templates did. That's what everything we built, uh, in the sort of pre-AI era of automation was all on how you ran existing applications.

And that's why we started Grit because every demo that you usually see that's hyped, you know, one of these ones on Twitter, it's usually type a prompt, get a new application from scratch, right?

Build something brand new. It's exciting. It goes really well on Twitter. That's not what devs do day in and day out. Developers spend most of their time modifying huge applications so that flights run on time.

And this goes into three sort of categories of developer tools, right? There's ID developer assistance. This clearly has the most product market fit today. It's really easy, right? Like Quinn was saying, you just do auto-complete, right?

That's a very simple thing. Then there's AI agents that are focused on lowering the floor, right? They're allowing people to do tasks that they otherwise don't have the skills for, right?

Allowing a product manager or other non-technical user to build an application that they don't have the skill set for. This is powerful, but I actually am pretty skeptical that that's how most software is being built in the future.

It requires a real thinking of how do you actually spec things out? How do you think about edge cases? Basically, how we train as engineers that's required to build great software.

Which is why with Grit, we focus on raising the ceiling of what great engineers can do, right? Principal engineers, the most high-level engineers that you work with, they're primarily limited by time, right?

How-- they can't be in ten places at once. But AI agents can be in ten places at once if there's the right engineer controlling them. And that's what we focus on, is supercharging the productivity of the top one percent of engineers.

It also helpfully gets around the problem of ninety-five percent of engineers not using AI. Uh, the great thing about Grit is ninety-five percent of our customers, ninety-five percent of their engineers don't touch it, right?

There'll be a hundred engineers on the team. They're not l-- using Grit. There's one engineer who is deeply embedded with Grit and is generating hundreds of PRs with their agents.

But to do this, tools need to change, right? The ID that you have today, it's a scalpel, right? It's focused on editing individual lines of code, and it's great for that, but it's not focused on editing hundreds of repos at once.

It's not focused on how do you open a thousand files and make those changes in them, and that's why we built Grit. It's bas-- So we wanna have bulldozers for code, right?

If you're generating huge quantities of code, how do you push that around in an effective way, uh, when you're not editing individual lines when you're working a higher level of abstraction?

And this is super necessary, right? We've seen an explosion in how much code is being generated. A lot of our customers are seeing, uh, twenty to thirty percent more code, uh, is coming out of their teams now just because there's more PRs, there's more CI, there's everything that's running because you have code gen.

And this is gonna accelerate, right? Once we go from five percent to maybe fifty percent of people actually using AI, there's gonna be way, way more code in the world, and we need better tools for managing that code once it's in production.

So just to give an example of what this looks like in practice. Uh, this is a real cus-customer that we had. Uh, they are-- been around. They've got thousands of repos.

They've got thousands of developers, and they wanted to use OpenTelemetry instead of logging, right? This is traditionally a massive effort, right? You have to coordinate across hundreds of teams to get them to understand OpenTelemetry, to understand how to instrument their code.

You have to get them to do the actual code changes to swap out their logging library. You have to do a bunch of education efforts. And it's actually very much a people and process problem usually, right?

Something where you have a program manager who has a massive Excel spreadsheet. That's when I say, like, Grit we compete with is actually Excel, not any other AI dev tool.

It's that you'll go have these spreadsheets where you go and manage these changes, right? And tens of thousands of developer hours go into a change like that. So a lot of companies just say, "You know what?

That's not worth it," right? "I'm not gonna migrate to OpenTelemetry. I'm not gonna go into the cloud. I'm gonna stay in my older ways." You know, people still have millions of lines of COBOL because it's just so much work to do this kind of coordinated change.

Uh, with Grit, you don't have to do that coordination effort, right? Because you can have one engineer who is actually coordinating that change, is driving individual AI developer agents to do the changes.

You don't have to have a bunch of meetings because it's just one person telling their, their little agents what to do, right? And you can do it with under a hundred developer hours because they're just doing high-level orchestration and then thousands of, uh, compute hours that the AI is doing as it's healing these changes.

And we've seen ch-- This is literally a project that they had postponed for years because it just was not feasible. They couldn't get the-- get enough on the roadmaps.

They got it done with Grit in a week, right? It just opened up a thousand PRs across all the repos, fixed them, uh, iterated on the changes, merged, and migrated over.

So how do we actually do a change like that? Uh, it's sort of a three-level process. Um, planning is a big part of it, right? So we index the entire code base.

Uh, we bui-do both semantic indexing, so understanding, uh, embeddings and understanding the intent of a file. Uh, but we also do a lot of traditional, more static analysis indexing, so we understand what's the structure of the code, what's being imported from where, what's the dependency graph, right?

This is all the sort of thing you need to know to actually do really high quality agentic flows.

Then once we have the plan of how we're gonna make changes, we execute the plan, right? So we use large language models that are gonna b- to take that change, delegate it to a sub-agent.

The sub-agent is gonna make a m- modification in one file. Uh, it uses something b- called GritQL, as well as diff generation from language models. Uh, GritQL is our custom query engine that's able to actually index, uh, millions of lines of code and find the right places to modify things, and then finally push it up for PR

review, uh, and be able to, uh, both have developers who are the director of it, right? So a typical scenario is that there'll be the principal engineer who's driving the change.

They'll review the PRs. And then individual developers, their primary interaction with Grit is just seeing a PR land in their repo, and they m- leave like one line comment, uh, that Grit will learn from.

But they don't actually open up the Grit UI ever because they're just responding to the changes that come from Grit.

Cool. So a little bit more about how we find code, right? So our goal here is to find all the error logs in the codebase, 'cause we wanna migrate those over to OpenTelemetry.

Uh, the naive approach that if you go to many of the workshops yesterday would be, all right, just chunk it, put it into RAG. Uh, you have a bunch of embeddings.

Uh, and you know, that, you know, theoretically could work here for maybe some document use cases. I'll tell you that absolutely will not work for this problem, right? If you just go to try to find stuff that looks like, looks like a logging error, uh, it's gonna find a lot of irrelevant context, right?

It's gonna find anything that looks like log-like. It, it-- LLM has a hard time differentiating between a user-facing log, like an alert in a UI, and an actual log that you wanna be putting into OpenTelemetry, right?

There's also unpredictable thresholds, right? You don't actually know how much code you're looking for. You can't do, you know, retrieve the top ten, uh, closest matches. In some cases you wanna retrieve ten thousand.

In a lot of cases, and developers don't even actually know how big of a change it is until Grit starts to propose it for them, right? So that's why we built the GritQL query engine.

It's our own custom query system that combines the best of static analysis with the best of AI. So we've got this query here that's looking for, uh, logger, uh, with some set of args, right?

So we're just gonna look for a function call, basically. Uh, and that's a syntactic query, so we're just looking for all of our function calls across our entire codebase.

Uh, and then we're gonna say that our args should be like an error occurred, right? And that's just, we're giving an example of like what's an error that-- message that we might be trying to look like.

Like there is a magical word that converts it into a semantic representation. So we wanna say, what's some code that, you know, embedding search, the cosine similarity is sufficiently above a threshold that this is an actual log message versus some other function call that we wouldn't be wanting to modify.

And then we can finally do a imported from. We've got a built-in library that's able to understand the whole dependency graph. So we can do things like make sure this is actually imported from log4j.

In this example, we wanted to make sure that we're only substituting our log4j logs, and that will go and traverse, uh, the import graph earlier in the program.

So that brings us to finding the code that we wanna change. But once we've actually found the code, how do we make reliable changes? Uh, and unfortunately, really smart models, uh, still have a t- hard time doing this completely autonomously.

Uh, just to give an example, uh, I just used Claude Sonnet today, uh, three point five. It's a really good model. Uh, and it uses, uh, we put, put the entire files, uh, so a bunch-- put a bunch of context into the context window, a hundred thousand tokens from Grit's VS Code extension on some of our linter

outputs. Uh, and we wanted to just write a function that's gonna convert from our linter JSON output, uh, and puts it into diagnostics for the Grit VS Code extension, right?

Pretty simple task. I promise that everything that's required was in the context window, right? It's not something where it had to go retrieve additional information. Uh, it was all there.

Uh, ca- came back with a pretty reasonable completion, converts ESLint to LSP diagnostics. Uh, this looks reasonable to me. Like I i- imagine if you looked at this code, you wouldn't be able to tell anything that's wrong with it.

I certainly couldn't tell anything f- that's wrong with it from eyeballing it, right? Uh, but this is wrong, right? And this is one of the main things to understand is that, uh, humans also can't look at this code and understand what's wrong with it, right?

This is why we have systems that allow us to, uh, type check, lint things, right? It allows us to understand code that even looks kinda correct is, in fact, incorrect and will fail in production.

But I went back and just asked, uh, Claude to fix the code for me. I said, "This broke in production. I tried to put it in my VS Code extension, it broke."

And I just ask it why. Uh, as you can kind of imagine, doesn't do any better than I do of just looking at it, looking and eyeballing the code and understanding why it's wrong, right?

Uh, so it just comes and spacks says some totally irrelevant answer of how to fix it because, again, it's not grounded in what the actual errors are.

Uh, fortunately, uh, we have a great tool for TypeScript called TSC, right? We can compile this, and it's gonna go and tell me actually, uh, Grit positions and Grit ranges have a slightly different type signature, uh, than LSP ranges, right?

And this is, you know, why compilers are great, is we can actually get that information really close in the dev, dev loop. With this information fed back into an LLM, it's able to correct that mistake, uh, no problem, right?

That's a pretty easy change. It uses the convert LSP range to Grit range, which by the way, was in the context window. It could have used that before. It just didn't realize that it needed to use that until it had the compiler error forcing it to, right?

So this is already how I think I-- It's important to see that IDs are already making us superhuman, and we need to make sure that all of our AI agents have access to the same tools that make them, uh, super AI.

Uh, so compilers rock, right? This basic flow of prompt, get some code, uh, build it, type check it, and then fix that output based on the LLM. This is actually really powerful.

This is probably, uh, half of what you need to do to build a really good agent, is make sure that you have this flow working reliably.

Uh, but they're really slow when you're talking about enterprise codebases. Uh, so this is real numbers from, like, one of our customers. Uh, it takes them ten minutes to build their application from scratch, and that's just for type checking.

It's not even producti- pushing a production build, right? And this is actually pretty typical if you look at very large scale enterprise codebases. Uh, that's why large companies have had to build a lot of caching, because it's hard to build a large codebase from scratch, which this is completely different than what people usually expect for AI.

People usually think inference takes a long time, right? You're waiting for an AI. And this is actually a pretty long prompt, thirty seconds, right? We're using a huge model to cr- generate this code.

Takes thirty seconds. But that's dwarfed by the ten minutes to build the application, right? This basically destroys our entire agentic flow if we're waiting ten minutes for every single change to validate that it's correct.

But- This is even more compounded if we're trying to do that in a loop, right? If we're trying to do a single change, it might take a day if you're just doing this naively.

Uh, there's some agent projects that, in fact, do take a day to make very basic changes because you don't have, haven't done this optimization level. But you might ask, like, how are you able to make changes in your IDE at a fast rate, right?

You're not waiting 10 minutes every time you make a single keystroke to get a compiler error. Uh, it's much faster than that. It's because there's been a lot of work with language servers, uh, to solve this so that you can do a bunch of upfront prep, so you can build the index in memory, have that in-memory queryable

index, and then only rewrite the parts or only recheck the parts that you've modified, right? On every keystroke, uh, most tools, like TS Server, for example, in TypeScript, uh, is doing live reconciliation of figuring out that specific file, right?

And this is much, much faster. You can do the 30-second prompt, then one-second recompute from the TS Server, then 30 seconds to fix it, and this is a much more reasonable flow, right?

So you obviously wanna be using the same kind of language server tools that you'd be using as a human, uh, not CLI-based tools, which often don't have the same heuristics in place to be able to optimize.

And then, you know, ideally, you do this in a nice loop. You eventually get to the point where you can commit and get a gr- fresh PR to do that migration to OpenTelemetry.

Th- but this is what it looks like in theory. In practice, at some point, it hits an error that it can't fix, right? It hits a error that gets into a loop, and it's continuously trying to fix the same error.

It uses five different techniques, then goes back, and your context window's completely polluted with the wrong errors, right? Everyone s- often says, like, "Agents don't work." Uh, this is probably half of the reason that agents don't work, is that you just have compounding failures, right?

We found anytime we actually have more than 10 prompts in a row, uh, our chance of having a successful PR is dramatically lower, right? Uh, so the way we work around that is instead of trying to repeatedly fix an error, uh, we should actually just save our o- original state, revert back to that, uh, and then re-

continue to edit from there, right? So if we went down a path that was just a bad path and we got stuck in a row, we wanna go back to a known good checkpoint, uh, and then build from there.

And this is actually how we're able to do this quickly. We don't wanna spend 10 minutes recomputing each time. Uh, we wanna actually build our in-memory graph that we talked about with TS Server.

We wanna save that. We wanna take a snapshot of memory. Uh, so we use Firecracker. It's a VM manager, uh, that's used for AWS Lambda, but we can actually use it for dev environments too.

Uh, and we can actually take the in-memory state, snapshot that, and then fork it into 10 different isolated environments that all have everything pre-computed. You can try 10 different changes in them and then figure out the correct change that is most likely to yield good results from there.

And in fact, this becomes massively parallel. You can end up with an AI system that looks more like a distributed database than it does a traditional agent or something that you're running on your laptop, right?

We actually have flows where we often have, uh, six up to 10 different agents working in parallel, all working from a known good state. They're supposed to report back once they're done.

And then we'll actually look at the different evaluations. We look at, uh, both some LLM-based evals, but also heuristics like how many errors that are currently in the code base, uh, how many unit tests are currently passing, uh, and then actually compute, like, what of these, which is the quorum, right?

It's actually similar to, uh, again, a database system where you would have a voting of, like, what's the new master. Uh, here it's like, what's the new good state that we wanna fork from?

Uh, if you have these four here that have similar states that we wanna use that as our new known good state, uh, save that as our known good state, and then fork from there going forward, right?

And this ends up being much, much more reliable because we can have an entire PR that, yes, we've done 30 or 40, uh, different generations on it, but in the final chain, there was only four different generations, right?

Because we had one, then we got went back to a known good state. Then second one is all operating from that quorum at each checkpoint.

But these edits get pretty expensive, right? If you're doing, uh, 40 different edits to make a single PR across very large files, uh, that's a lot of money that you're spending on inference.

Uh, this is a common problem with making good edits. Everyone naively just asks for, "Generate the whole file again," right? It's the simplest approach. You definitely should start with that if you're building your own AI tool.

Uh, but then you run into the classic problem of laziness. So this is actually still from, uh, Sonnet. It still said, you know, uh, the start of the function remains the same, right?

Left this comment in because it didn't wanna output that code. And it's just because output tokens are fundamentally more expensive. And if you look at GPT-40, it's a five to $15 ratio of input tokens to output tokens.

Uh, Claude 3.5 Sonnet is three to 15. Uh, this is pretty consistent across the board. And then response limits are not growing at the same level of context size, right?

We've got models out there that have 1.5 million tokens, 2 million tokens in their context window, and still only outputting 4,000 tokens at a time, right? Because it's autoregressive, it gets much more expensive.

So you really don't wanna output entire large files as you're making edits. You wanna find a good edit format. Uh, so whole edit format works well. Uh, it's very expensive though.

Uh, you can do diffs, right? You can say, like, "Generate a unified diff for this." Try to apply that. Um, there's some problems with this. One is, like, line numbers.

Uh, LLMs are still not very good at knowing what the right line number is, even if you give it them. They're just not that good at the math part.

Uh, and there's also off-distribution, right? Real-world code they're trained on is largely not trained on diffs, right? It's trained on actual full files. Uh, you can do simple search and replace with function calls.

Uh, the problem with this is function calls are underneath, for the most part, JSON. Uh, escaping code in JSON format is terrible. You end up using a lot of tokens just for, uh, just escape characters, right?

It's just not a very good format to use. Uh, so that's why we actually developed a GritQL loose search and replace. So we can actually do something that's similar to what you would have on the model of being just a before snippet, and this is something you might have, like, in a tutorial, which is like, "Replace this

with that," right? This is what we ... And this is actually the exact same output that comes from the LLM. We'll do a match. Like, we'll do a loose match to try to find what's the code that looks like that, and replace it with the code that looks most similar to that afterwards, right?

And this works really, really well because we don't have to ... We can elide irrelevant details, like what's currently inside the makeMatch function, and just give enough detail to make the replacement.

Cool. Uh, and just wanna leave you with where we're going next. This is our current UI. It still is very traditional, right? It still is, uh, building a, you know, what's a AI workflow.

Looks kinda like your CI, even though it's thousands of agents executing. Uh, I'm really excited about where we go next with this, uh, figuring out, like, what does it look like to manage an entire code base.

I think of, like, SimCity as, like, the ultimate where you can zoom in and out and understand, uh, different levels of granularity and edit things there. Cool. Uh, thanks so much.

I'm Morgante, and we are hiring, so, uh, scan the QR code. [upbeat music]