One Day, $1,000 in Tokens: An AI War Room for Legacy Code — Ford Prior, GEICO

For the past couple years, feels like all we've been talking about is how to increase the velocity of our SDLC. Y'all know what I'm talking about. And specifically, this big fat build chunk in the middle, okay? Where we've been doing a lot of our coding and intellectual work. And sure enough, at least where I work, that step has basically been obliterated, right? And we've reclaimed a bunch of time. But I think, and I'm no prophet in imagining some future where

agents are doing the work and humans are there in the middle as exec-- or, uh, as executives signing off on each stage. But I work at GEICO, and we're not a tech company. We're a risk management company, and risk management is baked into everything that we do, including the way we develop and deliver software. So while we've seen this inner loop speed up, to use Gene Kim's concept, we find ourselves seemingly stuck within the bounds

of this outer loop, which is to say everything else that happens other than the coding and the building.

And we find ourselves wondering what life might be like outside of these walls in a world where the outer loop keeps pace with the inner loop. So this is a story of how we broke free, as it were, and my intention with this talk is to empower you guys to do the same.

Now, when GEICO manages America's fleet of automobiles or the risk they're in, and something happens on the road like an accident, we incur what we call a loss,

which is typically reported by one of our policyholders, and it becomes a claim. And claims technology is where the policy comes to life and gets the customer back on the road as quickly as possible with minimal disruption, which we call a resolution. And I'm, I'm Ford Prior. I'm a senior staff engineer and tech evangelist embedded in the claims tech group at GEICO. And I think I'm the only tech evangelist at Berkshire Hathaway. But basically what that means is within our three hundred engineer

group, we have, like, thirty two-pizza teams, as we used to call them, with a tech lead and counterpart for each-- business counterpart for each. And my job is to find and implement best practices in our SDLC. I run a claims innovation lab, and I share stories like this one to attract the best engineers in the world to our problem space, which we find very interesting, not to mention important part of the American economy. So a little bit more context on how we develop software. Um, as many organizations

do, we conceive of our work in terms of semantic versioning. And about six months ago, we looked at our backlogs and found that our engineers, after they finished their latest patch or chased down the latest minor upgrade, had only twenty-five percent on a good day or less of their bandwidth to dedicate towards the major changes that keep us competitive, a step ahead of the competition. That's like V one to V two to V three. So we began strategically investing in GEICO-managed AI agents

on a variety of harnesses using a variety of models to offload that work all the way from spec to deployment for our patches and minor work. Um, with the goal of redirecting or rerouting that work so that our engineers can stay-- our, our brilliant engineers can stay focused on the stuff that matters and that they find most joyful, which is to do the major work, which again, as I mentioned, is key to our competitive edge. Okay, so this is current state. But as we look at this major work, we find ourselves

rate limited by our ability to collaborate in rooms full of the right individuals summoned at the right time to make the right decisions at every interval in whatever life cycle would need to happen in order to get, get the major work to prod.

And not only are we missing the ability to collaborate, but we're also missing this intelligence, this ability to weave together all the variables relevant to the problem space as we learn it and deepen our understanding of it, including the fact-checking, the code crunching, basically everything that needs to happen to serve up the insights as quickly as the decisions come up. And that's what ultimately gives us the confidence to bring those major changes to market, okay? So it's collaboration and intelligence. We're missing both of those at the major change scale. So to solve

for this, we create a multiplayer AI native war room, and it consists of a roster of key decision-makers and subject matter experts relevant to the topic, a driver to wield these expensive AI tools, and an AI native productivity suite, um, which includes our comms channels for all the chatter, an interactive design studio. I can't mention the vendor names, but y'all can take a wild guess. Um, to, uh, for all of our, our collaborative elements, and then obviously we have our code repositories.

And all of these are wired into a powerful combination of workflows, agent skills, MCPs, enterprise MCPs, um, and these agent fleets to do the heavy lifting, fact-checking, synthesis, et cetera. So my question was, how might we accelerate the delivery of agent-callable APIs for our legacy core capabilities? And for context, our legacy claims systems is, like, millions of lines of code, okay? So, um, and the deep context that we're trying to

build here, um, gets built once and then can be reused. So that's just some additional context. But my bet was that with a thousand dollars in hand of AI tokens, in one day we could map out one domain end to end, including like graph representations of the business topology, all of the complex code, um, in our legacy systems and everything in between. And we can have the agent find and propose one serious change Um, which in

our case, we scoped down to an agent-callable API into our legacy claims systems and have the whole thing deployed to our skills agent platform and running for a, a, a demo to my boss at four PM. So I put the pressure on myself. Um, but we ran this experiment three different times across three different code modernization missions. Um, and this little taxonomy here, uplift, transform, reimagine, is from the Anthropic code-modernization plugin, which I'll touch

on in a minute. Um, but the story is, um, that we first we looked at our rentals domain, which is a relatively simple one, um, that is like all the tech behind getting you guys your rental cars when your, your primary car gets in an accident. And we implemented a newer version of the same tech, which is again, the definition of uplift, and we extended an existing API. And in this case, it was an interesting story because the agent used some, uh, interesting, um, uh, process to find and distill data to figure out that we

could connect a couple dots and save ourselves eight and a half thousand calls per month by giving the customer up-to-date rental information. Um, so that was the first experiment that we ran. Um, and then the second one was transformation of our damages domain. And damages is like arguably our most complex-- No, sorry. Damages, I'm reading from notes here. Um, damages domain is how we put dollar signs on a reported loss. And so for this case, we built a, a set of new agent-callable

APIs to unlock a larger workflow that needed damages data. And then lastly, we reimagined our salvage domain, which is arguably the most complex of the three. Um, it's basically everything that happens when your car is totaled. Um, and in this case, we produced a fully vetted migration plan with API specs and design mock-ups, ratified, and this is the most important part, green-lit by product and tech at every level. Okay. So it was like a fully vetted project plan that's ready to be implemented, which is like, as you guys know, is like, that's like a, a milestone unto itself, getting everybody on the same page,

um, including the technical details. Um, and honestly, like, in this case, it understood the domains really well. And, um, and this-- it-- and the, the story with the salvage work was that it, it found a, a prod bug. I can't talk about more, but it was, it shocked everyone. Um, so anyways, uh, you may be curious about kinda what, like, a little bit more detail into how this all worked. Um, really, like we had our friends at Anthropic who provided a bunch of stuff, and then we had us that provided a lot of stuff. So on the left here, we have,

um, the harness, the model, this thing called FleetKit, which I'll explain in just a moment, and this plugin that did code modernization tasks. That's on the left, brought by the vendor. Okay, we could replace any of that, by the way, with another vendor, another model, another framework, et cetera. And on the right was the stuff that we can't replace. This is everything GEICO, and we need a ton of it. We had context in our knowledge base, including software standards and all of this past decision archeology we had to do. List of tools that the agent could access on my machine and

instructions for the agents on how to use it. We had the methods. I'm doing these out of order here, but we had methods, which is essentially like all of our skills and all of the, the, kind of the quirky, domain-specific, highly optimized way that we develop software, probably different from the way you guys do it. Um, and, um, and lastly, judgment, which is like who can decide what and at what gate, right? So both from the subject matter expert point of view and from the, um, the, uh, decision-maker leadership point of view. Um, and, um, we grouped them into two repos, and I'll explain why in a minute, but that allowed us to vers--

allows us to have the agents add to it and correct it, um, and also humans do the same. Um, but that AI, and this is kinda one of the interesting parts, uh, is, is expensive, right? To run at scale. And so we needed a fleet manager, like a PhD-level fleet manager. And this was a bit of a collaboration with, uh, with Anthropic, and it isn't open source yet, but I can share how it works today in hopes that you guys can construct your own version 'cause it's not rocket science. So FleetKit is this tight, structured collection of rules,

protocols, and instructions all written in Markdown, optimized for the Claude harness, specifically Fable, and designed around the core task of architecture transformation. Um, and as you guys can kinda guess, like, this type of thing doesn't really work with some of the previous iterations of models. I think like Opus 4.8 and beyond, I, I, I would, I would guess is like something that could handle this type of like, uh, steering, um, 'cause it's like one level above, um, the actual system prompting, right? This is like rules that it has to

follow, the agent has to follow. Um, and, um, it's pretty simple. Takes a goal, breaks it down, dispatches leaders to do all the research and decide about the approach. It puts everything into this like human-in-the-loop queue. You could use GitHub Issues, for example, something that the humans can comment on and improve or that other agents can add more context to. And then when everything looks tight from a approval standpoint, it dispatches builders to do the rest. And in this case, we use Beads. I'm not sure if y'all know about

Beads, but Beads kinda helps the agents keep track of everything, um, within their own agent space. Um, and, um, ultimately, the goal was to produce what we call a material change, which is like a, a change request into our systems or a code change into our code repositories. And the ultimate, like, idea with this though is that, like, developing software looks less like playing in a canvas and more like working in a factory. Um, okay, I need to go faster. Uh, so ultimately, on the left here, you see these three repositories that contain

everything that FleetKit needs to run, including the FleetKit source. Um, and by the way, all of these slides are available on the blog post that I'll link at the very end. I can't tell with these lights who's taking pictures, but I just wanna let y'all know that if you wanna look at the, uh, the, the visualizations that I'm showing here, they're all on a blog post that was released today, and I link it at the end of the slides. So anyway, um, so all this boots up, gets pulled into a Glod- Claude Code session with a plugin, gets invoked, and you give it a mission. It a-adopts or creates a Slack channel, um, or a, a-- in our comms

platform. And, um, and then that's, that, that, that mission channel is where everyone communicates. All the war room members, the people who are relevant to the decisions and the project are invited. And, um Every time the agent reaches a milestone, um, as it follows its nine-step process, it creates one of two artifacts. Um, and by the way, on the right there where it says War Room, that's where it, like, it can ping individual people based on the roster. So for example, if it's missing, missing some product details, it'll figure out who owns that from a product perspective and mention that individual person.

Um, but other than that, it creates these worksheets that are, like, actually really well done HTML using Claude artifacts. People can fill them out. They're automatically saved. Um, and that's to fill the gaps. And it also creates design documents, including, like, these, like, complex visualizations of our architecture that didn't exist before, um, and that teams were delighted to get 'cause they kept them afterwards, um, that we can then edit and then the agent can then reread our edited, like, uh, design document, like a draw.io or a Figma, and then all that gets that sl- all that gets slurped

back up into the, the, um, the session. Okay. Um, and if you had stumbled into our roo- war room in Richmond, Virginia, which is where I'm from, um, uh, this summer, you would have seen either a big Claude Code screen, um, with a bunch of people huddled around it or one of these two screens here. On the left is like a product worksheet that we had to fill out, and then on the right, um, there it's like one of our, our diagramming tools. And so, like, the big gist with this and the thing that was so delightful for me is that

these were really high-quality artifacts. So there wasn't, like, ever a question of, like, "Why is the agent asking me dumb questions?" It was like, "Wow, that's actually a great question that we've never thought about," identifying an edge case either in the business or with the technology. Um, and then the design artifacts, like I said, the teams could keep them afterwards, and oftentimes they were mapping stuff that just no one had taken the time to do. 'Cause as y'all, as y'all know, not every engineer loves doing diagrams. Okay, I'm skipping stuff here, but we're gonna keep moving 'cause I got, like, three minutes. Um, these War Room tools, um, just really simply,

uh, basically there's two types of agents, the code dives that we called them, um, where it did lots of code crunching and graphing of our, of our legacy, legacy domain-specific languages. Um, that happened in the cloud, and then we had local agents running that had access to all of my laptop, my-- all the MCPs on my laptop, uh, many of which were not in production yet, in enterprise-specific, highly optimized for our problem space, um, and that could surface really valuable data, including, like, chunked versions of the code bases that were being dived.

So, um, there's, and there's a bindings file that I gave the agent to basically describe what's on my machine, how it works, and how to map it to the FleetKit work. Um, and you can see the built-ins, MCP servers. The-- Really quick, the code modernization plugin, um, this is at the heart of the whole thing, and what's inside of it, uh, I think, uh, you guys will find somewhat interesting. Um, and I would encourage you to, to look into it after the fact, after this presentation to get more information. Um, and I'm gonna, I'm gonna keep moving here. Um,

ultimately, the driver's laptop, my laptop was the runtime, okay? So I had to run nine steps that are, that are, um, started with the booting step, so eight steps after that. And what I want you all to take from this super busy slide is that on the left here, those were all steps forked from the code mod plugin, okay? So we adapted it to our needs, as I would, would encourage anyone to do. And on the right, those orange-colored elements are worksheets that were created, so that was gonna require human approval. And so as we did this, like,

modernization journey, you can see how there was a lot of human input in the beginning, less on the, on the end, and how the core of it was this code modernization plugin.

And, uh, just an interesting detail, and this is something that the code modernization plugin does, is it does, like, a deterministic analysis of, like, did the previous step create the artifacts that it needed to, including a file? So some interesting lessons. Um, obviously I burned a lot of, uh, burned a lot of tokens. Um, and, um, one, uh, just as a little takeaway, do not let a Claude running Fable set up crons to check on work, uh, because you'll-- it'll launch a Claude, uh, a Fable session each time and burn your budget

immediately. Um, but, um, some other interesting takeaways we had were relevant to prompt caching, um, which was way higher than I thought it would be, and then some quirky subagent behavior. But one of the interesting things was is, like, after we did all the context building and had these agents dispatched to, like, write our complex legacy code, we ran benchmarks, um, against, like, using our existing code recently written as the golden data and found out that Fable or, uh, Sonnet 5.5 wrote it just as good as, if not better than Fable. So,

um, model selection for the win. Um, and then looking forward, um, I think, you know, uh, enhanced gate enforcement is gonna be the biggest win that we have, um, in my mind at least. Um, we also have some interesting updates to make with decision models within our harness, um, and within the-- our model selection harness framework. And then, um, our, our, like, t- our context was just, like, Markdown files, y'all. Like, we didn't really have an, a complex way to graph things out. Like, we weren't using anything fancy with a, um,

like, um, like GraphFi or, um, Obsidian. I mean, any of these, I mean, honestly, if you guys find me afterwards, I'd love your input on this, but, like, a more optimized way to do search and, uh, and look at all that context that we're using to inform this War Room. Um, and then lastly, um, audio as a data source. Like, I wanna turn a recorder on and have these conversations we have, like, automatically transcribed, but I had to, like, instead sit there and furiously write down all the notes that were happening during these meetings. Um, and so that was kind of, like, a distracting thing that I think, and I'm sure I

lost a bunch. Um, so a less lossy way to incorporate the, um, inputs from the War Room. And then lastly, how can we deploy this thing across the organization? Um, y'all find some group in your organization that doesn't know how to make decisions, deploy this, drive it, control the cost, make it yours. And at GEICO, we're, like, trying to figure out how, as these agents are hill climbing and figuring out better ways to do things, where can we put humans in the loop and, right? How can we, like, put our best, brightest engineers who are receiving

genius divine intervention, as it were, and have them pick the next hill as the agents climb the first one? All right, and that is it. Um, we broke free of our outer loop. Now it's your turn. Um, and that's the blog that you guys can take a picture of, um, if you'd like. All right, thanks for your time.