AI Engineer Code 2025
Making Codebases "Agent-Ready"
Read the talk
Making Codebases Agent-Ready
Reliable coding agents need more than a good model: they need an environment that makes engineering standards explicit, executable, and easy to verify.
From a talk by Eno Reyes
Before you start: Familiarity with pull requests, automated tests, linters, and continuous integration will help you follow the examples.
What can you verify?
What would it take to make an engineering organization capable of using autonomous software systems successfully? For Eno Reyes at Factory, bringing autonomy to software engineering requires work on the organization as well as the agent. That makes agent readiness a useful goal regardless of which coding, review, or documentation tools a team chooses.
The starting point is automation through verification. Reyes introduces it through Andrej Karpathy’s Software 2.0 framing; Karpathy’s Verifiability develops the closely related distinction. Traditionally, a programmer specifies an algorithm: given an input, perform these operations to produce an output. With a verifiable objective, a system can instead search through candidate solutions and check whether they satisfy it. Reyes connects that possibility to frontier-model post-training on tasks whose outcomes can be checked. The practical question becomes whether you can specify the objective and recognize a successful solution.
This works especially well when checking an answer is easier than discovering it. The discussion of Jason’s verification-asymmetry post matches Jason Wei’s Asymmetry of verification and verifier’s rule. The P-versus-NP analogy provides intuition about finding versus checking solutions, rather than a complexity-theoretic result about engineering work. A useful verifier has several properties:
| Property | What it provides |
|---|---|
| Objective truth | A defensible criterion for correctness |
| Fast validation | Short feedback cycles |
| Scalability | Many candidates checked in parallel |
| Low noise | Results that can be trusted |
| Graded feedback | Evidence of partial progress |
The last property matters because a candidate need not be simply right or wrong. Reyes illustrates graded correctness with 30%, 70%, and 100%; these are examples of feedback signals, not evaluation results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn engineering standards into executable checks
Software development already has a substantial verification infrastructure. Reyes attributes the progress of software agents partly to decades of investment in automated checks. Unit tests validate small pieces of behavior; end-to-end and QA tests exercise larger workflows. Browserbase and computer-use agents are examples of tools expanding what can be checked in complex visual and frontend changes.
Documentation and code quality provide additional checks. An OpenAPI specification makes an HTTP API’s interface explicit and gives tooling a description it can validate, although validating that description does not prove the implementation behaves correctly. Formatters and linters similarly turn some conventions into repeatable decisions.
The more demanding question is whether those checks express the team’s actual expectations. A linter that catches syntax problems may still accept code a senior engineer would reject. A test suite may pass while an agent introduces an unwanted implementation pattern. Agent readiness requires making more of those quality judgments executable. Reyes proposes a higher bar: checks that reject unwanted generated code and accept code meeting the team’s standards. This is a direction for improving validators, not a guarantee that linting can capture every senior engineer’s judgment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The gaps humans tolerate become agent constraints
People often compensate for incomplete automation without making that work visible. Reyes uses 50–60% test coverage, supplemented by manual testing, as an illustrative level an organization might tolerate. He also imagines a flaky build that fails every third run. An experienced developer may know when to retry, what to inspect manually, and which apparent success still deserves scrutiny. Those judgments let work continue even when the automated feedback is incomplete or noisy.
Reyes invokes organizations with more than 44,000 engineers to illustrate how tolerated validation gaps can persist at scale; he does not identify a measured organization. Such gaps need not prevent a human organization from functioning, but they constrain what agents can do across coding, review, documentation, and testing. Stronger validation gives agents more dependable evidence about whether their work is acceptable and gives the organization more confidence in using them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specify, generate, verify, iterate
Rigorous validation changes how a task can be organized. The familiar sequence is to understand the problem, design a solution, write code, and test it. With agents, the work becomes an explicit loop:
- Specify the desired behavior and the constraints a solution must satisfy.
- Generate a candidate implementation.
- Verify it using automated checks and human judgment.
- Iterate on the implementation or clarify the specification using what the checks reveal.
Human intuition remains part of verification. The shift is to make the acceptance conditions available before, and throughout, generation.
Specification and planning modes in coding tools support this workflow. Reyes points to Factory’s Droid and its Specification Mode, alongside other tools and IDEs organized around planning before implementation. These interfaces are useful when they help connect what should be built with how its correctness will be assessed.
That changes the procurement question. Reyes contrasts spending 45 days comparing tools for a hypothetical 10% advantage on SWE-bench with improving the practices that help every coding agent succeed. Those figures are rhetorical examples, not a reported tool evaluation. Shared validation improvements can benefit multiple tools, leaving teams room to choose tools their developers prefer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reliable single tasks come before parallel agents
Parallel agents make verification more consequential. Before assigning several agents work at once, an organization needs a way to assess whether each pull request achieves its purpose and is unlikely to break production. A large modernization project adds another requirement: decomposing the project into subtasks that can be executed and assessed separately. If the organization cannot reliably evaluate one change, generating many changes increases the unresolved review burden.
The basic unit is a task with a clear outcome, an intended approach, and a validation method. Reyes proposes nearly 100% success on such single tasks as a prerequisite for scaling to more complex agent workflows, not as an observed benchmark result. Decomposition and parallelism depend on that foundation.
Code review illustrates the same dependency from another direction. A review agent needs documentation of what the team expects. Agents may improve at finding tests, discovering lint commands, searching the repository, and working without explicit pointers. But finding an existing rule is different from supplying a rule the organization has never articulated. Better search cannot reliably recover missing acceptance criteria.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make implicit engineering knowledge available
Developers therefore remain responsible for shaping the environment in which software is built. They set constraints, build automations, and progressively encode more of the organization’s engineering preferences. Much of this work can begin without buying another tool. Reyes mentions Factory’s assistance, assessment tools, and ROI analytics, but the underlying questions are direct: which checks exist, and how much useful judgment do they contain?
The readiness assessment returns to the talk’s eight verification pillars. Begin with concrete questions: is there a linter, and how strong are its rules? Are there instructions for agents? The broadly supported AGENTS.md format provides a place to make project guidance explicit. Merely having a file or a command is not the endpoint; the assessment concerns whether the agent can discover and use the practices that matter.
Differences between developers can reveal what is missing. If tool-usage visibility shows that senior developers get good results from an agent while junior developers struggle, investigate the practices experienced developers supply implicitly. A junior developer’s difficulty may expose a niche convention or missing validation rule, rather than a lack of skill with the tool. Systematically improving those checks makes the knowledge available beyond the people who already possess it.
Reyes illustrates the protective role of validation by contrasting Google or Meta with a still-large organization of roughly 2,000 engineers. In his example, a new graduate with little context can make YouTube’s boundary slightly rounder without taking the service down for roughly a billion users. The point is the infrastructure surrounding a small change: extensive checks can protect deployment even when its author lacks deep organizational context. Coding agents can now help identify gaps in that infrastructure and propose fixes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use agents to strengthen their own working environment
An agent can inspect lint rules for places where the repository is insufficiently opinionated. It can also generate tests. Reyes quotes Factory engineer Alvin: “A slop test is better than no test.” His defense of that deliberately provocative advice has a condition: the initial test should pass when changes are correct and reasonably match the intended specification. It gives people something concrete to improve and future agents a pattern to follow.
For example, suppose a TypeScript project requires account names to be trimmed and lowercased, with empty names rejected. A small test can turn that previously informal rule into an explicit acceptance condition:
typescript
import assert from "node:assert/strict";
import test from "node:test";
import { normalizeAccountName } from "./account-name.js";
test("account names follow the project convention", () => {
assert.equal(normalizeAccountName(" ALICE "), "alice");
assert.equal(normalizeAccountName("alice"), "alice");
assert.throws(() => normalizeAccountName(" "));
});
This initial test does not settle every naming policy. It makes a few intended behaviors inspectable, so a developer or agent can extend the specification and its checks together. Existing tests also communicate how the repository expects new tests to be written.
That creates a reinforcing developer-experience loop. Better agents can improve the environment; the improved environment helps agents produce better work; the time saved can fund further improvements. The benefit extends across coding agents and review tools because they draw on the same repository standards and validation infrastructure.
The investment decision consequently expands beyond staffing. A request for ten more people is Reyes’s example of the familiar input to an engineering project. Investing in the environment offers another way to increase capacity: make the existing people, additional hires, and agents more effective at producing changes that can be trusted. Because agents can repeatedly apply those improvements, a better check can influence many subsequent tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From available checks to approved production changes
The agent still has to use the environment. A practical selection criterion is whether it proactively looks for linters, tests, and other validation criteria. Infrastructure outside the agent product matters only to the extent that the workflow takes advantage of it. Once an engineer’s preferences are expressed in reusable checks, an agent can apply them repeatedly; measurement and systematic improvement can extend that engineer’s influence across the organization.
The closing example makes the intended destination concrete:
- A customer reports an issue.
- A bug ticket is filed and picked up by a coding agent.
- The agent implements a change and presents feedback to a developer.
- The developer reviews the result and approves it.
- The code is merged and deployed to production.
Reyes envisions this issue-to-production loop taking one to two hours; he does not report a measured deployment result. Although he describes an autonomous flow, the example explicitly retains human approval.
Reyes argues that this workflow was technically feasible at the time of the talk and that organizational validation, rather than agent capability, was the limiting factor. His prediction is that investment in these foundations could produce 5×–7× gains rather than 1.5×–2× gains, and put adopters in the top 1–5% for engineering velocity. These are forecasts, not measured productivity results. The organizational commitment is the substantive requirement: AI will not automatically supply missing engineering standards. Teams have to choose which outcomes they will accept, encode the checks that support those choices, and invest in making the resulting feedback dependable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Jason Wei explains why checking solutions can be easier than finding them and identifies five useful properties of verification.
An open Markdown format for documenting project setup, test commands, and conventions for coding agents.
Current examples for browser tests with Stagehand or Playwright, including isolated sessions and failure debugging.
The formal specification for describing HTTP API operations, parameters, schemas, and responses.
Documentation for reviewing and refining a Droid implementation plan before proceeding with changes.
The original benchmark paper evaluates repository changes against real GitHub issue-resolution tasks.
Further reading
- VerifiabilityArticle
Karpathy explains how verifiable objectives and repeatable practice enable automation through neural networks.
Updates since the talk
- Introducing Agent ReadinessArticle
Factory's January 2026 announcement presents an eight-pillar readiness framework and repository assessment tooling.
Current guidance for assessing and improving repositories using five readiness levels and nine technical pillars.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music] Hey, everybody.
- 0:21
My name's Eno. Uh, really pumped to talk today about, uh, something that at Factory we care a lot about. Uh, when we started two and a half years ago, uh, we said that our mission is to bring autonomy to software engineering.
- 0:35
Um, and that is like got a ton of loaded words in it that sounds a little buzzwordy right now. But I think that the, the... my goal is that you guys leave this like roughly twenty minutes, uh, with a bunch of insights that will apply to your organization, uh, and the teams that you build, the companies you
- 0:51
advise, um, and if you're building products in the space, uh, insight into like sort of maybe how to think about building autonomous systems and also making your engineering org one that's able to use agents really successfully.
- 1:04
Um, a sort of like plus of this is that ideally this applies to any tools you're using that involve AI, so it won't be specific to like our product or any of the other amazing tools out there.
- 1:14
Um, uh, I'd like to start with a little bit about, uh, you know, Andrej Karpathy had a very well-timed tweet, uh, so of course, I'm gonna mention it. Uh, you know, he, he kind of talked about, uh, this idea of Software 2.0 coming from auto-- uh, the, the, the ability to verify things, right?
- 1:30
Um, this is something that's in sort of like the, the mind of Silicon Valley right now as, uh, the most frontier models are built with post-training that involve lots of like verifiable tasks.
- 1:41
Um, and really, I think the most interesting thing here is the sort of frontier and boundary of what can be solved by AI systems is really just a, uh, sort of an input function of whether or not you can specify an objective and search through the space of possible, uh, solutions, right?
- 1:57
And so, uh, we're used to building software, uh, purely via specification. We say like the algorithm does this and like input is X, output is Y. But if you sort of shift your mindset to thinking about automation via verification, uh, it is a little bit of a, of a, of a difference in what is possible to build.
- 2:16
Um, and there is another great blog post by, uh, uh, Jason, where he talks about the asymmetry of verification. Uh, this is like pretty intuitive to most people who know about like P versus NP.
- 2:27
Uh, it's like a, a thing that a lot of people have talked about throughout the like history of computing and, and software. But there are a ton of tasks that are much easier to verify than they are to solve, um, and, and vice versa.
- 2:39
But, but the, the most interesting sorts of, uh, easy-to-verify problems are ones where there's an objective truth. They're q- pretty quick to validate whether or not they're true. Uh, they're scalable, so validating a bunch of these things maybe in parallel, uh, is easy.
- 2:53
Um, it's low noise, so your chance of validating it is like really, really high. Um, and they have continuous sort of signals. Uh, it's not just like a binary yes/no, but like maybe you're thirty percent, seventy percent, a hundred percent accurate or correct.
- 3:09
Um, and you know, the reason I bring both these things up is software development is highly verifiable, right? This is like the frontier. It's why, uh, software development agents are the most advanced agents in the world right now.
- 3:22
Uh, and there i- are so much, uh, there's so much work that has been put in, uh, over the last, you know, twenty to thirty years around the automated validation and verification of software that you build.
- 3:34
Um, testing, right? Unit tests, end-to-end tests, QA tests, right? Um, the frontier of this is expanding. There's tons of cool companies like Browserbase and, you know, computer use agents, and all these things that are making it easier to validate, uh, really complex visual or front-end changes.
- 3:50
Um, docs, right? Having like an OpenAPI spec for your code base, uh, is something that can be automated, it's validated. Um, y- I, I, I can go through and enumerate a bunch of these, but I actually think it is sort of a nice checklist for yourself, right?
- 4:03
Do you have some automated validation for the format of your code? Uh, do you have linters? These things for professional software engineers are sort of like, "Oh, of course we do."
- 4:13
But I think you can go a step further, right? This is where that continuous validation component comes in. Um, do you have linters that are so opinionated that a coding agent will always make code that is exactly at the level of what your senior engineers will produce?
- 4:28
How do you do that? What does that even mean, right? Do you have tests that will fail when AI slop has been introduced, uh, and when high-quality AI code is introduced, those tests pass, right?
- 4:40
These additional layers of validators are things that most code bases actually lack because humans are pretty good at handling most of this stuff without the automated validation, right? Your company may be at some test coverage rate that's like fifty percent or sixty percent, and that's good enough because humans will test manually.
- 4:59
Um, you may have a flaky build that every third build it sort of fails, and everyone at your company secretly hates it, but no one says anything, right? These are the sorts of things that we know are true about large code bases.
- 5:10
And as you scale out to extremely large code bases, organizations with forty-four thousand plus engineers, right? Uh, this starts to become a very accepted norm that the bar is sort of maybe at fifty percent or sixty percent.
- 5:22
Um, and the reality is, is most software orgs can actually scale like that. Uh, it's sort of fine to be at that lower, uh, barrier. But when you start introducing AI agents into your software development lifecycle, and I don't just mean in interactive coding, but really across the board, right?
- 5:38
Uh, review, documentation, testing, all this stuff, um, this breaks their capabilities. Most of you have probably only seen an AI agent that operates in a code base that has, uh, a decent amount of validation.
- 5:50
Um, I think a lot of the best companies in the world right now actually have introduced very rigorous validation criteria, and it means that their ability to use agents is significantly greater than the-- your like average, uh, developer.
- 6:05
Uh, you know, and, and if you think about it, this like traditional loop of understanding a problem, designing a solution to the problem, coding it out, and then testing it, uh, sort of shifts if you have really rigorous validation.
- 6:18
Uh, it becomes a process of when you're using agents specifying the constraints by which you would like to be validated and what should be built, uh, generating solutions to that outcome, verifying, uh, both with your automated validation as well as with your, your own intuition, um, and then iteration, where you continue to iterate on that loop.
- 6:38
This move from sort of like traditional development to spec, specification-driven development is one that we're starting to see sort of bleed into all of the different tools. Different tools have spec mode.
- 6:48
Droids have like-- or Droid is our coding agent, have like specification mode, plan mode. Uh, there are entire IDEs that orient you around this like specification-driven flow. Um, and if you combine these two things together, this is really how you build reliable and high-quality solutions.
- 7:06
So if you think about it, what is like the best decision for you to make as an organization? Is it spending forty-five days comparing every single possible coding tool in the space and then determining that one tool is slightly better because it's ten percent more accurate at SWE-bench?
- 7:21
Or is it making changes to your organizational practices that enable all of these coding agents to succeed and then picking one that your, you know, developers like or honestly letting people choose from the tons of amazing tools out there?
- 7:35
And when you have these validation criteria, you can actually introduce way more complex AI workflows to your organization, right? Uh, if you cannot automatically validate whether or not a, uh, a PR is like reasonably successful or has code that won't definitely break prod, uh, you are not gonna be parallelizing several like agents at once, right?
- 7:57
You are not gonna be decomposing a large scale modernization project, uh, into a bunch of different subtasks. Like, that is, that is a very frontier style task to use AI for.
- 8:08
And if the single task execution, right? The simple, "I would like to get this done, here's exactly how I'd like it to be done, and here's how you should validate," if that does not work nearly a hundred percent of the time, you can sort of forget successfully using these other things at scale in your company.
- 8:24
Um, when you get into other tools like code review, right? Uh, if you want a really high quality AI-generated code review, you need documentation for your AI systems. Uh, and yes, uh, agents will get better at, you know, picking out, you know, whether or not to run lint or tests.
- 8:41
They will get better at finding solutions when you don't have explicit pointers. They'll get better at search. But they won't get better at just randomly creating this validation criteria out of thin air, right?
- 8:53
This is why we believe software developers, by the way, are gonna continue to be heavily involved in the process of building software, because your role starts to shift to curating the sort of environment and garden that your software is built from.
- 9:06
You're setting the constraints, you're building these automations and introducing continued opinionatedness, uh, into the, uh, into these automations. Um, and, you know, if your company doesn't have at least all of these, right?
- 9:19
Then that means that there's a lot of work that you can do, totally absent of a procurement cycle or buying one tool or trying out another one, uh. And so plug is that we help organizations do this, right?
- 9:31
Wha-- I think that it's great to have tools that allow you to, uh, go in and assess this stuff. They have ROI analytics that let you interact. Um, but I think that for most organizations, uh, there is actually like a very clear way to do this, right?
- 9:47
You can go and analyze where are you across those eight different pillars of like automated validation. Do you have a linter? How good is the linter? Do you have agents.md files, an open standard that almost every single coding agent supports?
- 10:01
Um, you can improve, uh, and systematically enhance, uh, these different validation criteria. Uh, and you can go through and say, "Well, we're seeing that coding agents are reliable enough for a senior developer to use, but our junior developers," if you have the tooling to, to tell, by the way, like which developer is using what tools, y-y-you can
- 10:21
ask questions like, "Maybe our junior developers are actually totally unable to use these coding agents." And you'll learn that the reason why is not because they're like more incompetent or they don't know how to use the tool, but because there's these niche practices that you don't have automated validation for, right?
- 10:36
And if you think about what, what is the difference between a like Google or a Meta and a, uh, a, a, a still large but like two thousand-person engineering org?
- 10:46
The difference is that a new grad with effectively zero context can go and ship a change to make YouTube's like boundary like slightly more round, and it won't, with some degree of confidence, take down YouTube for like a billion users, right?
- 10:58
And the reason that's possible is 'cause of the insane amounts of validation that have to happen on that code for it to be shipped. The big difference that we now have is we have coding agents that can go and identify exactly where these gaps are, and they can actually remediate those fixes, right?
- 11:14
So you can ask a coding agent, "Could you figure out where we're not being opinionated enough about our linters?" You can ask a coding agent to generate tests. We have an engineer named Alvin who...
- 11:24
I love this quote. He said, "A slop test is better than no test." Uh, and I think that that's slightly controversial. But the thing that I would argue here is that just having something there, right?
- 11:34
That it passes, uh, when changes are correct and somewhat accurately, uh, matches to the spec of what you want built, uh, people will enhance it, they'll upgrade it, and other agents will actually notice these tests.
- 11:48
They will follow the patterns. So the more opinionated you get, the faster the cycle continues. So I think that what you guys should be thinking about is what are the feedback loops in our organization that we are catering towards.
- 11:59
If you have better agents, they will make the environment better, which will make the agents better, which will mean you have more time to make the environment better. And this is sort of the new DevX loop as well that organizations can invest in, uh, that will enhance all of the tools that you're procuring, right?
- 12:13
So no matter whether it's a code review tool, a coding agent, et cetera, they will all benefit. Um, and I would argue that it sort of shifts your mental model about what you're, as a leader, investing in when you're investing in your software work.
- 12:26
Right now, the idea of, uh, you know, OpEx as like the input to engineering projects, like we are investing in, we want more people. In order to solve this problem, we need ten more people.
- 12:36
Um, I would, I would argue that, uh, the other thing that you can now start investing in is this environment feedback loop that enables these additional people to be significantly more successful, right?
- 12:46
And I think that that's the feedback loop that can actually take quite a lot of value because coding agents can just scale this out. So
- 12:53
You know, all of this is to say there's a lot that can be done outside of the, like, product itself, uh, to enable these systems, and the best coding agents will actually take advantage of these validation loops, right?
- 13:05
So if your coding agent isn't proactively seeking linters, tests, et cetera, then, you know, at the end of the day, it's not gonna be as good as one that will seek those validation criteria.
- 13:16
And in addition to that, when organizations, uh, uh, think about these sorts of things, if you're the person who's able to say, "Here's my opinion, here's how I want software to be built," it scales your capabilities out greater than ever before.
- 13:30
Like, one opinionated engineer can actually meaningfully change the velocity of the entire business if you take this to heart, uh, and you have a way to measure and systematically improve.
- 13:41
Um, so that's, uh, you know, the, the majority of, uh, what I came here to say. I think that the, the, the only thing that I'd leave you with, uh, is that when you think about where AI is going and, like, where we're at today, we're still really earn- e- early in our journey of using software development
- 13:58
agents. If you want a world where the moment a customer issue comes in, a bug is filed, that ticket is picked up, a coding agent executes on that, that feedback is presented to a developer, they click approve, that code is merged and deployed to production in a feedback loop that takes maybe an hour, two hours, that will
- 14:21
be possible, right? We all are sort of skeptical about that fully autonomous flow. That is technically feasible today. The limiter is not the capability of the coding agent. The limit is your organization's validation criteria.
- 14:34
So this is, like, an investment that made today will make your organization not 1.5X, not 2X, but that is where the real, like, 5X, 6X, 7X comes from. Um, and it's sort of a, a, a, an easy thing to say, and it's an unfortunate story because what that means is you have to invest in this.
- 14:51
It's not something that, like, AI will just magically give to you. Uh, it's a choice that you as an organization have. Uh, and if you make it now, I can guarantee you that you will be in the top one, five percent of organizations in terms of eng velocity, um, and you will outcompete everybody else in the field.
- 15:07
So highly recommend investing in this sort of stuff, and hopefully you found this helpful and have some lessons to take home. Thanks. [audience applauding] [upbeat music]