Making Codebases "Agent-Ready"

Talk transcript

1. 0:00

   [on-hold electronic music] Hey, everybody.

2. 0:21

   My name's Eno. Uh, really pumped to talk today about, uh, something that at Factory we care a lot about. Uh, when we started two and a half years ago, uh, we said that our mission is to bring autonomy to software engineering.

3. 0:35

   Um, and that is like got a ton of loaded words in it that sounds a little buzzwordy right now. But I think that the, the... my goal is that you guys leave this like roughly twenty minutes, uh, with a bunch of insights that will apply to your organization, uh, and the teams that you build, the companies you

4. 0:51

   advise, um, and if you're building products in the space, uh, insight into like sort of maybe how to think about building autonomous systems and also making your engineering org one that's able to use agents really successfully.

5. 1:04

   Um, a sort of like plus of this is that ideally this applies to any tools you're using that involve AI, so it won't be specific to like our product or any of the other amazing tools out there.

6. 1:14

   Um, uh, I'd like to start with a little bit about, uh, you know, Andrej Karpathy had a very well-timed tweet, uh, so of course, I'm gonna mention it. Uh, you know, he, he kind of talked about, uh, this idea of Software 2.0 coming from auto-- uh, the, the, the ability to verify things, right?

7. 1:30

   Um, this is something that's in sort of like the, the mind of Silicon Valley right now as, uh, the most frontier models are built with post-training that involve lots of like verifiable tasks.

8. 1:41

   Um, and really, I think the most interesting thing here is the sort of frontier and boundary of what can be solved by AI systems is really just a, uh, sort of an input function of whether or not you can specify an objective and search through the space of possible, uh, solutions, right?

9. 1:57

   And so, uh, we're used to building software, uh, purely via specification. We say like the algorithm does this and like input is X, output is Y. But if you sort of shift your mindset to thinking about automation via verification, uh, it is a little bit of a, of a, of a difference in what is possible to build.

10. 2:16

    Um, and there is another great blog post by, uh, uh, Jason, where he talks about the asymmetry of verification. Uh, this is like pretty intuitive to most people who know about like P versus NP.

11. 2:27

    Uh, it's like a, a thing that a lot of people have talked about throughout the like history of computing and, and software. But there are a ton of tasks that are much easier to verify than they are to solve, um, and, and vice versa.

12. 2:39

    But, but the, the most interesting sorts of, uh, easy-to-verify problems are ones where there's an objective truth. They're q- pretty quick to validate whether or not they're true. Uh, they're scalable, so validating a bunch of these things maybe in parallel, uh, is easy.

13. 2:53

    Um, it's low noise, so your chance of validating it is like really, really high. Um, and they have continuous sort of signals. Uh, it's not just like a binary yes/no, but like maybe you're thirty percent, seventy percent, a hundred percent accurate or correct.

14. 3:09

    Um, and you know, the reason I bring both these things up is software development is highly verifiable, right? This is like the frontier. It's why, uh, software development agents are the most advanced agents in the world right now.

15. 3:22

    Uh, and there i- are so much, uh, there's so much work that has been put in, uh, over the last, you know, twenty to thirty years around the automated validation and verification of software that you build.

16. 3:34

    Um, testing, right? Unit tests, end-to-end tests, QA tests, right? Um, the frontier of this is expanding. There's tons of cool companies like Browserbase and, you know, computer use agents, and all these things that are making it easier to validate, uh, really complex visual or front-end changes.

17. 3:50

    Um, docs, right? Having like an OpenAPI spec for your code base, uh, is something that can be automated, it's validated. Um, y- I, I, I can go through and enumerate a bunch of these, but I actually think it is sort of a nice checklist for yourself, right?

18. 4:03

    Do you have some automated validation for the format of your code? Uh, do you have linters? These things for professional software engineers are sort of like, "Oh, of course we do."

19. 4:13

    But I think you can go a step further, right? This is where that continuous validation component comes in. Um, do you have linters that are so opinionated that a coding agent will always make code that is exactly at the level of what your senior engineers will produce?

20. 4:28

    How do you do that? What does that even mean, right? Do you have tests that will fail when AI slop has been introduced, uh, and when high-quality AI code is introduced, those tests pass, right?

21. 4:40

    These additional layers of validators are things that most code bases actually lack because humans are pretty good at handling most of this stuff without the automated validation, right? Your company may be at some test coverage rate that's like fifty percent or sixty percent, and that's good enough because humans will test manually.

22. 4:59

    Um, you may have a flaky build that every third build it sort of fails, and everyone at your company secretly hates it, but no one says anything, right? These are the sorts of things that we know are true about large code bases.

23. 5:10

    And as you scale out to extremely large code bases, organizations with forty-four thousand plus engineers, right? Uh, this starts to become a very accepted norm that the bar is sort of maybe at fifty percent or sixty percent.

24. 5:22

    Um, and the reality is, is most software orgs can actually scale like that. Uh, it's sort of fine to be at that lower, uh, barrier. But when you start introducing AI agents into your software development lifecycle, and I don't just mean in interactive coding, but really across the board, right?

25. 5:38

    Uh, review, documentation, testing, all this stuff, um, this breaks their capabilities. Most of you have probably only seen an AI agent that operates in a code base that has, uh, a decent amount of validation.

26. 5:50

    Um, I think a lot of the best companies in the world right now actually have introduced very rigorous validation criteria, and it means that their ability to use agents is significantly greater than the-- your like average, uh, developer.

27. 6:05

    Uh, you know, and, and if you think about it, this like traditional loop of understanding a problem, designing a solution to the problem, coding it out, and then testing it, uh, sort of shifts if you have really rigorous validation.

28. 6:18

    Uh, it becomes a process of when you're using agents specifying the constraints by which you would like to be validated and what should be built, uh, generating solutions to that outcome, verifying, uh, both with your automated validation as well as with your, your own intuition, um, and then iteration, where you continue to iterate on that loop.

29. 6:38

    This move from sort of like traditional development to spec, specification-driven development is one that we're starting to see sort of bleed into all of the different tools. Different tools have spec mode.

30. 6:48

    Droids have like-- or Droid is our coding agent, have like specification mode, plan mode. Uh, there are entire IDEs that orient you around this like specification-driven flow. Um, and if you combine these two things together, this is really how you build reliable and high-quality solutions.

31. 7:06

    So if you think about it, what is like the best decision for you to make as an organization? Is it spending forty-five days comparing every single possible coding tool in the space and then determining that one tool is slightly better because it's ten percent more accurate at SWE-bench?

32. 7:21

    Or is it making changes to your organizational practices that enable all of these coding agents to succeed and then picking one that your, you know, developers like or honestly letting people choose from the tons of amazing tools out there?

33. 7:35

    And when you have these validation criteria, you can actually introduce way more complex AI workflows to your organization, right? Uh, if you cannot automatically validate whether or not a, uh, a PR is like reasonably successful or has code that won't definitely break prod, uh, you are not gonna be parallelizing several like agents at once, right?

34. 7:57

    You are not gonna be decomposing a large scale modernization project, uh, into a bunch of different subtasks. Like, that is, that is a very frontier style task to use AI for.

35. 8:08

    And if the single task execution, right? The simple, "I would like to get this done, here's exactly how I'd like it to be done, and here's how you should validate," if that does not work nearly a hundred percent of the time, you can sort of forget successfully using these other things at scale in your company.

36. 8:24

    Um, when you get into other tools like code review, right? Uh, if you want a really high quality AI-generated code review, you need documentation for your AI systems. Uh, and yes, uh, agents will get better at, you know, picking out, you know, whether or not to run lint or tests.

37. 8:41

    They will get better at finding solutions when you don't have explicit pointers. They'll get better at search. But they won't get better at just randomly creating this validation criteria out of thin air, right?

38. 8:53

    This is why we believe software developers, by the way, are gonna continue to be heavily involved in the process of building software, because your role starts to shift to curating the sort of environment and garden that your software is built from.

39. 9:06

    You're setting the constraints, you're building these automations and introducing continued opinionatedness, uh, into the, uh, into these automations. Um, and, you know, if your company doesn't have at least all of these, right?

40. 9:19

    Then that means that there's a lot of work that you can do, totally absent of a procurement cycle or buying one tool or trying out another one, uh. And so plug is that we help organizations do this, right?

41. 9:31

    Wha-- I think that it's great to have tools that allow you to, uh, go in and assess this stuff. They have ROI analytics that let you interact. Um, but I think that for most organizations, uh, there is actually like a very clear way to do this, right?

42. 9:47

    You can go and analyze where are you across those eight different pillars of like automated validation. Do you have a linter? How good is the linter? Do you have agents.md files, an open standard that almost every single coding agent supports?

43. 10:01

    Um, you can improve, uh, and systematically enhance, uh, these different validation criteria. Uh, and you can go through and say, "Well, we're seeing that coding agents are reliable enough for a senior developer to use, but our junior developers," if you have the tooling to, to tell, by the way, like which developer is using what tools, y-y-you can

44. 10:21

    ask questions like, "Maybe our junior developers are actually totally unable to use these coding agents." And you'll learn that the reason why is not because they're like more incompetent or they don't know how to use the tool, but because there's these niche practices that you don't have automated validation for, right?

45. 10:36

    And if you think about what, what is the difference between a like Google or a Meta and a, uh, a, a, a still large but like two thousand-person engineering org?

46. 10:46

    The difference is that a new grad with effectively zero context can go and ship a change to make YouTube's like boundary like slightly more round, and it won't, with some degree of confidence, take down YouTube for like a billion users, right?

47. 10:58

    And the reason that's possible is 'cause of the insane amounts of validation that have to happen on that code for it to be shipped. The big difference that we now have is we have coding agents that can go and identify exactly where these gaps are, and they can actually remediate those fixes, right?

48. 11:14

    So you can ask a coding agent, "Could you figure out where we're not being opinionated enough about our linters?" You can ask a coding agent to generate tests. We have an engineer named Alvin who...

49. 11:24

    I love this quote. He said, "A slop test is better than no test." Uh, and I think that that's slightly controversial. But the thing that I would argue here is that just having something there, right?

50. 11:34

    That it passes, uh, when changes are correct and somewhat accurately, uh, matches to the spec of what you want built, uh, people will enhance it, they'll upgrade it, and other agents will actually notice these tests.

51. 11:48

    They will follow the patterns. So the more opinionated you get, the faster the cycle continues. So I think that what you guys should be thinking about is what are the feedback loops in our organization that we are catering towards.

52. 11:59

    If you have better agents, they will make the environment better, which will make the agents better, which will mean you have more time to make the environment better. And this is sort of the new DevX loop as well that organizations can invest in, uh, that will enhance all of the tools that you're procuring, right?

53. 12:13

    So no matter whether it's a code review tool, a coding agent, et cetera, they will all benefit. Um, and I would argue that it sort of shifts your mental model about what you're, as a leader, investing in when you're investing in your software work.

54. 12:26

    Right now, the idea of, uh, you know, OpEx as like the input to engineering projects, like we are investing in, we want more people. In order to solve this problem, we need ten more people.

55. 12:36

    Um, I would, I would argue that, uh, the other thing that you can now start investing in is this environment feedback loop that enables these additional people to be significantly more successful, right?

56. 12:46

    And I think that that's the feedback loop that can actually take quite a lot of value because coding agents can just scale this out. So

57. 12:53

    You know, all of this is to say there's a lot that can be done outside of the, like, product itself, uh, to enable these systems, and the best coding agents will actually take advantage of these validation loops, right?

58. 13:05

    So if your coding agent isn't proactively seeking linters, tests, et cetera, then, you know, at the end of the day, it's not gonna be as good as one that will seek those validation criteria.

59. 13:16

    And in addition to that, when organizations, uh, uh, think about these sorts of things, if you're the person who's able to say, "Here's my opinion, here's how I want software to be built," it scales your capabilities out greater than ever before.

60. 13:30

    Like, one opinionated engineer can actually meaningfully change the velocity of the entire business if you take this to heart, uh, and you have a way to measure and systematically improve.

61. 13:41

    Um, so that's, uh, you know, the, the majority of, uh, what I came here to say. I think that the, the, the only thing that I'd leave you with, uh, is that when you think about where AI is going and, like, where we're at today, we're still really earn- e- early in our journey of using software development

62. 13:58

    agents. If you want a world where the moment a customer issue comes in, a bug is filed, that ticket is picked up, a coding agent executes on that, that feedback is presented to a developer, they click approve, that code is merged and deployed to production in a feedback loop that takes maybe an hour, two hours, that will

63. 14:21

    be possible, right? We all are sort of skeptical about that fully autonomous flow. That is technically feasible today. The limiter is not the capability of the coding agent. The limit is your organization's validation criteria.

64. 14:34

    So this is, like, an investment that made today will make your organization not 1.5X, not 2X, but that is where the real, like, 5X, 6X, 7X comes from. Um, and it's sort of a, a, a, an easy thing to say, and it's an unfortunate story because what that means is you have to invest in this.

65. 14:51

    It's not something that, like, AI will just magically give to you. Uh, it's a choice that you as an organization have. Uh, and if you make it now, I can guarantee you that you will be in the top one, five percent of organizations in terms of eng velocity, um, and you will outcompete everybody else in the field.

66. 15:07

    So highly recommend investing in this sort of stuff, and hopefully you found this helpful and have some lessons to take home. Thanks. [audience applauding] [upbeat music]