AI Engineer Code 2025
Fixing the PR Bottleneck — Matt Pocock, AIHero
Read the talk
Fixing the PR Bottleneck
Agents can produce pull requests faster than people can review them. Matt Pocock explains how better tests, a separate standards reviewer, risk-aware PR descriptions and retrospectives can reduce the work that reaches a human.
From a talk by Matt Pocock
At a glance
Ideas worth remembering
Inspect what tests prove: constant assertions, source-text ordering checks and overly simple mocks can pass while leaving important behavior untested.
Deep modules give agents a small interface through which to test substantial behavior, reducing dependence on implementation details.
A separate review context can apply repository-specific standards and commit fixes before a human receives the PR.
Allocate review by reversibility and blast radius; a small diff can still trigger irreversible consequences.
Use retrospectives to turn recurring findings into better checks, standards, navigation and tool choices.
A software factory needs brakes
Pull requests were piling up before coding agents arrived. Agents increase the pressure by making more work possible without increasing the attention available to review it. In this AI Engineer Paris talk, Matt Pocock approaches the bottleneck through the processes around a PR: improve the artifact before it reaches a person, then make the remaining decisions easier.
The software factory adds another source of acceleration: work can start without a human requesting each change. A classifier might turn an incoming item into a fix, a reproduction or a notification. A PlanetScale report about slow database queries might trigger another workflow. These are examples of automated initiation, with deterministic code pushing work into the factory.
Permanent acceleration produces what Pocock calls a “slop cannon”: more poor PRs than anyone can reasonably inspect. There is a second cost beyond the review queue. Code is the environment the next agent works in; confusing code gives future changes a worse starting point. The brakes therefore protect both today's reviewer and tomorrow's implementation.
Three layers carry different jobs:
- Automated checks: Linting, tests, type checking and code-quality metrics enforce repeatable conditions.
- Automated review: An agent examines problems the checks missed, including the structure of the codebase.
- Human review: A person inspects the resulting PR and makes the remaining judgments.
The intended speedup comes from reducing interventions at the final layer. More code reaching human review is only useful if less of it needs repair.
Deterministic checks are an attractive first investment because running them primarily costs CPU cycles, rather than model tokens or reviewer attention. When a check catches an agent's bug, fixing it does consume tokens—but those tokens buy a concrete correction. That makes frequent checks inexpensive relative to discovering the same problem during human review.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Green CI can certify the wrong thing
A green CI run establishes that the checks passed. It does not establish that the change is ready to merge. Pocock calls automated and human review “lie detectors” because they must also question what the checks actually prove. Three examples show how a test can look reassuring while missing the behavior that matters.
Reasserting a constant. The implementation sets an X post character limit to 280; the test expects that constant to equal 280. Both contain the same answer. The assertion checks the stored value without exercising how the application enforces the limit. It also couples the test to an internal identifier: renaming the constant requires changing the test even if the user-visible behavior stays the same. This is the tautological-test problem—an assertion can preserve the shape of the implementation while contributing little independent evidence about its behavior.
Reading source order instead of UI order. Another test is meant to verify that a pitch detail page places its video section after its content plan. Rather than rendering the page, it reads the module as text, finds the content-plan and video references, and compares their positions. The observed result is the order of strings in a file. A source refactor can therefore break the test without changing the rendered page. To understand why this test is fragile, the useful distinction to inspect is the file-position comparison versus the intended screen-position check.
typescript
describe("pitch detail page section order", () => {
it("renders Videos section after Content Plan", () => {
const content = fs.readFileSync(PITCH_DETAIL_PATH, "utf-8");
const contentPlanIndex = content.indexOf(">Content Plan<");
const videosIndex = content.indexOf('title="Videos"');
expect(contentPlanIndex).toBeGreaterThan(-1);
expect(videosIndex).toBeGreaterThan(-1);
expect(videosIndex).toBeGreaterThan(contentPlanIndex);
});
});
Mocking away failure. The useAudioBoost function uses the browser's AudioContext API. Its test substitutes fake methods for that API, but real AudioContext behavior includes error modes under particular conditions. The stub cannot produce those failures, so the test cannot detect them. The specific browser conditions are not identified in the talk; the supported failure is narrower and still consequential: replacing a dependency with an always-successful imitation removes its error paths from the test.
These failures do not require an agent to cheat deliberately. An agent can follow an instruction to write tests and produce assertions that are easy to satisfy because they inspect implementation details or omit difficult behavior. The design question becomes: how can the repository make meaningful checks easier to write, and superficial checks harder to substitute?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give tests a small interface with substantial behavior behind it
Codebase design can change the tests an agent naturally writes. Pocock uses John Ousterhout's idea of deep modules from A Philosophy of Software Design: a small interface hides a substantial implementation. The contrasting shallow module exposes many functions that individually do little. With more internal details available to callers, there are more opportunities for tests to depend on those details.
What makes a deep module useful for testing? The comparison below separates the interface a test should call from the implementation it should leave hidden. Tests written through the small interface can exercise substantial behavior while allowing the internals to change. That benefit depends on actually directing the agent to use the interface; merely creating one does not prevent tests from reaching inside the module.
Pocock's module-deepening skill proposes opportunities rather than automatically declaring a codebase fixed. It produces an HTML report with before-and-after designs, including a demonstrated proposal to reduce duplication and create a deeper, testable module. The next action is to implement the proposal. The useful question for that report is what moves behind the interface and what duplicated work disappears—not simply whether the resulting diagram looks tidier.
A companion codebase-design skill gives the team shared terms. Locality concerns how related code is located together and how changes travel through it. Leverage describes how much useful behavior a caller gets from a simple function call. The skill also names seams. Shared vocabulary gives reviewers and agents a consistent way to describe design changes, instead of relying on competing architectural labels.
What does the test have to know about the module?
Read the diagram as text
- Deep module A. Deep — Small interface; large hidden implementation. Tests use the interface rather than reaching into implementation details.
- Shallow module B. Shallow — Large interface; many callable functions that individually do little. More structure is exposed to callers and tests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move standards into a separate review context
Good design principles still need enforcement. Pocock's counterintuitive advice is to keep the detailed coding standards out of the implementer agent's context. That agent already has three substantial jobs: find the relevant code, make the change, and debug it by running checks. Adding a large standards burden asks the same context window to handle another kind of work.
The code-review skill instead receives the diff and reads a repository standards file. It runs in a subagent with its own context window and budget. The diff narrows the search, although the reviewer still explores surrounding code to understand the change. Its primary task is assessing standards, without carrying the original implementation and debugging workload.
This splits the work into two passes: make it work, then make it good. Pocock relates it to red-green-refactor, with implementation and refactoring handled in separate contexts. He reports that the split has worked well for him; the talk does not supply a measured comparison of review quality or context usage. The practical mechanism is specialization: fewer competing responsibilities leave the reviewer more room to apply detailed standards.
The file placement expresses that division. Put the standards in codingstandards.md, read by the review agent, rather than placing all of them in agents.md where they enter the implementer's general instructions. The advice is about where detailed standards receive attention: a separate review pass gives them a defined job instead of letting them compete with the task of getting the change working.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A reviewer should reduce work, not generate comments
A generic automated reviewer faces an awkward tradeoff. Broad rules can produce false positives that do not fit the repository; narrow rules can become specific to one language or environment. A TypeScript-focused reviewer, for example, is less useful to a Rust team. Pocock's experience trying to build a generic bug and security reviewer leads him to recommend developing the team's own automated review and standards over time.
Existing team documentation is useful input. Rules that sit unread in documents can become criteria the reviewer applies to every change. Shared standards also let the team refine the reviewer around its actual code, rather than accepting irrelevant findings as the unavoidable cost of automation.
The reviewer's output matters as much as its criteria. A stream of verbose PR comments leaves the human to read each suggestion, decide whether it is valid, and arrange the fix. Pocock's preferred default is commits: the reviewer changes the problems it can resolve, so the human receives an improved artifact. Questions and uncertain findings can remain comments. This turns review capacity into corrections rather than another queue of decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make merge danger and the change itself easy to see
Once checks and automated review have improved the code, the PR still needs to help a human understand it. Pocock introduces a PR skill that was in progress at the time of the recording. Its first principle is that reviews deserve different levels of attention. The distinction starts with whether merging is a one-way or two-way door.
A two-way door is a change whose consequences can easily be reversed. A one-way door has consequences that reverting code cannot undo. A small change that sends an email to 60,000 people is the memorable example: reverting the commit does not unsend the email. Expensive migrations and data loss also demand careful review. The size of the diff is therefore a poor substitute for understanding what execution will do.
Reversibility needs a companion question: what is the blast radius? A “merge danger” summary at the bottom of the PR records both the door classification and how far a failure could spread. A reversible change with localized effects can receive lighter attention. The summary makes the reason inspectable, rather than asking a reviewer to infer risk from a reassuring description.
The next question is what actually changes. Pseudocode and diagrams can make that answer faster to grasp. Pocock credits the Show Me skill from the HumanLayer skills repository, which uses images and diagrams to explain changes. His CLI example shows an added command and two added flags. A useful visual answers where those additions fit in the command interface; its exact command and flag names are not given in the transcript. Sequence diagrams serve a different question: what happens, and in what order?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn recurring review comments into changes to the system
Human review also exposes weaknesses in the process that produced the code. Teams share skill files and steering instructions, so a recurring mistake can have a shared cause. Pocock's target is simple: never write the same review comment twice. Fixing one PR is valuable; changing the check or instruction that would catch the same problem next time makes the intervention compound.
The Retro skill supplies the mechanism. Its input can be one agent session, a PR together with its session, or a week's PRs and reviews. It examines that work and suggests automated checks and coding standards for subsequent changes. The session matters because it contains how the agent reached its result, giving the retrospective more to inspect than the final diff alone.
Retro examines several distinct sources of wasted effort:
- Missing checks and standards: Propose automated checks or updates to
codingstandards.mdso later changes encounter the lesson earlier. - Navigation pointers: Examine how easily the agent found information, and whether a pointer in
agents.mdwould improve its next search. - Tool economy: Inspect whether different tools could perform the session's work with fewer tokens.
- Instruction bloat: Look for oversized steering files or skills that contribute to poor results, and opportunities to organize them better.
These are separate interventions. A missing test, an expensive tool choice and a hard-to-find document need different repairs.
The ending keeps human attention tied to consequences. Pocock's recommendation is that every one-way door receive review, while every two-way door need not. Stronger checks, a reviewer that fixes known problems, and a PR that explains risk make that allocation more practical. Retro then gives the team a way to use the attention it did spend to improve the next change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:01
All right. I'm here to talk about Fixing the PR Bottleneck, and this is kind of a grand title for trying to fix the thing that most organizations struggle with, I think, and have kind of historically struggled with before AI. You know, we have always had huge numbers of PRs just laying around that no one's bothered to review, and this has now increased massively because of the new strains on us, because of AI imposing all these weird constraints. And to do this, I'm gonna use the
- 0:31
rubric of my skills, which is-- you've kind of heard about, maybe you've used them. And I have a couple of new skills to announce that are going to hopefully improve the way that you do PRs, improve the way that or improve the speed at which you can review them and do them. Speed. Now, we're being pushed to do more with less, essentially, or more PRs, more work, more stuff. And this is kind of the central promise of AI, that we're going to be able to use these agents to
- 1:01
scale ourselves up to do more work. And this has resulted in the software factory, probably the biggest buzzword of the day. Everyone's talking about software factory that I've chat to. And I think of a software factory as primarily something where instead of the human initiating all of this work, we're gonna pass some of that initiation. Some of the initiation is gonna be done by agents. And I think of there
- 1:50
Maybe from there you have a classifier like Jev come in and say, "Okay, let's turn that into a fix or turn that into a reproduction," or maybe I ping people straight away. And maybe you have other things. Maybe you have PlanetScale hooked up, so it gives you query reports on the slow queries on your database. Maybe that then triggers a different, um, thing of your software factory. All of this is not humans triggering it, it's, uh, deterministic code triggering it, right? And so these accelerate your software factory. They push more code through it. But then you need brakes,
- 2:20
right? If you just have permanent acceleration pushing stuff through your factory, you're gonna end up with a slop cannon, right? You're just gonna end up with a ton of slop crappy PRs that you're not gonna be able to touch or review or even freaking look at. So you need brakes. These are mechanisms that slow down, that increase quality, that make sure that your code base doesn't turn into a software entropy nightmare, because code is the environment your agent operates in. And if you have bad code in your code
- 2:50
base, that is going to beget more bad code. And so I'm gonna talk about these three brakes in this talk, and talk about how we can use them to counterintuitively go faster. So automated checks. These are the deterministic checks in your repo that we've had for thousands of years or, you know, since the '50s. Deterministic checks where we have linting and tests and type checking and code quality metrics, all of these things going together, and they all work the same every time.
- 3:21
Layered on top of that, we have automated review. So we have agents who look at our code and say, "Okay, you know, these are for the things that the tests didn't catch," or, "This is looking at the structure of the code base in general." And then on top of that, the third layer, the final layer is human review. So people looking at the PR. And these three layers form this kind of cake that we end up with when we get to human review. And so more speed of course means more PRs, and so the goal here is to make human review
- 3:51
faster by leaning on those first two phases. And we've got to stop the slop. That's the first principle here, which is if you raise the quality of the code that you're shipping, you're gonna end up doing less human review because it's just gonna be better work, and so you're gonna end up needing to, uh, make fewer interventions. So automated checks. Now, automated checks are cheap. That's the cool thing about them, is that they don't cost tokens like automated review does. They don't cost human effort, they just
- 4:21
cost CPU cycles. So these are, for instance, you know, you run your tests on every code change. Maybe those tests do incur some tokens because let's say, um, an agent, uh, creates a bug and the tests catch it, then you need to spend some tokens to go and fix it. But those are tokens pretty well spent in my opinion. So checks are cheap. That means you can layer on loads and loads and loads of them on your repos, and you're probably not using enough of them or not being creative enough with your use. But checks can
- 4:50
lie.
- 4:53
Does a green CI mean that the code is ready for merge? No, it does not. And so we've always needed on top of these checks some extra layer to figure out if there's anything catastrophically wrong with the code before we ship it. And so all of the other phases, the human review and automated review, these are lie detectors. These are for finding lies in the automated checks. Now, I wanna show you some of these lies first of all, because this helps when we're thinking about code and thinking about automated checks to see how bad it is and how
- 5:23
bad things can get. The first is tautological tests, a test that just reasserts the implementation. Opus 5 got addicted to these. I don't quite understand why. It would have, for instance, x post character limit equals two hundred and eighty. Can anyone guess the test that was written to test this behavior, right? You've probably seen this a thousand times. This is real code from agents or from stuff that I found my agents doing. It said expect x post character limit to be two hundred and eighty. So the implementation
- 5:52
looked like that, and the test essentially reasserted the implementation. That is a tautological test. And tautological tests are bad because they're extremely structure sensitive. They're very sensitive to the actual internal workings of the system. So it means I cannot change that constant without a test failing. But I cannot rename that constant without a test failing. Like I have to do-- it's so tied into the structure of my system. And I found another
- 6:22
one which is even more egregious, I would say. This is incredible, uh, test. What it's doing here is it's essentially testing whether two things in the UI appear in the right order. So it's checking that the pitch detail page, uh, or rather the video section comes after the content plan. What it does is it doesn't render it to a screen, it just reads the actual file, the module into its own memory, and then it's-- finds the right thing, so
- 6:52
finds content plan, finds videos, and then it expects the videos to be after it in the source material. Which is crazy if you think about it, 'cause I can just m-- like change the way the source material looks and this test will fail. It's too sensitive to the structure of my code base. So that's another way that automated checks can fail. And there are also-- or sorry, automated checks can lie. There are also tests that literally cannot fail, and this will feel familiar to you if you've abused mocking in the past or
- 7:22
abused various things. For instance, here we have a useAudioBoost function, and this useAudioBoost function, uh, its internals use the AudioContext API in the DOM. Don't worry if you don't know any of this. But what we're doing here is we're just stubbing it out with some fake methods. And it turns out that AudioContext has some complicated error modes, and it will fail if you use it under strange conditions. And so just doing this means our tests cannot fail using those modes, and we're gonna hit strange errors in
- 7:52
production that our tests can't fix. And so the question is then, if you can cheat on automated checks, if, you know, and even in good faith ways as well. The, the AI isn't trying to write bad tests here, it's just taking our instructions and writing tests that are too tied into the structure instead of actually executing code. So how do we make automated checks harder to cheat? If we can do that, then we can increase the quality of those checks, which means which
- 8:22
increases our quality bar. And the first thing I really like about this is codebase design. So you can actually design your way out of these bad checks. So what does good codebase design look like? I've talked about this before in previous AI engineer talks I've given, which are deep modules. These are modules that hide complex behavior behind simple interfaces. This is a John Ousterhout idea from the philosophy of software design. If we look at these two modules, we've got A, which has a
- 8:52
large implementation hiding behind a tiny little interface at the top, okay? And then B is a large interface, lots of functions you can call, and those functions individually don't do very much. Does that make sense? Yeah? Now, if you have a deep module like A here, you're gonna have fewer structure-sensitive tests because you're hiding more of the implementation behind that interface. If it's just testing at that interface, you're gonna get better tests. And so your job
- 9:22
here is to force the agent to use that little interface instead of reaching into the implementation to test these weird implementation details. So I've got a skill for this. You can have like the weirdest vibe coded like code base, the crappiest code base that you've ever set your eyes on, and you can run this skill on it, and it will make it better. What this does, it essentially gives you opportunities for deepening modules, and kind of looks like this. Raise your hands if you've used this skill, by
- 9:52
the way. Not sure how many of my folks are in this room. Yeah, okay. It's really freaking nice. Essentially, it gives you a HTML document, and I'll get out of the way here, of all of the different, um, potential opportunities it sees. So we can see here we have a before and we have an after where we're sort of like reducing duplication. We're creating a nice deep testable module. And then you can go ahead and implement that. And attached to this, there's also this kind of language that I've put together for describing modules because like if you
- 10:22
ever try and read up about how to structure a code base, you're gonna find 20 different approaches, and they're all gonna be called DDD. And like what you need is a consistent language that you can use in your team to talk about this stuff. And so I have a little codebase design skill that defines what locality is, defines what leverage is, defines what a seam is. I was using seams before they were cool. And what locality means is kind of how, uh, well located together all of the code is. How can you
- 10:52
change like a small change in one module and have it ripple out? And also leverage is what you get when you have a deep module because the caller, the person who's actually calling that module, gets a lot of value of calling a simple function. Both of those are very good in code bases, and good for agents too it turns out. But I'm sort of describing all of these highfalutin coding standards, but how do we actually make sure the agent does them, right? How do you make sure the agent
- 11:22
creates deep modules and creates good tests and doesn't write these crap tautological ones or structure sensitive tests?
- 11:30
Well, I do think most people get this wrong. And my first piece of advice is don't put coding standards in your implementer agents, okay? Let me explain this. If you imagine the implementer agent kind of looks like this, where this is all of the things the agent needs to be able to do in its single context window. It needs to be able to explore, like to look for the code that it's going to change. It then needs to actually change it, so make the updates to the files in the
- 12:00
green. And then it needs some budget for actually debugging the thing, so for running those, uh, automated checks, for actually checking and verifying that it works. Now this is quite a lot of work it turns out, and if you try to impose your coding standards on it as well, it's going to perform worse.
- 12:21
So implementation is overloaded. That's the mental model I want you to have. And so can we find a way to impose those coding standards in a way that isn't so overloaded? Well, this is my effort. This is my code review skill, and it receives a diff, and it reads a file inside the repository called coding standards, which you can write, you can customize, and then it checks if the code follows those standards. And so if we look at the reviewer agent here, it also crucially
- 12:51
runs it in a sub-agent. So it's got its own context window to kind of handle here. It's got its own budget. This one, it needs to do some exploration, right? Because sure, it receives the diff, so it knows exactly where it's located, where the code is, but it should probably do a bit of exploration just so it has the wider context, understands the code. But it doesn't need to do any implementation, doesn't need to do any debugging. So while implementation is overloaded, review is actually underloaded, so it doesn't have that many
- 13:21
jobs to do. This means you can pile in a bunch of coding standards to it, and it will do a much better job than if you tried to do it with implement. So I think of this, and this is uncomfortable, right? 'Cause we, we all want to be able to just get good code out the first time. But I think of this as the two-part process for writing good code, which is implement, you make it work, and then code review, you actually make it good. You impose your coding standards. And for, you know, the retro developers among us,
- 13:51
this is essentially a red-green refactor approach. We use one context window to make it okay, do the red-green, and then we do another context window to refactor it. That's at least how it works in my head, and it's been very successful for me. This means that when you have coding standards, you don't put them in global scope. You don't put them in agents.md 'cause then they sort of drown out your implementer. It may read them, it may not. You put them in codingstandards.md, and that way just the code review agent does it.
- 14:22
Now, I've got another idea here, which is I've been talking to lots of people today, and lots of people saying, you know, I've been talking about review and automated review increased, you know, Fixing the PR Bottleneck. So many folks say, "Oh yeah, we just use a third-party service. We use, uh, Cursor Bug Bot, we use CodeRabbit," or something like that. I think that I've, I've really tried to make a generic code review skill in the past that finds all the bugs and does security review and that kind of thing. It turns out it's really, really hard because you either
- 14:52
make it too general and it just gives you false positives that aren't actually relevant to your use case, or you make it too specific. You say, "Okay, find all the TypeScript stuff," and then Rust people can't use it. So I would say don't outsource automated review. Build your own. Build up your own coding standards over time. Share them across your team. And if you have an opportunity to impose coding standards, if you've got some docs sitting around that no one reads, this is the place to put them in. And also,
- 15:22
when you have this automated reviewer, a really natural inclination for lots of people is to say, "Oh yeah, my code review agent, what it does is it reads the code and then it comments on the PR." Ugh. So what your code review d- agent is doing in that case is it's providing more work for the human reviewer. The human reviewer then has to read all of these verbose comments and figure out, "Okay, do we implement this? Do we implement that?" The reviewer should commit. It should make fixes. So it should actually change the things that it finds,
- 15:53
'cause then when the human comes around, you're reviewing a really nice artifact. If it finds anything that it has any questions over, then of course it can comment, but the default should be commits. Stop trying to one-shot good code. Stop trying to make the implementer agent the only thing that you do and go, "Okay, I'm gonna force it to be amazing." It takes a little bit of, you know, thinking your way out of there, but once you realize it, it is fabulous. So okay, with all of that process, we've run our automated
- 16:23
checks. We've now run our automated review to make sure the automated checks aren't lying. How do we then maximize the PR's quality in terms of human review? How do we get it, like, working the best it can? So we need a human-friendly PR, and this is a new skill, uh, coming into the repo, which is currently in progress, but I'll be releasing it soon, which is the PR skill. And this is one I've been mulling over for a long, long time, haven't quite figured out what the state of the art is, and I've realized the best way to make a
- 16:52
PR skill is just to steal all the best ideas that everyone's got, and it's very nice. Now, what does a good PR body look like? How would you recreate this skill on your own? Well, the first principle is some reviews are more important than others. Not every review is essential, right?
- 17:14
And once you understand this, you realize, okay, that means I can focus my energy on the really important reviews. But how do we categorize that? Well, you gotta think about the PR as using this AWS terminology, which is, is it a one-way door or is it a two-way door? Now, most PRs that you have will be two-way doors. You can merge the PR and then always pull it back later. It's the glorious benefit of being a software engineer as opposed to a civil engineer, right? Mostly when you're a civil engineer, it's a, it's a
- 17:44
one-way door, right? If you get something wrong, that bridge is going down. But if you have a two-way door, it means that you can easily revert the change. Now, that might be more, um, a bit more nuanced than you might expect. It might be that a very simple change accidentally blasts out an email to 60,000 people or something, in which case that is a one-way door. You wanna review that very, very carefully. Involves expensive migrations or data loss, that is a one-way door. Review the hell out of that PR. But also tied onto
- 18:14
that, we need to understand the blast radius of this PR. What can go wrong? And if things do go wrong, how bad is it? And this means I end up with a nice little sort of summary right at the bottom of all of my PRs, which is the merge danger. I can see this one is a two-way door. Its blast radius is localized, and so I can see, fantastic, I don't need to pay that much attention to this. I'm just gonna sort of review it a little. That's really important. Next, we need to understand what the PR is even doing, right? And I've
- 18:44
tried lots of different ways of figuring this out, and the best thing I've come up with is using pseudocode. Now, a huge, um, point of gratitude here to the Show Me skill from the HumanLayer skills repo by Dex Hawley. This is a phenomenal skill that just essentially dispenses with most text and shows things to you in images and diagrams instead. This makes it a lot easier to grasp actually what's changing and why it's changing. So you get the kind of standard
- 19:14
sort of set of like mermaid diagrams and UML for, you know, this is a sort of sequence of things that happened. You also just get these lovely simple ones like this, for instance. We can look at this and go, "Okay, we're working in a CLI. We can see a new command has been added, and we've got two new things, little flags up the top here." Just little summaries like this. It really does make a massive difference. It's hard to overstate. So you're m- trying to, uh, like make understanding the why as fast as possible. And
- 19:44
I think a third principle here is something you should be thinking about whenever you do human review. Because, because we're not doing like, um... Because our processes now are so sort of streamlined and all-- we're all collaborating around these same skill files, these same steering files. We're all building an environment for our agents to operate in together. You should think of the process that produces your code as just as important as the
- 20:14
code itself. In other words, when you do a human review, you're not just reviewing the code, you're reviewing the system that creates it. And the theory here is that you never want to write the same comment twice, right? You never want to catch the agent doing the same thing over two PRs. And so what's the mechanism by which you can make your human review matter? Well, this is a new skill. This is called Retro. Retro for retrospective. You essentially take a session that you've done. It can either be like a single,
- 20:44
um, agent session, or it can be a PR plus the session, or you can just get it to look at, okay, look at all of the PRs that we've done over the last week, all of the reviews, pull them all in. Let's do a retrospective on them. And it will suggest automated checks and coding standards to make the next one better. So this is the compounding effect, where you essentially by doing human review, you're making the quality of the next human review higher, and you're sort of saving less work from yourself next time. And
- 21:14
Retro is a really smart skill. It adds a bunch of stuff. So obviously it suggests automated checks. It suggests updates to codingstandards.md. It does other smart stuff too. So it looks at navigation pointers. How easily did the agent find its information? Can we provide a pointer inside agents.md to help it out next time? It looks at tool economy. Are there different tools that we're using in the session that can, you know, could be made more token efficient? It's amazing how many things this catches actually, 'cause those are often really
- 21:44
hard to debug from the outside. It just looks at bloat as well. So are there bloated steering files? Are there bloated skills that contribute to these bad results? Can we make them more organized? So that's the goal, is to make human review faster. We do that by layering up automated checks. We're layering up automated review, and we make the human review as painless, as simple, and as kind of optional as we need it to. You really don't need to review every single two-way door. Every
- 22:14
single one-way door you do. So these are my skills. aihero.dev/skills. I'm gonna be shipping version one point three this week. It has been glorious hanging out with you. It's been a really nice conference. I'm gonna be outside in the lobby if anyone wants to have a chat. Uh, thank you so much for having me. Thank you, Paris.