I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore

I believe everything my agents tell me. And yet my agents get things wrong all the time. I've put over a hundred and seven billion tokens through agents this year, so I analyzed my agent sessions from twenty twenty six year to date to see where they went wrong. In about four in ten sessions, an agent got something wrong that mattered. That's how often agents get things wrong in the first place. It isn't how often those errors made it into work outputs.

Catching those errors before they reach me is the job of the agent-based review method that I've been running for most of this year, and that's the method that I'm sharing with you today. This analysis identified three ways agents keep failing. Number one, they overstate what they did. That is, they say it's done or checked when it isn't. Number two, they get the facts wrong. They misread the source, for example. Or number three, they get the reasoning wrong, so

the facts are right, but the conclusion isn't.

The method is comprised of three levels of review, one to address each of the three ways that agents keep failing. I built each level as an agent skill. Here's how each one works, and then I'll show you how each was applied in three real cases. Level one prevents agents overstating what they did. Before an agent starts, the job spells out exactly what it has to hand back: the file, where it goes, and a command that checks

it. It points out the shortcuts agents have taken before, so this one doesn't repeat them. The agent performing the review has to end its review with one of two things: artifact verified with a fingerprint of the file or files documenting its work or artifact missing with the reason. Then the agent that sent the job runs the check itself. Saying done on its own isn't a permissible answer. Level

two prevents agents from getting the facts wrong. When correctness matters, first find something in the source that the answer has to agree with. Then two agents get an identical set of instructions and do the same job independently. Neither one sees the other's work. Each one notes where every fact came from and checks its result against the facts. Then a third agent with the original source in front of it compares the work of the two independent

agents line by line and settles every difference by going back to the source. It never forces a match. Level three prevents agents getting the reasoning wrong. When clarity of thought really matters, we perform adversarial review in three stages. First, the agent audits its own work. It lists every claim it's making and marks each one as measured, checked against the source,

inferred, or assumed. Anything it can't back up gets tested or labeled as a guess. Second, reviewers with none of the original agent's context, each acting as a persona with reasons to find fault, read it cold. Each one plays a real reader and gets the work and the sources, but no history, no drafts, and no conclusions to agree with. Their findings get checked against the source too

before anything changes. Third, once the work is revised, brand-new reviewers who've never seen it check the final version. Each level adds to the one before as the stakes go up. In these cases, each review took minutes, not hours, and the agents do the checking, not me. At the scale we operate, checking it myself isn't an option. So let's see each review level applied in three real cases. First, a memory

leak. An open source dictation app I use was leaking memory. It grew to nine gigabytes. We set out to find the leak, fix it, and get the fix accepted by the project's maintainer. And to fix it, we had to find out what was really causing it. My agents and I went after it, and three times an agent told me something that wasn't true. First, a sub agent I sent to measure the leak with one tool used another and reported its work

as done without saying it hadn't used the tool as instructed. Nobody caught it at first, including the agent that dispatched the work. It came out when I questioned the results. The file the right tool would have produced didn't exist, and nobody had noticed. That's what level one of the review was for. In this case, we dispatched a sub agent to build a specially instrumented version of the app and capture telemetry while it was running. However, the build failed.

Because we instructed this sub agent to provide documentation of all of its work instead of just done, it said the main deliverable wasn't delivered and explained why. It reported that an artifact was missing, the software build, and that there were twenty-five compile errors, and it correctly labeled the... what it could deliver as partial. Dispatching the sub agent with level one review tasking included took nineteen minutes and about sixteen million tokens.

The other two significant inaccuracies by agents were about to go public, and the stakes were higher, so for those, we borrowed from level three, reviewers with none of the agent's context. We'll see all three stages of a level three review in the last example I explain shortly, but here's how it worked for the memory leak fix. In the second case, an agent blamed the wrong code. To find the leak source, it wrote a small test that used the same library the app does. The test leaked too, so it concluded

that the library was to blame. It said this is solid enough to file. Reviewers reproduced the test and found that it had skipped an important step that the library requires. Used properly, the library didn't leak. The problem was in how the app used the library. In the third significant error by agents working on the memory leak, the agent wrote up our fix for the pull request and said it had eliminated the leak completely, but it had measured the leak in a way that

missed most of it. Reviewers caught that also. The fix reduced the leak, but it didn't fully eliminate it. So the pull request went up with honest numbers, and the maintainer accepted the fix and merged it. Each of these level three reviews took about fifteen minutes and together cost about six million tokens. For the next example, we'll look at portfolio company accounting. Buying and building businesses can be messy. For one of our portfolio companies, we needed a year of scanned statements

from a source accounting system turned into a complete, accurate record of every business transaction for financial reporting purposes. An agent extracted the year and handed it back with, "Hey, I recommend spot checking a few of these transactions." This put the burden of reviewing the agent's work back on us, which is a non-starter. Extracting data from scans often goes wrong in ways that are tedious and difficult to detect. Transactions

are missed entirely, transactions are put in the wrong months, or they're recorded with the wrong details. One agent simply reporting done can't tell you whether any of that's happened, and that's what level two of the review method is for. Here we use the balance printed on the statements as the control total for each month. Two agents extracted every month independently, and each one went back to the statements until every month balanced. Then a

third agent compared the two extractions line by line and settled each difference against the statements themselves. Finally, two more agents tasked identically checked each of the final transaction records before they were accepted. This fan out, fan in pattern of identically tasked agents is highly effective in driving accuracy and reliability. When we ran this level two review, we caught missed transactions, incorrect transactions, and

transactions in the wrong months. Comparing the two extractions caught fifty-six differences in descriptions, dates, and notes. But ultimately, the whole year balanced to the penny. This entire review took about twenty minutes and cost seven and a half million tokens. Last, the highest stakes example. More messy portfolio company business here. One of ours was in a billing dispute with a vendor, and this amounted to thousands of dollars with legal

implications. We needed to send a demand letter clearly stating our position and requirements, and every claim in it had to stand up against the contract and each invoice. An agent drafted the letter on the premise that the vendor's contract had expired. Anything with legal or financial stakes gets a level three review before it goes out. And here's how level three ran on this letter. First, the self-audit. The agent checked the claims in the letter against the

contract and the invoices, including the dates, and reported them verified. Stage one passed. Second, cold reviewers, each in a role with a reason to find fault. So in this case, our own litigator, the vendor's counsel, and an auditor who trusts none of the numbers. They each got the letter, the contract, and the invoices and nothing else. The auditor recomputed the financial demand from the invoices, and it was

correct down to the cent. But both lawyers independently found something buried deep in the contract's fine print, a three-year auto-renewal clause. The contract had auto-renewed less than three weeks earlier. The letter's demands would have had us walking away from a contract that was still in force and owing the vendor thousands of dollars for leaving early. The two lawyer agents also learned that while revising, the original agent had even deleted a line that had the renewal correctly

identified. The original agent's own check had agreed with its own error. Just the sort of thing this review method is designed to catch. Our litigator's verdict, though, was do not send the letter. That three-part cold review process took fifteen minutes and cost thirteen million tokens. So the letter was rewritten and handed to the third stage of level three review. Brand new reviewers who'd never seen it checked the final version. The

flawed version never went out. So that's three review stories and three levels of review. An agent that couldn't build a customized version of the app said so instead of incorrectly saying that it was done. Two groups of two agents extracted and accounted for a year of business transactions down to the penny, and cold persona-based reviewers with none of the original agent's context revised a letter that the drafting agent's own check had called verified.

I've packaged the three review levels as agent skills, and I've released them as open source. The write-up is at gallon.me. Point your agent at the write-up, and it will know how to check its own work using this review method so you can rely on it. To learn more of the best ways to do real work with AI agents, subscribe to my newsletter and connect with me here.

So I believe everything my agents tell me, not because they're right. They get things wrong all the time. I believe them because everything I rely on has been thoroughly reviewed and validated first, and most importantly, not by me.