Every Correction You Give Your Agent Is Data - Brian Douglas, Paper Compute

Brian Douglas could direct an agent to write Rust for Paper Compute. His own experience writing the language came mostly from side projects a decade earlier. He could ask for a change and get plausible code back. The harder part was recognizing the moment an experienced Rust engineer would interrupt.

Rick Rubin’s The Creative Act: A Way of Being gave Douglas a way to think about it. A producer can direct a performance that someone else plays. An observation from Addy Osmani made the analogy more uncomfortable. Junior engineers develop judgment through repeated attempts, feedback and another chance to make the decision. Hand those attempts to an agent and an engineer can ship code while getting fewer of those reps. What could Douglas learn from the occasions when a teammate did stop the agent?

Douglas built Tapes, an Apache 2.0–licensed project that records agent sessions. An OpenTelemetry collector, a session-capture service or a coding harness with its own history can also supply traces. At Paper Compute, the team paired session capture with a sandboxed agent runtime so it could record work in a reproducible environment. Keeping the session preserved exchanges that a finished patch could leave out. Tapes

One use for that setup was Sweeper Agent. Documentation, test writing and code cleanup gave parallel agents well-defined jobs, including tidying software Douglas had generated over the previous 18 months. He reported scaling Sweeper Agent to 10 agents and, in separate work, running about 200 Daytona sandboxes at once. That gave him more capacity to clean up code. He still depended on colleagues to recognize questionable Rust.

How could Douglas find a decision he lacked the experience to question? Paper Console brought his sessions and a teammate’s sessions into a shared store. He opened a session about Paper Cloud export memory limits because it carried the label pushback. The label told him an engineer had challenged the agent. The surrounding conversation was where he could look for the reason.

Douglas looked in particular for objections from Matt, a teammate working in Rust and Go. Two kinds of objection show what Matt could recognize.

• API structure. Rust code can compile while arranging an API in a way the team has already decided against. Matt knew how that code was supposed to interact with other parts of the codebase. The prior decision gave him grounds to challenge a plausible implementation.
• Service reuse. An agent missing part of the repository context could propose a service whose job an existing service already performed. Spotting the duplication required knowing what was already there.

With Douglas at the keyboard, the objection often came later. He tended to accept the generated Rust and open a pull request. A five-out-of-five automated review could satisfy him, yet the engineers reviewing the PR still had objections. Matt’s earlier conversations offered a way to bring that scrutiny forward.

The console captured each session, and Douglas could also inspect the details from the command line. He collected the traces with their labels into a dataset. To build Twin Review, he selected examples of Matt pushing back on Go or Rust code and used them for post-training. Each exchange connected a choice the agent had made with the objection it had prompted.

What did Twin Review need when it was time to review a change? Douglas invoked the command-line tool with PR 381 and asked for comments of the kind Matt would leave on a GitHub pull request. The PR supplied the code to inspect. Matt’s earlier objections had shaped the reviewer during training. The demonstration left open how often its judgments would agree with Matt’s on an independent set of PRs.

GitHub review threads would have been another possible source of examples. Most of Douglas’s team, though, had started reviewing at the command line. An engineer could ask an agent to review a change, challenge its answer and settle the issue inside the harness. The shared repository might receive the resulting code with little commentary about the decision. Twin Review drew its training examples entirely from session traces. Douglas used Fireworks for post-training and kept the prototype private because it contained sensitive data.

Pushback was one reason to revisit a trace. An agent’s apology could point Douglas toward the exchange that prompted it. A golden session was one someone had deliberately marked as worth studying. Regressions gave him examples of work where agents repeatedly ran into trouble.

One recurring failure was an agent writing its own version of an existing Go library. The team had seen this with mocking libraries for tests. If you know the library, you can stop the agent as soon as you recognize what it is building. If you have never used it, the new implementation can look like ordinary progress. That interruption gives a reviewer something specific to learn. Douglas also saw room to use examples like these in a local review harness without taking on model training.

Douglas’s first experiment was to turn Paper traces into skills, including one that let an agent check the recorded sessions. His local Tapes installation stored them in PostgreSQL, so he could work directly with his history while Paper Console was waitlisted. Across the team, almost 80,000 sessions were available to draw on for skills.

Consider Matt’s objection to a duplicate service. A skill could direct the agent to look for an existing service before proposing another one. An evaluation could then give a reviewer a change that duplicates the service and ask whether it flags the duplication. That makes a broad aim like following the team’s Rust conventions specific enough to test. The team could put the same question to a post-trained model later.

For his smaller fine-tuning experiment, Douglas likened an adapter to notes in the margin of a book. The book is the pretrained model. The adapter adds learned weights that specialize its behavior. He used quantization to fit the work on a smaller GPU and was pleased with the result he got from a four-billion-parameter model.

His experiments with direct preference optimization (DPO), which uses preferences between candidate responses to steer a model, took considerably more compute. Douglas said he needed a model with 70 billion parameters or more for his DPO experiment. He lacked the hardware locally, and Brev was the only service through which he could get access to the larger NVIDIA GPUs he needed at the time. For others considering that route, he cautioned that access can require a substantial prepayment.

Douglas’s closing recommendation was the cheaper experiment he had started with. Use the session history you can already capture, find an objection that explains a team decision, and write a skill you can test. The next time an agent starts building a mocking library, does the skill catch what the teammate caught?