# Every Correction You Give Your Agent Is Data - Brian Douglas, Paper Compute Brian Douglas, Paper Compute [Talk page](https://ai.engineer/talks/vg3JdjGuJwM-every-correction-you-give-your-agent-is) · [Video](https://www.youtube.com/watch?v=vg3JdjGuJwM) ## Every Correction You Give Your Agent Is Data A Rust program can compile and still violate a decision the team has already made. At Paper Compute, Brian Douglas used a teammate’s objections to build a prototype pull-request reviewer. His first use for the traces was simpler. He wrote a skill that could look back through recorded sessions. ### Writing Rust before you know when to object [0:01](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=1s) Brian Douglas could direct an agent to write Rust for Paper Compute. His own experience writing the language came mostly from side projects a decade earlier. He could ask for a change and get plausible code back. The harder part was recognizing the moment an experienced Rust engineer would interrupt. Rick Rubin’s *The Creative Act: A Way of Being* gave Douglas a way to think about it. A producer can direct a performance that someone else plays. An observation from Addy Osmani made the analogy more uncomfortable. Junior engineers develop judgment through repeated attempts, feedback and another chance to make the decision. Hand those attempts to an agent and an engineer can ship code while getting fewer of those reps. What could Douglas learn from the occasions when a teammate *did* stop the agent? ### Keeping the session alongside the work [1:47](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=107s) Douglas built Tapes, an Apache 2.0–licensed project that records agent sessions. An OpenTelemetry collector, a session-capture service or a coding harness with its own history can also supply traces. At Paper Compute, the team paired session capture with a sandboxed agent runtime so it could record work in a reproducible environment. Keeping the session preserved exchanges that a finished patch could leave out. [Tapes](https://tapes.dev/) One use for that setup was Sweeper Agent. Documentation, test writing and code cleanup gave parallel agents well-defined jobs, including tidying software Douglas had generated over the previous 18 months. He reported scaling Sweeper Agent to 10 agents and, in separate work, running about 200 Daytona sandboxes at once. That gave him more capacity to clean up code. He still depended on colleagues to recognize questionable Rust. ### Finding the decision behind a `pushback` label [3:43](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=223s) How could Douglas find a decision he lacked the experience to question? Paper Console brought his sessions and a teammate’s sessions into a shared store. He opened a session about Paper Cloud export memory limits because it carried the label `pushback`. The label told him an engineer had challenged the agent. The surrounding conversation was where he could look for the reason. Douglas looked in particular for objections from Matt, a teammate working in Rust and Go. Two kinds of objection show what Matt could recognize. - **API structure.** Rust code can compile while arranging an API in a way the team has already decided against. Matt knew how that code was supposed to interact with other parts of the codebase. The prior decision gave him grounds to challenge a plausible implementation. - **Service reuse.** An agent missing part of the repository context could propose a service whose job an existing service already performed. Spotting the duplication required knowing what was already there. With Douglas at the keyboard, the objection often came later. He tended to accept the generated Rust and open a pull request. A five-out-of-five automated review could satisfy him, yet the engineers reviewing the PR still had objections. Matt’s earlier conversations offered a way to bring that scrutiny forward. ### Giving Twin Review PR 381 [6:06](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=366s) The console captured each session, and Douglas could also inspect the details from the command line. He collected the traces with their labels into a dataset. To build Twin Review, he selected examples of Matt pushing back on Go or Rust code and used them for post-training. Each exchange connected a choice the agent had made with the objection it had prompted. What did Twin Review need when it was time to review a change? Douglas invoked the command-line tool with **PR 381** and asked for comments of the kind Matt would leave on a GitHub pull request. The PR supplied the code to inspect. Matt’s earlier objections had shaped the reviewer during training. The demonstration left open how often its judgments would agree with Matt’s on an independent set of PRs. GitHub review threads would have been another possible source of examples. Most of Douglas’s team, though, had started reviewing at the command line. An engineer could ask an agent to review a change, challenge its answer and settle the issue inside the harness. The shared repository might receive the resulting code with little commentary about the decision. Twin Review drew its training examples entirely from session traces. Douglas used Fireworks for post-training and kept the prototype private because it contained sensitive data. ### What a recurring failure looks like in Go [8:12](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=492s) Pushback was one reason to revisit a trace. An agent’s apology could point Douglas toward the exchange that prompted it. A *golden session* was one someone had deliberately marked as worth studying. Regressions gave him examples of work where agents repeatedly ran into trouble. One recurring failure was an agent writing its own version of an existing Go library. The team had seen this with mocking libraries for tests. If you know the library, you can stop the agent as soon as you recognize what it is building. If you have never used it, the new implementation can look like ordinary progress. That interruption gives a reviewer something specific to learn. Douglas also saw room to use examples like these in a local review harness without taking on model training. ### Turning a correction into a skill you can test [9:32](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=572s) Douglas’s first experiment was to turn Paper traces into skills, including one that let an agent check the recorded sessions. His local Tapes installation stored them in PostgreSQL, so he could work directly with his history while Paper Console was waitlisted. Across the team, almost 80,000 sessions were available to draw on for skills. Consider Matt’s objection to a duplicate service. A skill could direct the agent to look for an existing service before proposing another one. An evaluation could then give a reviewer a change that duplicates the service and ask whether it flags the duplication. That makes a broad aim like following the team’s Rust conventions specific enough to test. The team could put the same question to a post-trained model later. ### The hardware cost of adapting the model [10:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=629s) For his smaller fine-tuning experiment, Douglas likened an adapter to notes in the margin of a book. The book is the pretrained model. The adapter adds learned weights that specialize its behavior. He used quantization to fit the work on a smaller GPU and was pleased with the result he got from a four-billion-parameter model. His experiments with direct preference optimization (DPO), which uses preferences between candidate responses to steer a model, took considerably more compute. Douglas said he needed a model with 70 billion parameters or more for his DPO experiment. He lacked the hardware locally, and Brev was the only service through which he could get access to the larger NVIDIA GPUs he needed at the time. For others considering that route, he cautioned that access can require a substantial prepayment. Douglas’s closing recommendation was the cheaper experiment he had started with. Use the session history you can already capture, find an objection that explains a team decision, and write a skill you can test. The next time an agent starts building a mocking library, does the skill catch what the teammate caught? ## Key takeaways - Keep the exchange around an expert correction. A `pushback` label helps you find it again, and the trace connects the proposed implementation to the team decision it violated. - Look for examples where engineers actually review code. For Douglas’s team, command-line sessions held objections that rarely reached the GitHub discussion. - A recurring mistake can supply an evaluation before a team spends compute on post-training. Give a reviewer a change that reinvents an existing testing library and see whether it recognizes the duplication. ## Slides ### If You Are Going to Verbally Abuse Your Agents, at Least Label the… Brian Douglas opens with a joke about venting at a coding agent. When you correct the agent, he argues, the outcome is worth recording too. [0:00](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=0s) [Large white text on a dark background reads “If You Are Going to Verbally Abuse Your Agents, at Least Label the Outcome.”](https://assets.ai.engineer/sites/aiecode2025/sha256/960cb083be127e31032290713fcdc7c31e0eeddc8b05d2c1c24f0c9dca5d452e.webp) If You Are Going to Verbally Abuse Your Agents, at Least Label the Outcome AIE SF Code @b[illegible] ### Earning Judgment Addy Osmani argues that junior engineers used to develop taste through repeated work. When agents take over those reps, juniors have to seek out practice that builds their judgment. [1:11](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=71s) [A red slide shows the article “Earning Judgment” above the caption “agents will automate all of these reps.”](https://assets.ai.engineer/sites/aiecode2025/sha256/94a2a99c0879cb6acf7005398592694ce57ac84c6ca6d4cb30966ddace06b76c.webp) Earning Judgment THE JUNIOR TO SENIOR PATH IN THE AGENTIC ERA Addy Osmani @addyosmani Earning Judgment Taste used to be a byproduct of the reps. Agents took the reps. So if you're junior you now have to go get the taste (and judgment) on purpose. “agents will automate all of these reps.” Interface text: Article 30 71 437 1.3M ### Vision: Monuments A giant cassette labeled tapes.dev stands like a monument in a traffic circle. It introduces Tapes, the open-source tool Brian Douglas uses to record agent sessions. Other session capture tools can do the same job. [1:46](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=106s) [An aerial city view shows a giant cassette labeled “tapes.dev” standing in the middle of a traffic circle.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca0877542c79a46ea4b420664791b451201de92166f4b030eab97ce6ada831f1.webp) Vision: Monuments tapes.dev ### Memory and feedback in agent systems Douglas argues that most agents lack memory and feedback. He uses tapes.dev to capture and stream every LLM call, and stereOS to give the agent a reproducible sandbox where he can replay a session. [2:13](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=133s) [The headline highlights “memory + feedback” in red. Two cards describe how tapes.dev captures model calls and stereOS provides a reproducible, replayable sandbox.](https://assets.ai.engineer/sites/aiecode2025/sha256/b464718341f3069e8334ee0681377f5b893f0a8604b1feea54c90e33f69cf0d1.webp) THE THESIS 05 / 25 THE BIG IDEA The missing piece in most agent systems isn't intelligence. It's memory + feedback. MEMORY tapes.dev Captures every LLM call. Streams it. Owns the data. ENVIRONMENT stereOS The sandbox the agent runs in. Reproducible. Replayable. TAPES.DEV B SIDE TAPES 004 TAPES [illegible]DEV [illegible]ompute.com TAPES · STEREOS · SWEEPER ### Better signals Douglas records agent sessions in a reproducible environment. Here he turns to the signals that the environment itself provides. [2:33](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=153s) [A red “09” appears above the white heading “Better signals” on a dark background.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f610f4544ecd2acaa4b8f9f3892695736491e07d32de41d94493714e8e25893.webp) 09 Better signals DEVRELCON @b[illegible] ### 842 Lint Errors, 5 Parallel Agents, 54 Minutes Auto-fix cleared 1,150 of contributor.info’s 1,992 ESLint errors. Sweeper grouped the remaining 842 errors by file and built a prompt for each of the 99 files so five Claude Code agents could work in parallel. [3:10](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=190s) [A blog article explains how Sweeper distributes lint fixes across parallel agents. Of 1,992 ESLint errors, 842 remained across 99 files after automatic fixes.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7d24fac88a674e20e8cc971bd3563045f16e81b2ab0e132eebe33a9ce238af7.webp) 842 Lint Errors, 5 Parallel Agents, 54 Minutes March 15, 2026 I had 1,992 ESLint errors in contributor.info. The project had been through a heavy vibe coding phase and the codebase showed it. Auto-fix handled 1,150 formatting issues. That left 842 errors across 99 files that needed an agent to reason about. I could have opened Claude Code and started grinding through files one at a time. But I’d been building Tapes and stereOS for months and wanted to prove a thesis: agent infrastructure changes what agents can do. Not better prompts. Not bigger context windows. Infrastructure. So I built Sweeper. What it does Sweeper takes any linter output and turns it into parallel agent work. Point it at golangci-lint, ESLint, Clippy, or a custom script that checks for AI slop. It groups issues by file, builds a prompt for each file, and fans out Claude Code sub-agents to fix them. sweeper run -- npm run lint # arbitrary linter command sweeper run # default: golangci-lint npm run lint | sweeper run # piped input Interface text: briandouglas.me/posts/2026/03/15/sweeper-why-agent-infrastructure-matters/ bdougie Home Blog Subscribe Newsletter ⌘K Back to all posts ### Paper Traces Tapes captures the team’s agent sessions in one place. Paper Console shows where the week’s $880.11 went: each task has a row with its status, cost, token count and duration. Of 86 sessions, 78 completed. [3:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=209s) [The Paper Traces slide shows a sessions dashboard with total spend, tokens used, agent time, tool calls and success rate above a table of runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/d35242aca6d5547df08083498c3310ecc142ac0d19ce0a9c3891aa195a49e775.webp) Paper Traces Sessions All My Team 24h 7d 30d Total Spend $880.11 $10.23 avg Tokens Used 1.33B 1.33B in / 5.39M out Agent Time 7d 2h 1h 59m avg Tool Calls 5773 67 avg Success Rate 91% 78/86 complete All Running Completed Failed Search sessions... Session | Status | Name | User | Turns | Cost | Tokens | Duration | Started 019f1f51 | Running | Review open PRs for mergeability /Users/bdougie/code/pcc-labs/pokemon-kafka | Brian Douglas (You) | 14 | $5.46 | 10.91M | 7m | 7m ago 019f1f49 | Running | Checkout and pull main branch /Users/bdougie/code/papercomputeco/site | Brian Douglas (You) | 7 | $4.30 | 2M | 14m | 16m ago 019f1f3f | Completed | Redesign evals UI for quick scanning /Users/bdougie/code/pcc-labs/paperplane | Brian Douglas (You) | 4 | $7.90 | 5.12M | 12m | 27m ago 019f1f3c | Completed | Run benchmarks on model weights /Users/bdougie/code/pcc-labs/empirical-evidence | Brian Douglas (You) | 5 | $10.57 | 14.24M | 16m | 30m ago 019f1f46 | Completed | A competent general-purpose agent with NO special sk... unknown | Brian Douglas (You) | 1 | $0.00 | 596 | 0s | 19m ago 019f1f46 | Completed | An agent has access to this skill: Steps to Captu... unknown | Brian Douglas (You) | 1 | $0.00 | 853 | 0s 019f1f46 | Completed | A competent general-purpose agent with NO special sk... unknown | Brian Douglas (You) | 1 | $0.00 | 604 | 0s 019f1f46 | Completed | An agent has access to this skill: Steps to Captu... unknown | Brian Douglas (You) | 1 | $0.00 | 893 | 0s 019f1f46 | Completed | A competent general-purpose agent with NO special sk... | Brian Douglas (You) | 1 | $0.00 | 588 | 0s Interface text: Claude Claude Claude Ask Gemini console.papercompute.com/?period=7d&sort=start_time&direction=desc&page=1&gateway=default Work paper compute co. pokemon › Sessions Sessions Skills Agents SOON Insights ### 30-day session overview in Paper Console Douglas captures every agent session. In the 30-day view, Paper Console breaks down $2,635.82 in spend by model across 200 sessions. The highlighted autolabeler session cost $8.16 over 19 turns. [6:12](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=372s) [A terminal dashboard shows spending by model, an 86% completion rate and a sortable session list.](https://assets.ai.engineer/sites/aiecode2025/sha256/093b069013b29da7984401ba38628e9e6cc8ab1433618610616e690359bcbb14.webp) paper console last 1540h30m · 200 sessions 24h 7d 30d 90d 120d (p to change) TOTAL SPEND $2635.82 $13.18 avg TOKENS USED 4850.2M in / 12.8M out 24.3M / 64.0K avg AGENT TIME 1540h30m 7h42m avg TURNS 1443 7 avg SUCCESS RATE 86% 173/200 complete cost by model gpt-6-astra $1019.59 9 claude-opus-5-5 $863.47 13 claude-fable-5-1 $715.74 20 claude-sonnet-4-6 $19.94 123 gpt-6.1-sol $16.64 5 session status completed 86% (173) failed 0% (0) abandoned 0% (0) in progress 14% (27) eff: $13.18/sess 52612 tok/m sessions (sort: date desc, status: all) [s] sort [f] filter [/] search label | model | dur | tokens | in / out | cost | turns | status | date | project 01 Skills-evaluator-cassette PR 11 twin review | claude-fable-5-1 | <1m | 422.8K | | $ $0.63 | 1 | completed | Nov 06'26 | twin-reviewer 02 Running tapes-deck locally | claude-fable-5-1 | 3m | 2.5M | | $ $2.00 | 3 | completed | Nov 06'26 | papercomputeco > Autolabeler identification question | claude-fable-5-1 | 101m | 12.0M | | $ $8.16 | 19 | completed | Nov 06'26 | labeler-cassette 04 Paper cloud export memory limits | claude-fable-5-1 | 434m | 109.9M | | $ $47.71 | 7 | completed | Nov 05'26 | big_export 05 Bottlerocket admin container sshhole exposure | claude-fable-5-1 | 1m | 360.8K | | $ $1.06 | 1 | completed | Nov 05'26 | bottlerocket-forest 06 Paperd twin reviews data | claude-fable-5-1 | 1m | 5.8K | | $ $0.97 | 1 | completed | Nov 05'26 | twin-reviewer 07 Hyperbolic blog formula notation | claude-fable-5-1 | 16m | 577.2K | | $ $2.84 | 1 | completed | Nov 04'26 | matt 08 DGX Sparks Bottlerocket image | claude-fable-5-1 | 3406m | 175.5M | | $ $137.97 | 127 | completed | Nov 03'26 | bottlerocket-forest 09 Scale and Start pricing removal | claude-fable-5-1 | 122m | 16.9M | | $ $9.42 | 9 | completed | Nov 03'26 | site 0a Write a brief catch-up for a user returning t... | gpt-6-astra | 0m | 11.7K | | $ $0.06 | 1 | completed | Nov 02'26 | showing 1-10 of 200 loaded · more available · press n Interface text: .../code/papercomputeco/tapes ⌘1 paper-console ⌘2 Running tapes-deck locally ⌘3 Skills-evaluator-cassette PR 11 twin review ⌘4 + j down · k up · enter drill · h back · s sort · f status · / search · p period · r replay · n more · c chat · q quit ### Inspecting a conversation in Paper Console Douglas selects message 11 in a 19-turn session. The detail panel shows the user asking for a different approach to export, just before the largest token spike. Douglas collects session traces like this in a labeled dataset. [6:17](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=377s) [A 19-turn session shows cost and token totals above a conversation timeline. A selected row in the message list opens its detail panel on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c2322d49c4fa429c7799159177da96b8b8ceaee245243391a464128180c5275.webp) paper > labeler-cassette > is this the autolbaler? ● completed 01a1ae3 · 19 turns TOTAL COST $8.22 ↓ 37.6% vs avg $13.18 avg TOKENS USED 12.2M total ↓ 49.8% vs avg 12.2M in 100% 54.8K out 0% AGENT TIME 1h44m ↓ 77.5% vs avg 7h42m avg TURNS 19 ↑ 163.3% vs avg 7.2 turns avg conversation timeline (sort: time, 38 groups, 38 messages) tokens Y-axis ticks: 2.4M, 1.2M, 0 Timeline: 1, 6, 11, 16, 21, 26, 31, 36 Legend: user, assistant, tools, current conversation [s] sort [↑↓] navigate (grouped) # role time tokens cost delta 2 asst 00:26:33 210.3K $0.96 0m 3 user 00:28:02 0 $0.00 1m 4 asst 00:28:37 444.6K $0.30 0m 5 user 00:28:56 0 $0.00 0m 6 asst 00:29:16 153.0K $0.07 0m 7 user 00:29:40 0 $0.00 0m 8 asst 00:34:29 735.7K $0.68 4m 9 user 00:54:35 0 $0.00 20m 10 asst 00:57:45 395.4K $0.15 3m > 11 user 01:06:42 0 $0.00 8m 12 asst 01:11:34 2388.6K $1.84 4m 13 user 01:11:41 0 $0.00 0m 14 asst 01:12:04 560.5K $0.21 0m 15 user 01:12:47 0 $0.00 0m 16 asst 01:12:50 140.8K $0.05 0m 17 user 01:13:46 0 $0.00 0m 18 asst 01:13:47 141.2K $0.04 0m 19 user 01:14:55 0 $0.00 1m 20 asst 01:15:40 432.2K $0.28 0m message detail Role: User Time: 01:06:42 (+8m) Tokens: In 0 tok Out 0 tok Total 0 tok Cost: In $0.00 Out $0.00 Total $0.00 Message: instead of export can we use papaerd direct Interface text: …/code/papercomputeco/tapes ⌘1 paper-console ⌘2 Running tapes-deck locally ⌘3 Skills-evaluator-cassette PR 11 twin review ⌘4 + j down · k up · enter drill · h back · s sort · f status · / search · p period · r replay · n more · c chat · q quit ### Completed twin review of a pull request Brian trained a reviewer on Matt’s corrections during Go and Rust coding sessions. On tapes#381, it flags one of nine hunks. The agent reached the right design from the wrong example. The menu tray is just another client of the daemon. [6:50](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=410s) [A boxed terminal result shows one comment about the menu tray’s relationship to the daemon. Eight other hunks pass without comment, and the tool reports 96% confidence.](https://assets.ai.engineer/sites/aiecode2025/sha256/f26253c4e307a4ce8c86e8ffdb0f93990427b777e374ee025157d5c421425cbc.webp) ➜ tapes git:(main) twin review matt papercomputeco/tapes 381 Reviewing tapes#381 · feat: add copilot agent support to tapes start · 9 Go/Rust hunks ✓ Matt reviewed Matt on tapes#381 · feat: add copilot agent support to tapes start Matt would leave 1 comment and let the other 8 hunks go. cmd/tapes/start/start.go:1114 This is a curious tell to the agent's understanding of this code. It got to the right design but from the wrong example. We shouldn't think of the menu tray as anything more than another client to the daemon. We might want to update the AG. Let go without comment: cmd/tapes/start/start.go (6 hunks), cmd/tapes/start/start_test.go (2 hunks). Twin confidence: high (96%) · rarely flags what Matt would let go, so these comments are worth reading Decision by Matt's adapter · wording by qwen3:8b ➜ tapes git:(main) Interface text: .../code/papercomputeco/tapes ⌘1 paper-console ⌘2 Running tapes-deck locally ⌘3 Skills-evaluator-cassette PR 11 twin review ⌘4 + ### Fine-tuning on agent sessions with Fireworks The tapes-eject-fireworks README shows how to split local Codex and Claude Code history. User corrections become test cases for comparing the base and tuned models. The remaining turns supply training data for a LoRA adapter on Fireworks. [8:02](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=482s) [A GitHub README explains how local agent history supplies training examples for a LoRA adapter and how user corrections help compare the tuned model with the base model.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5fb5f6df187e67227c500e04335be014db06ac441739ac3a1af9021f696e4f2.webp) tapes-eject-fireworks Fine-tune a model on your own Codex and Claude Code sessions with Fireworks serverless training. tapes imports your agent history into a database on your laptop. This repo reads it from there, finds the turns where you corrected the agent, turns the rest into a training set, trains a LoRA adapter on Fireworks, and scores the base model against the tuned one on your corrections. Everything up to training runs locally. The only data that leaves your machine is the training set and the eval prompts sent to Fireworks. No GPU is needed: Fireworks trains on a shared pool and bills per token. How it works ~/.codex/sessions, ~/.claude/projects └─ tapes-skills-demo import history into local tapes (D[illegible] └─ tapes-eject label find pushback and apology turns └─ tapes-eject mark mark sessions golden or regression └─ tapes-eject export sessions, turns, labels -> da[illegible] └─ tapes-eject prepare training.jsonl, eval_cases.jsonl Interface text: README MIT license No packages published Publish your first package Contributors 1 bdougie Brian Douglas Languages Python 100% Chrome File Edit View History Bookmarks Profiles Tab Window Help Sun Nov 8 9:31 PM Paper Compute Console Paper Compute Console pcc-labs/tapes-eject-firewor[illegible] tapes — Telemetry for your c[illegible] Paper Compute - Run self he[illegible] Ask Gemini github.com/pcc-labs/tapes-eject-fireworks Work Relaunch to update ### How session labels shape training and evaluation Four in five golden sessions go into training. The rest join every regression session and every corrected turn in evaluation. A judge checks whether the model can answer the original request before the user has to correct it. [8:14](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=494s) [The README lists pushback, apology, golden and regression labels. The paragraph below explains that corrected turns become evaluation cases and four in five golden sessions supply training examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b99b597cff512ed1118d73aca68f7955c8db5df59b901a7b32e8a89b54b3581.webp) Labels decide the data: • pushback: a turn where you corrected the agent ("no, ...", "don't ...", "you deleted ..."). Found by label. • apology: a turn where the agent backtracked ("you're right", "my mistake"). Found by label. • golden: a session worth learning from. Set by hand with mark. • regression: a session that went wrong. Set by hand with mark. Sessions with no correction or failure label become training examples, along with four in five golden sessions. Every corrected turn, every regression session, and the remaining golden sessions become eval cases. A session is never in both sets. An eval case asks the model to answer the conversation up to your correction, and a judge model checks whether the answer already does what you had to ask for. Requirements • Python 3.11+ and uv • Docker, running • [illegible] └─ train/fireworks_sft.py LoRA SFT on Fireworks -> score bas[illegible] Interface text: README MIT license Chrome File Edit View History Bookmarks Profiles Tab Window Help Sun Nov 8 9:32 PM Paper Compute Console Paper Compute Console pcc-labs/tapes-eject-firewor[illegible] tapes — Telemetry for your c[illegible] Paper Compute - Run self he[illegible] Ask Gemini github.com/pcc-labs/tapes-eject-fireworks Work Relaunch to update ### The Tapes skill and its trigger phrases The highlighted skill description tells the agent when to search past sessions. A request like “check the tapes” cues the agent to start Tapes services if needed and query a local SQLite database of recorded sessions across projects. [9:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=569s) [A GitHub preview of SKILL.md outlines the Tapes skill’s name and description in red. Below it, a list of phrases for finding past agent sessions begins with “check the tapes.”](https://assets.ai.engineer/sites/aiecode2025/sha256/95d75656dad300be9796591c65b3ba2beaa8621f6ffa900a6389bae2ffeb80c6.webp) Tapes name tapes description Use when the user says "check the tapes", "search tapes", "tapes search", or wants to look up past agent sessions. Starts tapes services if needed and queries the local SQLite store at ~/.tapes/. Query local tapes session data. The tapes SQLite database lives at ~/.tapes/ and contains recorded LLM agent sessions across all projects. Trigger phrases • "check the tapes" • "search tapes for ..." • "tapes search ..." • "look up sessions about ..." • "what did I work on ..." Interface text: papercomputeco / skills Type / to search Code Issues Pull requests Actions Security and quality Insights Settings Files main Go to file T .github skills confluent-cloud-setup dagger-check tapes SKILL.md .gitignore README.md flake.lock flake.nix skills / skills / tapes / SKILL.md bdougie Change SQLite db path from tapes.db to tapes.sqlite e2428bb · 3 months ago History Preview Code Blame 48 lines (33 loc) · 1.45 KB Raw ### From skills to models Douglas explains the step behind the arrow: a skill can supply the criteria for an eval. That eval can then guide model post-training on a task such as checking whether Rust code follows a team’s conventions. [9:55](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=595s) [A right-pointing arrow connects “skills” to “models” in large black text on a white slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/34df905e6820a6dd5ed33cf9a87465137c1e2bccffe513ecf09458aa4ef041f4.webp) skills → models ### I Trained a Model Instead of Writing a Skill A skill gives an agent a workflow to read each session. In his article, Brian Douglas describes training a LoRA adapter on a frozen base model so the model could learn the pattern. He still suggests writing a skill first. [10:11](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=611s) [A browser shows the article “I Trained a Model Instead of Writing a Skill,” dated May 20, 2026. The section “Training is smaller than it sounds” describes a LoRA adapter on a frozen base model.](https://assets.ai.engineer/sites/aiecode2025/sha256/adca61d9144e9ee9524abefbe484e0890d418d2a7dcbf52c96197e217040d511.webp) I Trained a Model Instead of Writing a Skill May 20, 2026 Skills are popular right now, and worth paying attention to. A skill is a markdown file that teaches an agent a pattern. You write the workflow down once, point the agent at it, and the agent picks up the pattern without you re-explaining it every session. That works. It works well enough that I started wondering what would happen if the agent learned the pattern instead of reading it every time. That is the line between writing a skill and training a model. So I tried my hand at training a model. I did most of this while also learning what reinforcement learning actually is, which I will admit upfront. Experts would likely get better results, but you got to start somewhere, so why not now? These are my results. Training is smaller than it sounds I did not train a model from scratch. Nobody does that anymore for this kind of work, its 2024 anymore. What I trained was a LoRA adapter, which is a small set of low-rank matrices that sit on top of a frozen base model. The base model holds all the general capability. The adapter is the margin notes. Interface text: bdougie Home Blog Subscribe Newsletter ⌘K Back to all posts Intro to training on code - Go… Paper Compute - Run self he… I Trained a Model Instead of W… briandouglas.me/posts/2026/05/20/i-trained-a-model-instead-of-writing-a-skill/ Ask Gemini Work Relaunch to update ### 77k sessions with embedded skills Brian Douglas’s team had 77,000 agent sessions they could draw on to write reusable skills. [10:22](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=622s) [Large centered text reads “77k session with embedded skills.” The word “skills” appears on the second line.](https://assets.ai.engineer/sites/aiecode2025/sha256/a285ec8508c7c53007aa45ea62c5922b4ebf4d11aeca00dd130786d0349b51c6.webp) 77k session with embedded skills. ### SFT + DPO Douglas describes two ways to train a model on session data. He says direct preference optimization takes more compute, so he favors specialized fine tuning and uses quantization to train on smaller GPUs. [10:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=629s) [The heading reads “SFT + DPO.” Beneath it, the slide labels SFT “Specialized Fine Tuning” and DPO “Direct Preference Optimization.”](https://assets.ai.engineer/sites/aiecode2025/sha256/82da2af07b5ca8c5d55bc196941000e85f5c4e82932f96805d4648cd3a7710a3.webp) SFT + DPO Specialized Fine Tuning Direct Preference Optimization b.dougie.dev ### SFT with QLoRa Douglas uses QLoRa to quantize the model so he can fine-tune it on smaller GPUs, such as a 5090. [10:40](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=640s) [Large black text centered on a white slide reads “SFT = QLoRa.”](https://assets.ai.engineer/sites/aiecode2025/sha256/b27b5ccc53fb1430f63fdf89126d2a301c13317412e7c9f9bbb1722e1f27c4e9.webp) SFT = QLoRa b.dougie.dev ### SFT adapters as margin notes Douglas compares fine-tuning to writing in the margins of a book. The printed text represents the pretrained model. The handwritten notes represent the adapter weights that fine-tuning adds. [10:47](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=647s) [“SFT =” appears beside an open book with underlined passages and handwritten notes in the margins.](https://assets.ai.engineer/sites/aiecode2025/sha256/44645010bcceaf7f6d5616f2510cb6b1f91848c6b4cf63389edcf247ee52b80a.webp) SFT = THE LION, THE WITCH AND THE WARDROBE been for him. The Witch was turning our troops into stone right and left. But nothing would stop him. He fought his way through three ogres to where she was just turning one of your leopards into a statue. And when he reached her he had sense to bring his sword smashing down on her wand instead of trying to go for her directly and simply getting made a statue himself for his pains. That was the mistake all the rest were making. Once her wand was broken we began to have some chance—if we hadn’t lost so many already. He was terribly wounded. We must go and see him.” They found Edmund in charge of Mrs. Beaver a little way back from the fighting line. He was covered with blood, his mouth was open, and his face a nasty green color. “Quick, Lucy,” said Aslan. And then, almost for the first time, Lucy remembered the precious cordial that had been given her for a Christmas present. Her hands trembled so much that she could hardly undo the stopper, but she managed it in the end and poured a few drops into her brother’s mouth. “There are other people wounded,” said Aslan while she was still looking eagerly into Edmund’s pale face and wondering if the cordial 196 Margin notes: Edmund hero How old? Symbol: Lucy heals Ed[illegible] [illegible] THE HUNTING OF THE WHITE STAG would have any result. “Yes, I know,” said Lucy crossly. “Wait a minute.” “Daughter of Eve,” said Aslan in a graver voice, “others also are at the point of death. Must more people die for Edmund?” “I’m sorry, Aslan,” said Lucy, getting up and going with him. And for the next half-hour they were busy—she attending to the wounded while he restored those who had been turned into stone. When at last she was free to come back to Edmund she found him standing on his feet and not only healed of his wounds but looking better than she had seen him look—oh, for ages; in fact ever since his first term at that horrid school which was where he had begun to go wrong. He had become his real old self again and could look you in the face. And there on the field of battle Aslan made him a knight. “Does he know,” whispered Lucy to Susan, “what Aslan did for him? Does he know what the arrangement with the Witch really was?” “Hush! No. Of course not,” said Susan. “Oughtn’t he to be told?” said Lucy. “Oh, surely not,” said Susan. “It would be too awful for him. Think how you’d feel if you were he.” “All the same I think he ought to know,” 197 Margin notes: So if Aslan had [illegible] Ed [illegible] would have been lost L’s moment of weakness But what cost? [illegible] doesn’t know what JC did for him b.dougie.dev ### SFT result on Qwen 4B Brian Douglas used SFT to train an adapter for a 4-billion-parameter Qwen model. He gives the result a thumbs-up and says, “It works pretty good.” [11:03](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=663s) [A thumbs-up follows “Result → SFT on 4b parameter Qwen model,” with SFT in bold.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ec794e86d5173a5c383ba9cc3d7112e0fa753c2d1407e033f92b885dc149de2.webp) Result → SFT on 4b parameter Qwen model 👍 b.dougie.dev ### DPO: success and failure pairs A developer sweats over two buttons: copy the error into Claude or read the message. The pear emoji supplies the pun on “pair”: Douglas uses a success and a failure to illustrate direct preference optimization. [11:10](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=670s) [A sweating cartoon character must choose between copying an error into Claude and reading the error message. Beside the cartoon, “DPO = success/failure” ends with a pear emoji.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b28c128e156791b475c7e78fb677d08cbe434fc75fd7f1593dcdf6eefa1b5da.webp) DPO = success/failure 🍐 Copy and error into claude Actually read the error message JAKE-CLARK.TUMBLR b.dougie.dev ### DPO as Cliff’s Notes Brian Douglas likens specialized fine-tuning to notes in a book’s margins. For direct preference optimization (DPO), he puts a yellow Cliff’s Notes study guide on the other side of the equals sign. [11:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=689s) [“DPO =” appears beside a yellow Cliff’s Notes cover with black diagonal stripes.](https://assets.ai.engineer/sites/aiecode2025/sha256/512759db2ccad4488ec81c2c340544725fddbf148d683e646b9bc1433072b971.webp) DPO = CLIFF'S NOTES SCOTT MACLEOD ### DPO result on a 70B-parameter Qwen model Douglas gives direct preference optimization on the 70B-parameter Qwen model a thumbs-up. He says it takes at least 120 GB of VRAM and a well-structured dataset, and that paying for the hardware up front is expensive. [11:44](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=704s) [A thumbs-up marks the 70B Qwen result. A note lists 120 GB or more of VRAM, a well-structured dataset, and substantial cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/52d9684d341e14a00f18e650c80b4384ec7f828339222a09d9b3d9911796c327.webp) Result → DPO on 70b parameter Qwen model 👍 (requires 120GB+ of VRAM and really well structure dataset and lots of 💰) b.dougie.dev ### Create More Skills Douglas suggests tracing the corrections you give an agent and using them as repeated practice to build skills. The cassette points to tapes.dev as one tool for this. He also suggests using a tool you already have. [12:05](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=725s) [Under “Create More Skills,” the slide reads “Trace the data → Reinforce with learning reps.” A cassette below is labeled “tapes.dev.”](https://assets.ai.engineer/sites/aiecode2025/sha256/e49a07c894ecc094fd6e591f55d4f63e17d66ce4dbbed52490e4fc5aae848131.webp) Create More Skills Trace the data → Reinforce with learning reps. B tapes.dev Normal Bias 120μs EQ A|60 ### bdougie closing card [12:39](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=759s) [An orange closing card shows a circular portrait above the name “bdougie” and a row of social platform icons. The speaker appears in a small inset at lower right.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf88ccfc9dbc7726926c52b3301ae2b6303859b9570561603da4ea1930039d27.webp) bdougie in X DEV ## Full transcript [0:01](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=1s) Hey, folks. [0:02](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=2s) Thanks for coming to my talk. [0:03](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=3s) I'm BDougie, Brian Douglas, and I'm going to talk about if you are going to verbally abuse your agents, at least label the outcome. [0:11](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=11s) Now, this is a talk I'm going to quickly just run through one example, and this will be a short talk, mainly because there's actually not too much to this. [0:19](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=19s) But outside of that, I want to talk about some of the work I've been doing for the past six to eight months. [0:25](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=25s) And it actually came out with this idea early this year. [0:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=29s) Everyone definitely got AI-pilled when it comes to agents. [0:32](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=32s) So obviously, cloud codes have got to be one of the most popular ones out there. [0:35](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=35s) We've got quite a few other ones that you can leverage, like the Codex or ChatGPT. [0:40](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=40s) And there was a book I read years ago, well, quite a handful of years ago, from Rick Rubin about the creative act, a way of being. [0:48](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=48s) And this was actually around the idea that Rick Rubin produced the Beastie Boys and a number of other acts. [0:54](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=54s) But he's actually never played music. [0:55](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=55s) And a lot of folks have made this correlation with vibe coding, where Rick Rubin can produce the best music without actually playing the piano. [1:04](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=64s) We're in a world where folks can vibe code a bunch of Rust code without never writing Rust. [1:10](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=70s) And that's exactly where I'm at. [1:13](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=73s) There was another article months ago from Addy Osmani, who's now currently at a [1:18](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=78s) the Anthropic team, the Quadco team specifically, and was talking about the opportunity for juniors are basically, it's gone. [1:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=89s) When it comes to junior engineering, it was all about growing from junior to senior was about reps. [1:34](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=94s) So the more reps you got, the more engagement you got, opportunity to basically just level up as an engineer. [1:40](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=100s) And without those reps, we're handing those reps specifically to agents, we're losing the opportunity to level up. [1:47](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=107s) So this comes out of something I was doing earlier this year, which built this open source project called Tapes. [1:53](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=113s) Tapes.dev is open source Apache 2. [1:56](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=116s) There's a lot of other similar stuff out there, but what I'm getting at is it's not about Tapes, it's about the idea of collecting your agent session data, and that's what this does. [2:03](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=123s) So a lot of OTEL collectors, a lot of session capture services, [2:07](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=127s) A lot of the harnesses already do this for you out of the box. [2:10](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=130s) Just wanted to mention that this is what I use. [2:12](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=132s) You can use anything else you would like. [2:14](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=134s) I've also, in addition to tapes, me and my team, we've developed this thing called stereos, and this is the runtime for your agent. [2:21](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=141s) Again, sandboxes, not a concept that everyone needs to hear about today, but just knowing that in all the work I've been doing, I've been capturing these sessions and recording them by having a reproducible environment. [2:34](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=154s) And the environment themselves, they have these signals. [2:38](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=158s) And the one thing I've been doing with this little setup of tapes and stereos is I've been building self-healing infrastructure. [2:44](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=164s) So this is a little project called Sweeper Agent. [2:47](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=167s) And Sweeper Agent itself is a way for me to build parallel agents. [2:51](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=171s) I can scale up to 10 of these at this point. [2:54](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=174s) I've done quite a few Daytona sandboxes as well. [2:57](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=177s) I've done about 200 at the same time. [2:59](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=179s) That will just solve a specific problem. [3:01](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=181s) So documentation, test writing, code cleanup, stuff that you want to scale that's like very, very defined work that you want to spin off to multiple agents in tandem. [3:11](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=191s) Now, a lot of this has been cleaning up most of my vibe coding software that I built in the last 18 months at this point. [3:20](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=200s) Just pointing out that there's a lot of work we can do to basically solve these problems, but it doesn't change the fact that I still don't know how to write Rust code. [3:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=209s) I've never written Rust code specifically myself, professionally rather. [3:34](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=214s) Ten years ago, I did write Rust code, but it was mostly for little side projects to learn Rust. [3:39](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=219s) But we use Rust day-to-day at the company I work for, which is Paper Compute. [3:43](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=223s) And what we've been doing is because of this technology underneath this called [3:47](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=227s) tapes we've been capturing traces and tapes is our trace capture service telemetry capture and paper we have a console so for the moment I'll move away from the slides and show you the paper console now paper console itself this showing that I can capture sessions the difference of doing this local in my own sessions is that I can have a centralized data store if I not just me but also my teammate and what's cool about this is I can also label sessions so this is a session that [4:17](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=257s) oh interesting yeah sorry i'm gonna zoom out to see the labels but yeah so here i can see i've got a session here which uh working with some paper cloud export memory limits and we've got this word this label called pushback now i actually went to specifically a session that had pushback in it and it's specifically every time an engineer pushes back on the agent now [4:40](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=280s) There's a thing that I think recently Anthropic or I guess Rumors, I don't know if this is actually grounded by any facts, but Anthropic was pushing back on if you're being rude to the agent, you could potentially lose your account or get blocked. [4:55](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=295s) But what I find specifically when pushback, when I'm specifically looking for any Rust code or Go code that [5:01](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=301s) Matt has written. [5:03](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=303s) And whenever he pushes back, it's a good signal that he has knowledge that the agent does not. [5:08](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=308s) And this usually comes in the form of we have decided our API is structured a certain way. [5:14](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=314s) Despite the fact that the Rust code compiles, we have made decisions on how that code should be set up and how that code should interact with different parts of the code base. [5:23](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=323s) Also pushing back on things like duplicating services that maybe the agent didn't have context around. [5:28](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=328s) There's a bunch of reasons, but what I'm getting at, me as a quote-unquote junior Rust engineer, I don't have enough context to be able to push back to the agent to say, hey, you're doing this wrong or you're making weird choices. [5:41](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=341s) I usually take everything that the agent writes in Rust for face value, and what that ends up turning into is when I open up PR, I get a lot of pushback from the engineers, the human engineers, at review time. [5:52](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=352s) This is mainly because we know I don't write Rust code. [5:54](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=354s) There's not a lot of pushback I can provide to the code base. [5:58](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=358s) or even at the time of review. [5:59](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=359s) So by the time it gets to greptile, it's a five out of five review. [6:03](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=363s) I just let it slide. [6:04](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=364s) So how do I prevent that pushback? [6:06](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=366s) What I've done is actually started collecting every single one of these labels and the traces. [6:11](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=371s) And then one thing I'll point out is that [6:13](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=373s) The console itself, I capture every single session. [6:17](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=377s) I've got all the specific details. [6:19](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=379s) This is actually just a command line version of that. [6:21](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=381s) What I've been doing is I've been capturing all these sessions and then dropping into a data set that's been labeled. [6:28](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=388s) So now I can just run a little command line tool called Twin Review. [6:31](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=391s) And then I'll grab a PR number, 381, and I'll run that. [6:36](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=396s) And I'll actually run through and... [6:38](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=398s) essentially review the code as if it was Matt. [6:42](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=402s) Now, the way I'm doing this is I'm capturing traces, I created the data set, I've also done some post-training, and this is like where, you know, choose your own adventure, but I've done some post-training specifically on every time Matt pushes back within Go or Rust code, and I asked for it to provide me comments as if you were Matt [7:00](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=420s) on my GitHub PR. [7:01](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=421s) So this is all data that's only specifically within the traces. [7:04](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=424s) So this is not taking any GitHub PR review data, which I think you potentially could also follow that pattern. [7:10](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=430s) But what I find is like the majority of our team have started reviewing code on the command line. [7:15](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=435s) So you'll have Claude review the code or the PI hardest review code, or you review this PR, but you don't do a lot of pushback or a lot of commentary on GitHub itself. [7:24](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=444s) So [7:25](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=445s) What we're seeing right now, we're shifting from code being reviewed specifically on GitHub by agents, well, previously humans and now agents, to now code is specifically just getting reviewed in your harness without any sort of oversight or commentary or anything that sort of navigates directly to your shared code base. [7:46](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=466s) So with that being said, what I've navigated is I've created this thing called a twin reviewer. [7:51](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=471s) And this as well is open sourced. [7:54](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=474s) So... [7:55](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=475s) It is, but I did not have this open. [8:00](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=480s) Oh, actually, TwinReviewer is not open source. [8:02](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=482s) I realized I never open sourced it because I got sensitive data. [8:06](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=486s) But what I've done is actually I set up that same pattern using Fireworks and using Fireworks to post-train specifically. [8:12](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=492s) And what's cool about this is I've looked for any sort of pushbacks, apologies, any sort of golden sessions, sessions that actually have been marked as you should probably pay attention to this, and also regressions for any things that [8:25](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=505s) Usually agents will just fall over. [8:27](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=507s) A lot of times it turns into like an agent will start reinventing Go libraries that already exist. [8:34](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=514s) That usually is granted to be pushed back. [8:36](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=516s) Ideally, you should push back on that. [8:38](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=518s) But again, if you don't know these libraries exist, you probably take it for face value and then you're reinventing things like Kubernetes, which sounds like an overstatement, but we've seen it reinvent things like mock libraries for testing and et cetera. [8:51](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=531s) And [8:52](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=532s) very popular code base or very popular open source projects. [8:55](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=535s) So that's essentially it. [8:57](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=537s) This is basically, if you're going to push back and you're going to have some really, really valuable insight under reviews and your code base, my recommendation is take a look at opportunities within the traces to maybe train or post-train a model or create your own little harness for reviews locally. [9:15](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=555s) Choose your own adventure. [9:16](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=556s) I think you could actually do a lot of this by just building your own harness. [9:19](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=559s) But if you did want to use a, [9:22](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=562s) which I've done specifically in this situation, you'd also approach that. [9:28](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=568s) A lot of the stuff is available to you, a lot of tools. [9:32](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=572s) What I did actually to start is I started taking the paper traces and creating skills, skills that I can then specifically do things like check the tapes. [9:40](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=580s) So if you have a database, which tapes specifically is a Postgres database local on my machine, [9:45](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=585s) So you don't have to use a paper console. [9:46](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=586s) Actually, paper console is currently waitlisted. [9:49](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=589s) So probably best use case is going to use something like tapes or a brain trust or something similar to capture your traces. [9:57](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=597s) And then what's cool about this is once you have skills, you can create evals. [10:00](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=600s) Once you have evals, you can start post-training on a Quinn model and for specific reasons, for identifying if Rust code is up to speed to how you're writing code across your organization. [10:12](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=612s) So I did write a blog post around training a model instead of writing a skill. [10:16](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=616s) My recommendation is probably write a skill first. [10:18](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=618s) You can probably get most mileage out of a skill today. [10:22](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=622s) And I specifically across my team, we had about almost 80,000 sessions that we could [10:29](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=629s) create skills from which is pretty cool uh cool thing about this is also you do specialized fine tuning or direct preference optimization the latter is going to take a lot more compute or a larger some large hardware to do that so probably specialized by tuning uh if you know about quick q or if that's how i'm doing this i'm quantizing it on smaller gpus like a 50 90. [10:48](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=648s) And if you don't know about specialized fine tuning, this is like writing notes in the margin of a book. [10:53](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=653s) If your book is your model, the notes in the margin are your weights, or your adapters, rather. [10:59](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=659s) So the weight you generate on top of the model that's already been pre-trained. [11:04](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=664s) Cool. [11:04](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=664s) And I did this, as you saw, on a Quinton 4B model. [11:08](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=668s) It works pretty good. [11:09](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=669s) I did try to do this on some larger models. [11:13](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=673s) So specifically for direct preference optimization, it took a lot of compute. [11:17](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=677s) The only time I could make this work is by using NVIDIA GPUs on the Brev. [11:22](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=682s) service or actually I never got access to some of the stuff on Fireworks, so I only did smaller 6000s. [11:30](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=690s) But if you care to know direct preference optimization, if the margin notes are your specialized fine tuning, DPO is gonna be like your cliff notes. [11:40](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=700s) Not possible doing a small language model or a small 4B model. [11:46](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=706s) You would need like 70B+, which is going to take some hardware. [11:50](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=710s) Me specifically, I didn't have any as hardware local, so I had to use the services that I mentioned, Fireworks and, well, not even Fireworks. [11:57](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=717s) Brev was the only one I could get access to at the time. [12:00](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=720s) Good luck if you can get access to larger, well, you got to prepay a lot for this stuff, so. [12:05](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=725s) Best of luck on that one. [12:07](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=727s) But my suggestion is create skills. [12:09](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=729s) Try out tapes.dev if you're interested or leverage the tool that you have at your ready. [12:14](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=734s) And reminder, you should probably read this book. [12:17](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=737s) This is a great book if you think about, you know, how we're looking forward into... [12:21](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=741s) how we're approaching this world when it comes to AI and vibe coding and leveraging agents to basically build your software factory. [12:28](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=748s) And I do recommend, please check out Adi Asmani's blog post. [12:33](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=753s) He has a series of these on the X. I think they're all great. [12:36](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=756s) Definitely take a look at them. [12:36](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=756s) He's got some clear-minded dots to check out. [12:41](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=761s) I'm B. Dougie, and I do appreciate you watching this video. [12:43](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=763s) I try to keep it brief because it was pretty straightforward. [12:46](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=766s) Hopefully it was. [12:47](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=767s) Cheers. [12:47](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=767s) Cheers.