I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft

Keiji Kanazawa had more podcasts he wanted to hear than hours to listen. The Microsoft product manager, who started as a developer and had worked in AI research, had a practical reason to care about the debate over reading agent-written code. At one end of what he calls the L/Z spectrum, a builder delegates freely and spends the tokens to get the work done. At the other, a builder slows down and reads the code. Kanazawa put himself firmly at the delegation end. His podcast backlog gave him a reason to try it.

The first version downloaded podcasts, transcribed them and summarized the text. Then it started crawling X threads and linked pages. A scheduler kept the collection running, and an email inbox gave Kanazawa a convenient way to send in a link and get a summary back. What began as a few skills had grown into an LLM wiki with a steady supply of material.

Downloading a YouTube video and transcribing it had a known sequence of steps. The new jobs called for judgment. One would do deep research using the wiki. Another would look back over his coding-agent sessions each day, suggest improvements to the agents’ instructions and turn recurring work into skills. Kanazawa specified a runtime with a registry of agents he could create and run on schedules, including one to monitor the system itself. Claude Code and Codex took turns implementing changes, reviewing each other’s code and fixing bugs.

One day a digest put a sentence in the wrong section, attributing material to the wrong source. Kanazawa wanted to find where the mix-up happened and use the failure to design an evaluation. He asked Claude Code for the trace of that particular session. There was no agent trace to inspect. Reading the code, he found that the digest still ran through the old deterministic workflow. He also recognized an uncomfortable detail about the design he disagreed with. He had approved it.

What did adding a skill actually change? The original workflow had steps that invoked actions such as downloading a video or transcribing it. The refactor gave skills and actions two names for the same thing. Agents appeared as a separate concept in the design, yet underneath Kanazawa found the same machinery again. A supposed agent skill had neither the SKILL.md nor the script he expected to find. The agents ran without an agent SDK. The plans he had approved used familiar terms like agent skills, and he had let those words stand in for an implementation decision.

While Kanazawa was working with a coding agent on repairs, a token provider emailed him. It was the first week of the month, and he had already passed his normal monthly budget. Two checks revealed very different problems.

• Unit tests. Codex had wired the tests to production inference. Running the suite therefore made real model calls, and the agent had run about 30 test cycles in the day before the alert. Spending had reached roughly $100, about five times his usual weekly spend.
• Daily reflection. The scheduled job was supposed to read coding sessions and suggest improvements. It kept running, but opening the reports revealed empty content. The skill/action/agent refactor had broken the path that invoked the prompts. The schedule could still fire even though the work that gave the report its value never ran.

After three weeks of asking the coding agents to refactor this system, Kanazawa had little reason to commission another pass over the same codebase. He decided to start fresh. He still wanted the agents to write the code.

The rebuild started with fresh specifications and goal rubrics that described the results the agents should produce. The new runtime treated each job as an agent and kept a registry of the different agents. That made the distinction missing from the first refactor concrete enough to try in a working interface.

Which agent was responsible for what the user saw? In the middle of the interface, Kanazawa could give a coding-agent SDK instructions for an agent he wanted built. The SDK created that agent. The new agent then had a canvas on the right where it could put the results of its own work. That division gave him a common task for a four-framework comparison. For the Claude version, he asked Claude Code to build the experience around the Claude Agent SDK. For the Codex version, he asked Codex to use itself to create and run the agents.

He settled on Claude Code and Codex for personal projects. Pi illustrates what mattered in that choice. It quickly produced an interface he liked, then asked him to make decisions about the SQLite database tracking the agents and the design of the TypeScript APIs. An engineer who wanted a say in those details might welcome the questions. As a product manager, Kanazawa wanted to spend that attention on what the experience should do. With his tools chosen, he added transcripts and summaries back into the system and built evaluation harnesses alongside them.

The old YouTube summarizer only read transcripts. Then Kanazawa came across a blackboard lecture about LLM inference on GPU clusters. Roofline analysis was relevant to his day job, and some of the math he wanted to understand was on the board. His video deep dive agent extracted about 440 keyframes from that lecture. It used an AI evaluation rubric to select shots, pulled out the blackboard equations and incorporated them into the canvas.

What could he do when he got stuck on one equation? In the demonstration, the canvas divided the lecture into chapters and showed the equations with explanations. He could jump from a section back to that part of the video, or highlight an equation and ask the agent to explain that particular part in more detail. The chapter gave him a way back to the lecturer’s reasoning, and the selection let him ask a follow-up without having to describe which piece of mathematics he meant.

Kanazawa’s earlier practice had been to give an agent a specification and approve or reject its plan. For the deep dive, he also described how to judge the finished canvas. He wanted simple, well-explained material and links that landed at the right moments in the video. The same reading experience could work for a slide deck, a coding demonstration in VS Code or Cursor, or a terminal recording. He worked out much of what he wanted on the one blackboard lecture.

The agents wrote the rubric under his direction. Once he set it, they could hill-climb against it, iterating on the output to better satisfy the criteria. He would then read the resulting canvas. An early version put all the blackboard equations in a clump at the beginning, followed by a wall of explanatory text. It had recovered both ingredients and left the reader to match them up.

His correction specified a relationship the original criteria had missed. Each equation should appear with its own explanation, and the pairs should follow the lecture’s order. He asked the agents to put that requirement into the rubric. The next pass would then have a reason to preserve the pairing whenever it generated a canvas.

Kanazawa had been reading the canvases more closely than the rubrics. One agent-written rubric was tailored so tightly to the inference lecture that it was easy to succeed on that example. Running it on five other videos exposed the problem. Performance was poor away from the lecture he had used to develop it. He directed the agent to revise the criteria so they would apply beyond that one video. He reports better scores across the videos after the change, without quantifying the improvement.

When a canvas disappoints, what should change on the next pass? Kanazawa had two ways to act on the result. With a rubric held fixed, the agents could keep improving the canvas against its criteria. When he saw a layout that made him hunt for an explanation, or a rubric that worked on only one lecture, he could change the criteria themselves. That change gave subsequent runs a different target.

Code review served a separate purpose. Kanazawa still distrusted individual implementation decisions, so he had Claude and Codex inspect each other’s work closely and suggest fixes over several rounds. Reading the code himself had exposed the failed refactor. It had also helped him decide where he wanted to spend his own time afterward. He still delegates most implementation and rarely reads every line. What he keeps is the decision about what an agent should do, how its output should work for him, and when an evaluation that looks successful deserves another test.