# I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft Keiji Kanazawa, Microsoft [Talk page](https://ai.engineer/talks/6TjFjHPjVH0-i-let-agents-refactor-my-codebase-3) · [Video](https://www.youtube.com/watch?v=6TjFjHPjVH0) ## I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. A misplaced sentence sent Keiji Kanazawa looking for a trace his research assistant couldn’t produce. Rebuilding it taught him how to delegate the code while keeping a hand in the harder decision of what counts as a good result. ### A podcast downloader grows into a personal research assistant [0:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=0s) Keiji Kanazawa had more podcasts he wanted to hear than hours to listen. The Microsoft product manager, who started as a developer and had worked in AI research, had a practical reason to care about the debate over reading agent-written code. At one end of what he calls the L/Z spectrum, a builder delegates freely and spends the tokens to get the work done. At the other, a builder slows down and reads the code. Kanazawa put himself firmly at the delegation end. His podcast backlog gave him a reason to try it. The first version downloaded podcasts, transcribed them and summarized the text. Then it started crawling X threads and linked pages. A scheduler kept the collection running, and an email inbox gave Kanazawa a convenient way to send in a link and get a summary back. What began as a few skills had grown into an LLM wiki with a steady supply of material. Downloading a YouTube video and transcribing it had a known sequence of steps. The new jobs called for judgment. One would do deep research using the wiki. Another would look back over his coding-agent sessions each day, suggest improvements to the agents’ instructions and turn recurring work into skills. Kanazawa specified a runtime with a registry of agents he could create and run on schedules, including one to monitor the system itself. Claude Code and Codex took turns implementing changes, reviewing each other’s code and fixing bugs. ### One misplaced sentence exposes the old workflow [4:02](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=242s) One day a digest put a sentence in the wrong section, attributing material to the wrong source. Kanazawa wanted to find where the mix-up happened and use the failure to design an evaluation. He asked Claude Code for the trace of that particular session. There was no agent trace to inspect. Reading the code, he found that the digest still ran through the old deterministic workflow. He also recognized an uncomfortable detail about the design he disagreed with. He had approved it. What did adding a skill actually change? The original workflow had steps that invoked actions such as downloading a video or transcribing it. The refactor gave skills and actions two names for the same thing. Agents appeared as a separate concept in the design, yet underneath Kanazawa found the same machinery again. A supposed agent skill had neither the `SKILL.md` nor the script he expected to find. The agents ran without an agent SDK. The plans he had approved used familiar terms like *agent skills*, and he had let those words stand in for an implementation decision. ### The tests spent real tokens, and the daily reports came back empty [6:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=388s) While Kanazawa was working with a coding agent on repairs, a token provider emailed him. It was the first week of the month, and he had already passed his normal monthly budget. Two checks revealed very different problems. - **Unit tests.** Codex had wired the tests to production inference. Running the suite therefore made real model calls, and the agent had run about 30 test cycles in the day before the alert. Spending had reached roughly $100, about five times his usual weekly spend. - **Daily reflection.** The scheduled job was supposed to read coding sessions and suggest improvements. It kept running, but opening the reports revealed empty content. The skill/action/agent refactor had broken the path that invoked the prompts. The schedule could still fire even though the work that gave the report its value never ran. After three weeks of asking the coding agents to refactor this system, Kanazawa had little reason to commission another pass over the same codebase. He decided to start fresh. He still wanted the agents to write the code. ### Four frameworks get the same agent-building task [8:19](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=499s) The rebuild started with fresh specifications and goal rubrics that described the results the agents should produce. The new runtime treated each job as an agent and kept a registry of the different agents. That made the distinction missing from the first refactor concrete enough to try in a working interface. Which agent was responsible for what the user saw? In the middle of the interface, Kanazawa could give a coding-agent SDK instructions for an agent he wanted built. The SDK created that agent. The new agent then had a canvas on the right where it could put the results of its own work. That division gave him a common task for a four-framework comparison. For the Claude version, he asked Claude Code to build the experience around the Claude Agent SDK. For the Codex version, he asked Codex to use itself to create and run the agents. He settled on Claude Code and Codex for personal projects. Pi illustrates what mattered in that choice. It quickly produced an interface he liked, then asked him to make decisions about the SQLite database tracking the agents and the design of the TypeScript APIs. An engineer who wanted a say in those details might welcome the questions. As a product manager, Kanazawa wanted to spend that attention on what the experience should do. With his tools chosen, he added transcripts and summaries back into the system and built evaluation harnesses alongside them. ### A blackboard lecture gives the video agent something to look for [11:55](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=715s) The old YouTube summarizer only read transcripts. Then Kanazawa came across a blackboard lecture about LLM inference on GPU clusters. Roofline analysis was relevant to his day job, and some of the math he wanted to understand was on the board. His video deep dive agent extracted about 440 keyframes from that lecture. It used an AI evaluation rubric to select shots, pulled out the blackboard equations and incorporated them into the canvas. What could he do when he got stuck on one equation? In the demonstration, the canvas divided the lecture into chapters and showed the equations with explanations. He could jump from a section back to that part of the video, or highlight an equation and ask the agent to explain that particular part in more detail. The chapter gave him a way back to the lecturer’s reasoning, and the selection let him ask a follow-up without having to describe which piece of mathematics he meant. ### Getting each equation next to its explanation [14:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=840s) Kanazawa’s earlier practice had been to give an agent a specification and approve or reject its plan. For the deep dive, he also described how to judge the finished canvas. He wanted simple, well-explained material and links that landed at the right moments in the video. The same reading experience could work for a slide deck, a coding demonstration in VS Code or Cursor, or a terminal recording. He worked out much of what he wanted on the one blackboard lecture. The agents wrote the rubric under his direction. Once he set it, they could hill-climb against it, iterating on the output to better satisfy the criteria. He would then read the resulting canvas. An early version put all the blackboard equations in a clump at the beginning, followed by a wall of explanatory text. It had recovered both ingredients and left the reader to match them up. His correction specified a relationship the original criteria had missed. Each equation should appear with its own explanation, and the pairs should follow the lecture’s order. He asked the agents to put that requirement into the rubric. The next pass would then have a reason to preserve the pairing whenever it generated a canvas. ### Five other videos expose an overfit rubric [16:35](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=995s) Kanazawa had been reading the canvases more closely than the rubrics. One agent-written rubric was tailored so tightly to the inference lecture that it was easy to succeed on that example. Running it on five other videos exposed the problem. Performance was poor away from the lecture he had used to develop it. He directed the agent to revise the criteria so they would apply beyond that one video. He reports better scores across the videos after the change, without quantifying the improvement. When a canvas disappoints, what should change on the next pass? Kanazawa had two ways to act on the result. With a rubric held fixed, the agents could keep improving the canvas against its criteria. When he saw a layout that made him hunt for an explanation, or a rubric that worked on only one lecture, he could change the criteria themselves. That change gave subsequent runs a different target. Code review served a separate purpose. Kanazawa still distrusted individual implementation decisions, so he had Claude and Codex inspect each other’s work closely and suggest fixes over several rounds. Reading the code himself had exposed the failed refactor. It had also helped him decide where he wanted to spend his own time afterward. He still delegates most implementation and rarely reads every line. What he keeps is the decision about what an agent should do, how its output should work for him, and when an evaluation that looks successful deserves another test. Which feedback changes the canvas, and which changes the rubric? An evaluation can send the agents around another pass with the same criteria. Reading the output and trying other videos can send Kanazawa back to the criteria that guide every pass. - Goal rubric — The agent writes criteria under Kanazawa’s direction, including how equations and explanations should appear together. - Generated canvas — The video agent produces a reading view for the lecture. - Assess against the current rubric — The agents optimize the output with the target held fixed. - Inspect the result and test its reach — Kanazawa reads the layout. Trying five other videos also reveals criteria specific to one lecture. - Goal rubric → Generated canvas (Guides generation) - Generated canvas → Assess against the current rubric (Supplies the output to judge) - Assess against the current rubric → Generated canvas (Guides another attempt at the same target) - Generated canvas → Inspect the result and test its reach (Shows what the reader actually gets) - Inspect the result and test its reach → Goal rubric (Changes what future runs are judged on) ## Key takeaways - A request for the trace of one bad digest exposed a mismatch that approving plans had missed. The new architectural names still led to the old workflow. - The most useful canvas criterion described how two pieces of information belonged together. Each blackboard equation needed its explanation beside it, in lecture order. - An agent can improve against a rubric that rewards the wrong thing. Five other videos gave Kanazawa a way to test the rubric itself before asking the agents to optimize further. ## Slides ### I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. Keiji Kanazawa gave coding agents three weeks to refactor his codebase. The final line shifts from white to gold as Kanazawa takes over: “Then I Read the Code.” [0:18](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=18s) [Keiji Kanazawa’s title slide highlights “Then I Read the Code.” in gold. The speaker appears in a lower-left inset.](https://assets.ai.engineer/sites/aiecode2025/sha256/23e888b5051a9eb0dcf8ac9ac6894d0dcc60ca5a7744a78085669383d014cf28.webp) I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. ◆ AI ENGINEER WORLD'S FAIR · 2026 Keiji Kanazawa · @gojira AG · L → Z → ? 01 / 22 AI Engineer World’s Fair ### Started as a skill, snowballed into an agent. Kanazawa used to paste YouTube links into PodExplore for summaries. With more sources and a 24/7 scheduler, it grew into AG, a wiki he could search and use for deep research. He could also email it a link and get a summary back. [2:42](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=162s) [Below the PodExplore workflow, eight labels list new sources and features, including papers, repositories, scheduling, digests and email. A closing line names the searchable LLM wiki AG. Its three-column dashboard is shown on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/161b49734eb202e3707e6ec1be091980b398be0b354da41b6911fd85281af387.webp) WHERE IT ALL STARTED Started as a skill, snowballed into an agent. "PodExplore": download YouTube → transcribe → summarize. Run it manually — paste the URLs in. X + web articles arXiv papers GitHub repos 24/7 scheduler daily + weekly digests email in + out self-healer coding lessons An LLM Wiki I search and run deep research over. I renamed it AG. AG · L → Z → ? 04 / 22 Interface text: AG Latest Channels Digest Queue Agents Eval Latest Filter: Filter by title, guest, or channel... YOUTUBE (655) Latent Space Ready 2026-06-21 ⚡ Every product of the future will be a living system — Ronak Malde, Trajectory.ai Guest: Ronak Malde [illegible] chapters · [illegible] Lenny's Podcast Ready 2026-06-21 Building the most AI-pilled engineering team in the world | Fiona Fung (Anthropic) Guest: Fiona Fung TWEETS (2328) @SakanaAILabs Ready [illegible] Introducing Sakana Fugu: A full multi-agent... @levie Ready [illegible] Agents will use software 100X more than people. When... @levie Ready [illegible] Hugely underrated point. You basically state this, but... WEB (1672) web_articles Ready [illegible] Claude Code and Claude Agent SDK web_articles Ready [illegible] Web Dev (Agentic) web_articles Ready [illegible] Opus 4.8's Hyperfocus on Agents is Making It Worse at Design ### I want everything to be an agent Kanazawa’s spec put independent agents in charge of turning email, RSS and web inputs into emailed digests. The agents could run on different SDKs and use the existing deterministic workflows as skills. [3:11](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=191s) [A flow diagram runs from email, RSS and web inputs through agents and deterministic workflows to database and search, then to emailed digests.](https://assets.ai.engineer/sites/aiecode2025/sha256/ef521d03dae05dcc9912ecdeafa8f3fc814a3c1e8739d50973d829a998e795ef.webp) THEN I GOT AMBITIOUS "I want everything to be an agent." So I wrote a spec for a real agent runtime — hardcoded workflows → agent skills, in independent agents I could hand to different agent SDKs. INGEST email poll · RSS · web ▼ OPERATING MODE Agents promoted to primary — but the deterministic workflows stay. Agents · primary Workflows · deterministic ▼ STATE database · search ▼ OUTPUT digests → email send AG · L → Z → ? 05 / 22 ### The missing agent SDK trace Kanazawa asked Claude Code for a turn-by-turn trace of one session. Claude Code explained why it had no trace to show. The code wasn’t using the agent SDK he thought he’d asked for. [4:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=280s) [The speaker asks for a turn-by-turn trace. The agent says the implementation does not use the SDK, and the speaker recalls explicitly requesting it.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c6817710e153c7cf6cd4fc367b533613b1268f47fe7e0b75a34701a291f9bb2.webp) ◆ THE REALIZATION ME "Show me the agent SDK trace. Turn by turn." AGENT "We don't have that. This isn't really an agent — it's not using the SDK." ME "...I thought I told you to use the agent SDK." AG · L → Z → ? 06 / 22 ### Reconsidering an approved design When Claude Code mentioned “agent skills,” Kanazawa told it to keep going. At 9 PM, after reading the code more closely, he told Claude Code he disagreed with the design and conceded he had approved it. [4:57](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=297s) [Large italic text reads “I don't agree with this design at all — that I undoubtedly approved.” The attribution identifies it as the speaker's remark to Claude Code at 9 PM.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d728d7ea6be4a6335630469018ffd5a12f4985aeba321ea90140f61adcc38ef.webp) "I don't agree with this design at all — that I undoubtedly approved." — ME, TO CLAUDE CODE · 9 PM AG · L → Z → ? 07 / 22 ### Building AG2 with the Claude Agent SDK Kanazawa started fresh with AG2, specifying the agent’s goal and how to judge its work. In the center chat, he asks the agent to turn a purple page orange. The canvas on the right shows the new color with “Hello, ag2!” still centered. [8:49](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=529s) [Text specifies a Claude Agent SDK agent, a spec, a rubric and a goal. Beside it, an AG2 screenshot shows a session list, a chat and an orange canvas reading “Hello, ag2!”](https://assets.ai.engineer/sites/aiecode2025/sha256/8e590b3f6efeac8039465ac2fec4e26ea7e6bb03d05e817860e4cb0b31291523.webp) WHAT I DID NEXT I built a new head: AG2. No grinding refactor. Now when I want a real agent, I'm explicit: “A Claude Agent SDK agent. Here's the spec. Here's the rubric. Set the goal.” Build me index.html with a purple gradient and Hello, ag2! centered. Done! I've created an index.html file with a purple gradient background (indigo to violet) and “Hello, ag2!” centered on the page. The page uses flexbox for perfect centering and is fully responsive. Let's make it orange Done! Changed the gradient to a warm orange (from peachy-orange to deep orange-red). Hello, ag2! AG · L → Z → ? 11 / 22 Interface text: Latest Channels Digest Build Runtime ADD SOURCE Search feeds, channels, episodes... LIGHT SESSIONS NEW Filter sessions... The math behind how LLMs are trained and served – Reiner Pope — canvas RLM Agent Poke Untitled session Circle [illegible] Space invader demo sandcheck HIDE WORK ARTIFACT index.html OPEN index.html added Message the agent [illegible] ATTACH SEND ### Agents as first-class Building artifacts and browsing content share one Runner interface. The Claude Agent SDK runs agents today, with slots for three other runtimes. Each turn gets a trace, a cost and an evaluation. [11:32](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=692s) [Build and Consume connect to one registry and Runner interface. The Claude Agent SDK is checked, with three other runtime slots alongside it. Lower rows show evaluation, adversarial review, read-only connectors, per-turn tracing and costs, and SQLite storage.](https://assets.ai.engineer/sites/aiecode2025/sha256/5a20aa3efcee67ec86280e254a8f047412917ee3f8af843541ee154304d7cc09.webp) WHAT I WANTED ALL LONG Agents as first-class. BUILD chat + artifact canvas CONSUME browse · read · search AGENT RUNTIME one registry · one Runner interface — the runtime an agent runs on Claude Agent SDK ✓ Codex app-server Copilot CLI Pi EVAL HARNESS gates → 9-cat judge → failure tags ADVERSARIAL REVIEW build → attack → fix → re-review · different models CONNECTORS read-only: vault · corpus · search HEADLESS RUNTIME every turn traced + costed STORE SQLite + shared corpus Claude SDK today; the rest are slots behind one interface. Every turn traced, costed, judged. AG · L → Z → ? 13 / 22 ### I can't just watch the idea go by. Kanazawa had been summarizing YouTube transcripts. Reiner Pope’s lecture drew him to the video frames so he could follow the blackboard math. Inference takes at least as long as the slower of memory access and computation. [12:11](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=731s) [Text about wanting to understand inference math appears beside a still from Reiner Pope's blackboard lecture. The board compares memory time and compute time.](https://assets.ai.engineer/sites/aiecode2025/sha256/553db1a30079bf5992873217df9431b764ef5965a0c856e37d903b6f7e453524.webp) GOING DEEPER I can't just watch the idea go by. An AI research background, from before all of this. When someone like Reiner Pope works through inference math on a blackboard, I want to grok it. t ≥ max(t_mem, t_compute) t_compute = (B · N_active) / FLOPS How GPT, Claude, and Gemini are actually trained and served — Reiner Pope AG · L → Z → ? 14 / 22 Interface text: Premium Search Dwarkesh Patel 1.34M subscribers ### The rubric Kanazawa set the goals for a video deep dive, and agents wrote nine rubric files. Two excerpts show how the rubric judges the result: can readers find the evidence, and does the analysis explain a causal chain beyond what the summary says? [14:02](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=842s) [Four questions assess mechanism, citations, coverage and where to explore next. Two excerpts show how the rubric scores page usability and analysis that goes beyond a summary.](https://assets.ai.engineer/sites/aiecode2025/sha256/200c0b4ae5f1a1147111209de75f6eb57e7fc2a8652f87df2b767d03da573784.webp) The rubric. ◆ HOW I DO L NOW MECHANISM · 18 "Explains the mechanism — or just names it?" CITATIONS · 13 "Can I trust every important claim?" COVERAGE · 12 "Did it keep what the summary already knew?" DIG DEEPER · 12 "If I want to go further — where next?" prompts/eval/rubric/html_usability.md Score: scanability, visual density, wide-layout use, timeline/navigation, collapsible technical depth, evidence drawers… Penalize a tall wall of stacked figures/equations above the fold. prompts/eval/rubric/analytical_surplus.md Reward unifying thesis, second-order implications, causal chains, real contradictions/tensions, falsifiable predictions… Penalize summary-rephrased-not-transformed. Two of 9 rubric files — all agent-written, near-verbatim from the AG2 repo AG · L → Z → ? 16 / 22 ### Then I hill-climb to the bar The agents use failure tags to revise the prompt or schema until the canvas scores about 90/100. When Kanazawa found the rubric favored one episode, he had the agents revise the rubric itself. Scores rose by 3 to 11 points. [15:38](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=938s) [A loop generates output, runs checks, scores samples across nine categories and uses failure tags to revise the prompt or schema. The target is about 90 out of 100.](https://assets.ai.engineer/sites/aiecode2025/sha256/4caef8417a0bef3cd71753de8c83bca25d8698be16dd8e75a05286ba648fd8d1.webp) ◆ LOOPING Then I hill-climb to the bar. # until score ≥ ~90 / 100 generate → gates (free) → judge (9 cats × 3 samples) → failure tags → fix prompt / schema → repeat The loop even feeds its own errors back: "The previous canvas.json failed validation. Rewrite a complete, corrected canvas.json now." A rubric can be gamed — I caught the judge over-rewarding an episode I'd steered, de-benchmaxed it, and real scores moved +3 to +11. AG · L → Z → ? 17 / 22 ### Claude and Codex review each other Claude reviews Codex’s changes. Codex writes a test before fixing the issue, then a fresh Claude checks the fix. The first pass caught 3 high-severity bugs before shipping across roughly 3,000 lines in 45 files. [17:30](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1050s) [Codex builds, Claude reviews the changes, Codex makes a test-first fix and a fresh Claude reviews again. The slide reports three high-severity bugs caught on the first pass.](https://assets.ai.engineer/sites/aiecode2025/sha256/166967fbff6adbe3dd8306ddad573597b23442ad28fea9e7260767c909a8bb35.webp) ◆ MORE LOOPS Claude and Codex review each other. I don't referee. Codex builds Claude attacks the diff Codex fixes — test first fresh Claude re-reviews # complete only if adversarial_review_status in (passed, waived) First pass: 45 files · ~3,000 lines · 3 high-sev bugs caught before ship. One real cycle discarded a complete answer because the run's SDK stop reason looked like truncation. The review question that caught it: which stop states are real failures — and which are just warnings? AG · L → Z → ? 18 / 22 ### Own the goal. Own “done.” Kanazawa rarely reads the agents’ code. He puts his effort into writing clearer specs and rubrics that tell the agents what to build and how he’ll judge when the work is done. [18:52](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1132s) [A three-line heading ends with “Not every line.” The paragraph below emphasizes specifications, rubrics and a clear definition of done.](https://assets.ai.engineer/sites/aiecode2025/sha256/b1cfd0f69722459ba1dba02d30859be20009e773dc9e4cbf0c0f6028f9bc3a2b.webp) ◆ WHERE I LANDED Own the goal. Own "done." Not every line. I'm never getting to Z — and that's the right call. The realization didn't move me toward Z. It made me sharper at L: specs, rubrics, a clear definition of done. AG · L → Z → ? 19 / 22 ### Tokenmaxxing is what makes a Product Manager an AI Engineer Gold type links “Tokenmaxxing” to “AI Engineer.” Kanazawa makes the point personal: as a product manager, he credits AI tools with helping him build experiences he’d wanted to create for years. [19:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1140s) [A centered statement reads “Tokenmaxxing is what makes a Product Manager an AI Engineer.” Gold type highlights “Tokenmaxxing” and “AI Engineer.”](https://assets.ai.engineer/sites/aiecode2025/sha256/03dad78f9e062830edd68b6c33125b32c49d7298d3ca942d00b610794c961cd3.webp) Tokenmaxxing is what makes a Product Manager an AI Engineer. AG · L → Z → ? 20 / 22 ## Full transcript [0:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=0s) All right, welcome. [0:15](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=15s) Everybody, I'm Keiji Kanazawa. [0:19](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=19s) I let my coding agents refactor my code base for three weeks. [0:25](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=25s) And then I read the code. [0:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=28s) Listen on for what happened after that. [0:33](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=33s) So most of you here in this track probably are aware of the Lopopolo Zechner debate. [0:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=40s) How many people don't know what that is? [0:44](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=44s) Okay, so Lopopolo Zechner, back in AI engineering Europe a few months ago, Lopopolo, he works at OpenAI. [0:53](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=53s) He said, reading code the liability. Just spend billions of tokens to get stuff done. [0:59](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=59s) Mario Zechner at the same conference said, read the fucking code. Slow the fuck down. [1:07](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=67s) Okay, so that's the LZ spectrum. [1:10](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=70s) How many people here are like on the L side? [1:15](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=75s) Who's on the Z side? [1:17](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=77s) All right, awesome. [1:19](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=79s) So I am very, very much on the L end. [1:23](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=83s) Now, I'm a product manager. [1:25](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=85s) I also have an AI research background. [1:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=88s) I was started off as a dev. [1:30](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=90s) But these days I work on product. [1:34](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=94s) And back last December, like a lot of people, [1:39](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=99s) Clock Code, Opus 4.5, really amazing moment, right? [1:42](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=102s) When things just started to work. [1:44](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=104s) So I have a day job that's pretty busy. [1:47](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=107s) I work at Microsoft, work with the Frontier Labs, OpenAI and Anthropic to provide their models to the Microsoft customers. [1:58](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=118s) So, but I love to listen and watch the podcasts. [2:01](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=121s) I never have time. [2:03](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=123s) So I set up the, you know, a project to create some skills to download podcasts, transcribe them, summarize. [2:11](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=131s) Many of you provide them the same thing. [2:13](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=133s) But this project then snowballed. [2:16](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=136s) I added X threads. [2:18](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=138s) It would go in, crawl X, you know, topics as well as the pages, summarize all of those. [2:24](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=144s) I started to add, I added a scheduler. [2:27](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=147s) So we'll do it all the time. [2:29](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=149s) Email inbox, like I could send links and it would go crawl those links and send me, you know, kind of the summaries. [2:35](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=155s) And then it kind of snowballed. [2:37](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=157s) It became what Andre Karpathy calls an LLM wiki, right? [2:43](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=163s) And so I basically at this point, I realized this is no longer like a set of agent skills. [2:51](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=171s) This is my agent, you know? [2:53](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=173s) And so I decided, hey, I'm going to go full bore. [2:57](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=177s) And I want to actually make rewrite this thing or reorient this thing to be really agent scope. [3:03](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=183s) So at this point, it's all kind of deterministic workflows. [3:07](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=187s) Download YouTube, transcribe it step by step, right? [3:11](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=191s) But I also started to build some agentic workflows on top of that LLM wiki, like doing deep research, like doing daily self-reflection of my cloud code usage, and then creating skills out of that, suggesting agent instruction improvements. [3:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=208s) So I said, hey, I wrote a spec for the agents. [3:32](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=212s) And I said, hey, I want to have an agent runtime with a registry of agents that I create. [3:38](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=218s) They're going to run on schedules. [3:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=220s) They're going to do all the things that I wanted to do, like the self-reflection, monitoring the system itself. [3:46](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=226s) And then I had, by this time, codex was taking off. [3:52](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=232s) So I had codex and clog kind of take turns implementing stuff, reviewing each other's code, fixing each other's bugs. [3:58](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=238s) And things kind of slowly started to work. [4:02](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=242s) One day, I read one of the digests. [4:06](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=246s) And I realized it wrote a sentence that really belonged to another section of the digest. [4:13](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=253s) You know, basically kind of misattributed stuff. [4:16](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=256s) So like a good AI engineer, I'm like, I'm going to go do some error analysis. [4:21](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=261s) I'm going to look at traces of the agent, what it did, maybe write some evals so that it'll improve in the future. [4:27](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=267s) And I said to Cloud Code, hey, tell me for this particular session where the traces are. [4:35](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=275s) And Cloud Code says, you know what, we don't actually have that because it's not really an agent. [4:43](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=283s) And so then I went and looked at the code in a little bit more detail. [4:48](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=288s) And after a little while, I said to Cloud Code, you know what, I do not agree with this design at all. [4:56](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=296s) And that I undoubtedly approved, okay? [5:01](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=301s) So I had, you know, written a bunch of specs, having to write a bunch of plans, right? [5:06](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=306s) And then it said the words like agent skills. [5:10](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=310s) And I was like, okay, fine, you know, keep going. [5:13](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=313s) Who has this experience? [5:14](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=314s) You tell an agent, you know, you want to do something and it's like, okay, fine. [5:19](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=319s) And then it's not what you asked for, right? [5:21](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=321s) So, now, and so while I'm kind of like, sort of like, [5:26](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=326s) struggling with that, I, you know, then I go look at the code in a little more detail. [5:31](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=331s) And so, remember, this is started off as kind of a deterministic workflow system, right? [5:36](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=336s) So it's got steps and it had actions, which are like the YouTube download or transcribe, whatever. [5:43](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=343s) And then I had told it to add skills and actions, right? [5:46](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=346s) Agent skills. [5:48](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=348s) What you'll notice here is like, it says two things, skills and actions are actually the same thing. [5:54](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=354s) And then agents are something else. [5:56](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=356s) And when I go and look deeper, these are all like the same thing. [6:02](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=362s) So, you know, they have these plausible names for these constructs in the system, but they weren't what they say they are. [6:09](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=369s) So like a skill was not really an agent skill. [6:11](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=371s) It had no skill.md. [6:12](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=372s) It had no script. [6:14](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=374s) The agents, of course, again, were not running on agent SDKs, right? [6:18](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=378s) And at this point, I'm pretty kind of like disappointed. [6:24](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=384s) And I'm kind of thinking, well, what am I supposed to do? [6:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=388s) So like as I've worked with a coding agent to try to fix things, next day I get an email alert from one of the token providers. [6:36](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=396s) And it's basically like this is like the first week of the month. [6:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=400s) You have blown past what your normal budget is for the month. [6:44](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=404s) And so this is like $100, but it's like five times more than I was normally spending in a week, right? [6:51](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=411s) So it turns out Codex, a lot of people back then were saying, hey, Codex is like the genius suite, right? [6:59](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=419s) It's the one that you want to give your coding jobs to. [7:02](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=422s) Maybe you have Claude create the plans and the designs, but, you know, Codex is the one you want doing the coding. [7:08](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=428s) Well, Codex turns out it wired up all the unit tests to production inference. [7:13](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=433s) And over the, you know, the last day before I got that alert, it had ran like 30 cycles of unit tests where it was burning production tokens. [7:23](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=443s) I also looked at some of the, you know, runs of, for example, the daily reflection agent that I had been running. [7:31](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=451s) That's the one that looks at my coding agent sessions and suggests improvements. [7:37](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=457s) It weren't every day. [7:39](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=459s) But when I looked at the content, they were empty. [7:42](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=462s) Because again, when it was doing that skill action agent refactoring, it just completely broke stuff. [7:49](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=469s) So that none of the prompts are actually running to create what I had asked it to do. [7:54](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=474s) So what does somebody do at this point, right? [8:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=480s) I just start over fresh because at this point, I don't trust Claude code. [8:06](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=486s) I don't trust Codex. [8:07](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=487s) I do not trust them to go and do a full refactor of this system because I had been telling it to do it for the, you know, previous three weeks. [8:16](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=496s) Right. [8:17](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=497s) And it didn't do a particularly good job at that. [8:19](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=499s) So I built a new head. [8:20](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=500s) I wrote a fresh set of specs. [8:23](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=503s) Again, I'm a product manager. [8:25](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=505s) I'm not about to go write this myself. [8:27](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=507s) I'm still going to use a coding agent, but I'm going to start fresh with hopefully better specs, some rubrics in terms of giving the agents better goalposts of where it should aim for. [8:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=520s) And I wrote, you know, I created this new agent runtime. [8:44](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=524s) Everything is an agent. [8:45](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=525s) There's a registry or different agents. [8:47](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=527s) And what I ended up with looks, this is like almost three months ago. [8:52](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=532s) So it's like what all the coding agents look like these days. [8:55](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=535s) Right. [8:56](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=536s) So in the middle, I give a coding agent SDK instructions on what I want built. [9:03](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=543s) That's kind of the middle section. [9:05](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=545s) So that creates an agent that has a canvas that it can output, you know, what it's doing. [9:12](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=552s) Right. [9:13](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=553s) So, and remember the part where I'm like, I don't trust Claude. [9:18](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=558s) I don't trust Codex. [9:20](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=560s) So I actually do a bake off. [9:22](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=562s) Okay. [9:23](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=563s) Four different frameworks. [9:24](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=564s) For each one, I tell the coding agent, I want you to create this experience where I type into, for example, the Claude code, I would say, I want you to build an experience where I type into Claude Agent SDK that I interact with. [9:39](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=579s) It will create an agent for me. [9:42](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=582s) That agent will output stuff on the right. [9:44](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=584s) Right. [9:45](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=585s) So I did that with four different ones. [9:47](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=587s) Each one differently. [9:48](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=588s) With Codex, I would say, use Codex itself to create the agents and then run the agents. [9:54](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=594s) Right. [9:55](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=595s) So I settled ultimately on Claude code and Codex. [9:59](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=599s) Codex, I work at Microsoft. [10:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=600s) I use it probably 90% of the time at work. [10:05](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=605s) The rest, Claude code, it's really good. [10:10](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=610s) Codex, if you have, you know, we released it about a month ago. [10:14](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=614s) That's really good. [10:15](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=615s) I use it all the time. [10:17](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=617s) I really love it. [10:18](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=618s) For my personal projects like this one, I use Claude code and Codex. [10:22](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=622s) Pi is an interesting one. [10:24](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=624s) Pi is the framework. [10:26](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=626s) Who knows what Pi is? [10:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=628s) Yeah, Pi is the agent framework that was written by Mario Zechner. [10:34](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=634s) The guy who said, slow the fuck down. [10:38](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=638s) Right. [10:39](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=639s) And so it actually is. [10:41](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=641s) This is the screenshot of the one created by Pi. [10:45](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=645s) Pi did a really nice job, really nice looking experience very, very quickly. [10:51](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=651s) So I'm really impressed with it. [10:54](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=654s) One thing, though. [10:56](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=656s) It asks me questions targeted at an engineer like Mario Zechner. [11:02](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=662s) It asks me things like, hey, you know the SQLite database that we have to keep track of all the agents? [11:10](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=670s) Well, do you want to do this or that? [11:12](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=672s) Or like, do you want to architect the TypeScript APIs to look this way or that way? [11:17](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=677s) And these are like, I don't know. [11:20](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=680s) Like, I think this is like, Pi speaks like a love language for Mario Zechner, right? [11:25](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=685s) For like, hardcore engineers. [11:27](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=687s) It's not what a product manager needs to, needs to like, think about. [11:30](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=690s) So, I love it, but it's not the one I'm using. [11:33](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=693s) And I don't think it's the one that, you know, millions of people will be using to build agents and run agents, right? [11:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=700s) So, now that I got going again, I started to build everything back. [11:46](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=706s) Adding those transcripts, adding the summaries, adding eval harnesses on top. [11:52](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=712s) I'll get back to this in a second. [11:55](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=715s) And as I'm writing it, so the YouTube stuff that I wrote previously, that was all text, right? [12:04](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=724s) It was just taking a transcript of the YouTube and then summarizing the text. [12:09](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=729s) I see this lecture float by on YouTube, or by Dwarkesh Patel, his podcaster, YouTube channel. [12:16](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=736s) There's this guy, Reiner Pope, who did a lecture, a Blackboard lecture on LLM inference. [12:23](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=743s) Kind of the math and how you think about the workloads of LL inference on GPU clusters. [12:31](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=751s) And I work, that's my day job. [12:34](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=754s) So, I'm like, roofline analysis. [12:36](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=756s) I really want to understand this stuff. [12:38](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=758s) And so, I decide, hey, this time I'm going to go not just do the text, but I'm going to go dig deep. [12:45](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=765s) Into the YouTube images themselves. [12:49](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=769s) So, I build a video detive agent, where now I'm going to go, you know, basically take keyframes out of the video. [12:59](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=779s) For that video, I had like 440 of them. [13:02](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=782s) And then I use an AI eval rubric to get the best shots. [13:08](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=788s) Pull out the Blackboard equations. [13:10](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=790s) And then put them into a canvas that I can explore that YouTube. [13:15](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=795s) This is how I do most YouTube these days. [13:18](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=798s) I don't want the whole YouTube. [13:19](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=799s) I go to my system and look at the canvas of the detail. [13:23](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=803s) So, let me just show you what that looks like. [13:27](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=807s) There's no audio, but basically this is the summary of that YouTube. [13:37](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=817s) Where it, you know, it breaks it up into chapters. [13:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=820s) It shows the Blackboard equations that's in the lecture. [13:44](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=824s) It provides explanations for those. [13:47](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=827s) And then I can, you know, jump to a particular section of the YouTube. [13:51](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=831s) I can highlight an equation. [13:53](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=833s) I can start to chat with it. [13:55](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=835s) Or chat with the agent to say, hey, explain this part a bit better for me. [13:59](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=839s) Right? [14:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=840s) And the way I did this is with a rubric. [14:07](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=847s) So, with an eval framework. [14:09](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=849s) So, again, I don't, in the old, six months ago what I was doing was I was telling an agent to, [14:16](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=856s) this is the spec for your work. [14:19](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=859s) It creates a plan and I either approve it or I don't. [14:21](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=861s) Now I also tell it how I want the outcome to be. [14:26](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=866s) So, the goal rubric for what the agent should work toward. [14:31](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=871s) So, in the particular case, so I really went deep on this particular YouTube because I wanted to get the system really, really right. [14:39](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=879s) So, that it does the right thing. [14:41](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=881s) And it turns out this generalizes to other types of YouTubes. [14:45](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=885s) For example, AI engineer videos that have, like, you know, slides, not just blackboards. [14:52](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=892s) Or that have people showing off VS Code or Cursor or something or terminals. [14:58](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=898s) So, it works on all of those. [14:59](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=899s) But to get there, I went deep with this one video. [15:03](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=903s) Worked on the rubric to say, hey, here's what I want from this canvas output. [15:09](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=909s) I want everything well explained. [15:11](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=911s) I want everything simple. [15:13](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=913s) I want to be able to jump to the right places in the video, right. [15:16](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=916s) And this process itself was something that I iterated on. [15:21](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=921s) Now, these rubrics, guess who wrote them? [15:26](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=926s) AI agents, right. [15:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=928s) I did not write these rubrics. [15:30](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=930s) I directed the agent to say, hey, this is how I want the rubrics to be created. [15:34](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=934s) And it works great. [15:37](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=937s) Now, you have to iterate quite a bit. [15:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=940s) But then what I do is I hill climb, right. [15:43](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=943s) I set the rubric. [15:44](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=944s) I let the agents hill climb to optimize toward that rubric. [15:49](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=949s) Once it's done a pass to generate the canvas based on that, then I go back and I see, look, for example, if there's something I don't like in the result canvas, I go back and tell the agent, hey, let's change the canvas. [16:03](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=963s) Let's change the rubric in a particular way because I want it to look like this. [16:09](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=969s) So in the very beginning, what happens is all those blackboard equations, they were just clumped together at the beginning. [16:16](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=976s) And then I would see like a big wall of text. [16:19](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=979s) Not what you want, right. [16:21](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=981s) So you want the each equation accompanied by the explanation interleave in the right order. [16:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=988s) So that's what I worked. [16:29](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=989s) You know, I basically told the agents to update the rubric to represent. [16:33](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=993s) That's what I wanted. [16:35](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=995s) And guess what else? [16:38](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=998s) I caught the agents cheating again. [16:41](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1001s) So what happened was I wasn't reviewing the rubrics completely either. [16:46](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1006s) It turned out the agents created one rubric that was really, really just completely set up to succeed on that one Ryan or Pope video. [16:56](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1016s) So it was basically kind of bench maxing, you know, kind of trying to game itself to succeed at that one video. [17:04](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1024s) But when I ran it on like five of the videos, it didn't do well because it was just so tuned to that video that it didn't do well. [17:14](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1034s) But actually when I told the agent to go fix it, you know, don't try to just game yourself to be good at that one video. [17:22](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1042s) It actually generalized better and it got overall better scores on everything. [17:27](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1047s) So yeah, and then so that's basically a loop, right? [17:34](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1054s) It's a loop of creating some rubrics with an agent. [17:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1060s) It's a loop of now once you've created a rubric, it hill climbs to get to the best possible outcome. [17:49](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1069s) And then I loop on top of that to iterate on the rubric itself to improve the rubric so that the outcome gets better. [17:58](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1078s) So that's one level of looping. [18:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1080s) There's also other loopings, Claude and Codex. [18:03](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1083s) At this point, remember the part where I'm like, I don't always trust what they're doing? [18:09](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1089s) I have them check each other, okay? [18:11](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1091s) Like in these adversarial code reviews where I have them deeply look at what they're doing and then suggest fixes. [18:18](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1098s) And this goes for like several rounds. [18:21](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1101s) So another loop. [18:22](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1102s) So basically where I've landed at this point, I was all the way at L. [18:30](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1110s) I looked at the code, but I'm back, double down on L, right? [18:37](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1117s) I own the goal. [18:38](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1118s) I tell the agents what I want. [18:40](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1120s) I tell the agents how they should output, how they should work. [18:45](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1125s) But I still don't read the code for the most part, right? [18:50](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1130s) Definitely not every line. [18:52](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1132s) So I'm never going to get to Z. [18:55](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1135s) I'm always going to be on the L spectrum, right? [18:58](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1138s) And that's the right place for me. [19:00](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1140s) So token boxing is what makes a product manager an AI engineer. [19:12](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1152s) And I want to thank you for being here. [19:15](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1155s) I want to thank everybody who creates the AIs and the tools that makes it possible for a product manager like me to create these experiences that I've wanted to build for years. [19:28](https://www.youtube.com/watch?v=6TjFjHPjVH0&t=1168s) So thank you again.