[
  {
    "startMs": 0,
    "text": "All right, welcome."
  },
  {
    "startMs": 15000,
    "text": "Everybody, I'm Keiji Kanazawa."
  },
  {
    "startMs": 19000,
    "text": "I let my coding agents refactor my code base for three weeks."
  },
  {
    "startMs": 25000,
    "text": "And then I read the code."
  },
  {
    "startMs": 28000,
    "text": "Listen on for what happened after that."
  },
  {
    "startMs": 33000,
    "text": "So most of you here in this track probably are aware of the Lopopolo Zechner debate."
  },
  {
    "startMs": 40000,
    "text": "How many people don't know what that is?"
  },
  {
    "startMs": 44000,
    "text": "Okay, so Lopopolo Zechner, back in AI engineering Europe a few months ago, Lopopolo, he works at OpenAI."
  },
  {
    "startMs": 53000,
    "text": "He said, reading code the liability. Just spend billions of tokens to get stuff done."
  },
  {
    "startMs": 59000,
    "text": "Mario Zechner at the same conference said, read the fucking code. Slow the fuck down."
  },
  {
    "startMs": 67000,
    "text": "Okay, so that's the LZ spectrum."
  },
  {
    "startMs": 70000,
    "text": "How many people here are like on the L side?"
  },
  {
    "startMs": 75000,
    "text": "Who's on the Z side?"
  },
  {
    "startMs": 77000,
    "text": "All right, awesome."
  },
  {
    "startMs": 79000,
    "text": "So I am very, very much on the L end."
  },
  {
    "startMs": 83000,
    "text": "Now, I'm a product manager."
  },
  {
    "startMs": 85000,
    "text": "I also have an AI research background."
  },
  {
    "startMs": 88000,
    "text": "I was started off as a dev."
  },
  {
    "startMs": 90000,
    "text": "But these days I work on product."
  },
  {
    "startMs": 94000,
    "text": "And back last December, like a lot of people,"
  },
  {
    "startMs": 99000,
    "text": "Clock Code, Opus 4.5, really amazing moment, right?"
  },
  {
    "startMs": 102000,
    "text": "When things just started to work."
  },
  {
    "startMs": 104000,
    "text": "So I have a day job that's pretty busy."
  },
  {
    "startMs": 107000,
    "text": "I work at Microsoft, work with the Frontier Labs, OpenAI and Anthropic to provide their models to the Microsoft customers."
  },
  {
    "startMs": 118000,
    "text": "So, but I love to listen and watch the podcasts."
  },
  {
    "startMs": 121000,
    "text": "I never have time."
  },
  {
    "startMs": 123000,
    "text": "So I set up the, you know, a project to create some skills to download podcasts, transcribe them, summarize."
  },
  {
    "startMs": 131000,
    "text": "Many of you provide them the same thing."
  },
  {
    "startMs": 133000,
    "text": "But this project then snowballed."
  },
  {
    "startMs": 136000,
    "text": "I added X threads."
  },
  {
    "startMs": 138000,
    "text": "It would go in, crawl X, you know, topics as well as the pages, summarize all of those."
  },
  {
    "startMs": 144000,
    "text": "I started to add, I added a scheduler."
  },
  {
    "startMs": 147000,
    "text": "So we'll do it all the time."
  },
  {
    "startMs": 149000,
    "text": "Email inbox, like I could send links and it would go crawl those links and send me, you know, kind of the summaries."
  },
  {
    "startMs": 155000,
    "text": "And then it kind of snowballed."
  },
  {
    "startMs": 157000,
    "text": "It became what Andre Karpathy calls an LLM wiki, right?"
  },
  {
    "startMs": 163000,
    "text": "And so I basically at this point, I realized this is no longer like a set of agent skills."
  },
  {
    "startMs": 171000,
    "text": "This is my agent, you know?"
  },
  {
    "startMs": 173000,
    "text": "And so I decided, hey, I'm going to go full bore."
  },
  {
    "startMs": 177000,
    "text": "And I want to actually make rewrite this thing or reorient this thing to be really agent scope."
  },
  {
    "startMs": 183000,
    "text": "So at this point, it's all kind of deterministic workflows."
  },
  {
    "startMs": 187000,
    "text": "Download YouTube, transcribe it step by step, right?"
  },
  {
    "startMs": 191000,
    "text": "But I also started to build some agentic workflows on top of that LLM wiki, like doing deep research, like doing daily self-reflection of my cloud code usage, and then creating skills out of that, suggesting agent instruction improvements."
  },
  {
    "startMs": 208000,
    "text": "So I said, hey, I wrote a spec for the agents."
  },
  {
    "startMs": 212000,
    "text": "And I said, hey, I want to have an agent runtime with a registry of agents that I create."
  },
  {
    "startMs": 218000,
    "text": "They're going to run on schedules."
  },
  {
    "startMs": 220000,
    "text": "They're going to do all the things that I wanted to do, like the self-reflection, monitoring the system itself."
  },
  {
    "startMs": 226000,
    "text": "And then I had, by this time, codex was taking off."
  },
  {
    "startMs": 232000,
    "text": "So I had codex and clog kind of take turns implementing stuff, reviewing each other's code, fixing each other's bugs."
  },
  {
    "startMs": 238000,
    "text": "And things kind of slowly started to work."
  },
  {
    "startMs": 242000,
    "text": "One day, I read one of the digests."
  },
  {
    "startMs": 246000,
    "text": "And I realized it wrote a sentence that really belonged to another section of the digest."
  },
  {
    "startMs": 253000,
    "text": "You know, basically kind of misattributed stuff."
  },
  {
    "startMs": 256000,
    "text": "So like a good AI engineer, I'm like, I'm going to go do some error analysis."
  },
  {
    "startMs": 261000,
    "text": "I'm going to look at traces of the agent, what it did, maybe write some evals so that it'll improve in the future."
  },
  {
    "startMs": 267000,
    "text": "And I said to Cloud Code, hey, tell me for this particular session where the traces are."
  },
  {
    "startMs": 275000,
    "text": "And Cloud Code says, you know what, we don't actually have that because it's not really an agent."
  },
  {
    "startMs": 283000,
    "text": "And so then I went and looked at the code in a little bit more detail."
  },
  {
    "startMs": 288000,
    "text": "And after a little while, I said to Cloud Code, you know what, I do not agree with this design at all."
  },
  {
    "startMs": 296000,
    "text": "And that I undoubtedly approved, okay?"
  },
  {
    "startMs": 301000,
    "text": "So I had, you know, written a bunch of specs, having to write a bunch of plans, right?"
  },
  {
    "startMs": 306000,
    "text": "And then it said the words like agent skills."
  },
  {
    "startMs": 310000,
    "text": "And I was like, okay, fine, you know, keep going."
  },
  {
    "startMs": 313000,
    "text": "Who has this experience?"
  },
  {
    "startMs": 314000,
    "text": "You tell an agent, you know, you want to do something and it's like, okay, fine."
  },
  {
    "startMs": 319000,
    "text": "And then it's not what you asked for, right?"
  },
  {
    "startMs": 321000,
    "text": "So, now, and so while I'm kind of like, sort of like,"
  },
  {
    "startMs": 326000,
    "text": "struggling with that, I, you know, then I go look at the code in a little more detail."
  },
  {
    "startMs": 331000,
    "text": "And so, remember, this is started off as kind of a deterministic workflow system, right?"
  },
  {
    "startMs": 336000,
    "text": "So it's got steps and it had actions, which are like the YouTube download or transcribe, whatever."
  },
  {
    "startMs": 343000,
    "text": "And then I had told it to add skills and actions, right?"
  },
  {
    "startMs": 346000,
    "text": "Agent skills."
  },
  {
    "startMs": 348000,
    "text": "What you'll notice here is like, it says two things, skills and actions are actually the same thing."
  },
  {
    "startMs": 354000,
    "text": "And then agents are something else."
  },
  {
    "startMs": 356000,
    "text": "And when I go and look deeper, these are all like the same thing."
  },
  {
    "startMs": 362000,
    "text": "So, you know, they have these plausible names for these constructs in the system, but they weren't what they say they are."
  },
  {
    "startMs": 369000,
    "text": "So like a skill was not really an agent skill."
  },
  {
    "startMs": 371000,
    "text": "It had no skill.md."
  },
  {
    "startMs": 372000,
    "text": "It had no script."
  },
  {
    "startMs": 374000,
    "text": "The agents, of course, again, were not running on agent SDKs, right?"
  },
  {
    "startMs": 378000,
    "text": "And at this point, I'm pretty kind of like disappointed."
  },
  {
    "startMs": 384000,
    "text": "And I'm kind of thinking, well, what am I supposed to do?"
  },
  {
    "startMs": 388000,
    "text": "So like as I've worked with a coding agent to try to fix things, next day I get an email alert from one of the token providers."
  },
  {
    "startMs": 396000,
    "text": "And it's basically like this is like the first week of the month."
  },
  {
    "startMs": 400000,
    "text": "You have blown past what your normal budget is for the month."
  },
  {
    "startMs": 404000,
    "text": "And so this is like $100, but it's like five times more than I was normally spending in a week, right?"
  },
  {
    "startMs": 411000,
    "text": "So it turns out Codex, a lot of people back then were saying, hey, Codex is like the genius suite, right?"
  },
  {
    "startMs": 419000,
    "text": "It's the one that you want to give your coding jobs to."
  },
  {
    "startMs": 422000,
    "text": "Maybe you have Claude create the plans and the designs, but, you know, Codex is the one you want doing the coding."
  },
  {
    "startMs": 428000,
    "text": "Well, Codex turns out it wired up all the unit tests to production inference."
  },
  {
    "startMs": 433000,
    "text": "And over the, you know, the last day before I got that alert, it had ran like 30 cycles of unit tests where it was burning production tokens."
  },
  {
    "startMs": 443000,
    "text": "I also looked at some of the, you know, runs of, for example, the daily reflection agent that I had been running."
  },
  {
    "startMs": 451000,
    "text": "That's the one that looks at my coding agent sessions and suggests improvements."
  },
  {
    "startMs": 457000,
    "text": "It weren't every day."
  },
  {
    "startMs": 459000,
    "text": "But when I looked at the content, they were empty."
  },
  {
    "startMs": 462000,
    "text": "Because again, when it was doing that skill action agent refactoring, it just completely broke stuff."
  },
  {
    "startMs": 469000,
    "text": "So that none of the prompts are actually running to create what I had asked it to do."
  },
  {
    "startMs": 474000,
    "text": "So what does somebody do at this point, right?"
  },
  {
    "startMs": 480000,
    "text": "I just start over fresh because at this point, I don't trust Claude code."
  },
  {
    "startMs": 486000,
    "text": "I don't trust Codex."
  },
  {
    "startMs": 487000,
    "text": "I do not trust them to go and do a full refactor of this system because I had been telling it to do it for the, you know, previous three weeks."
  },
  {
    "startMs": 496000,
    "text": "Right."
  },
  {
    "startMs": 497000,
    "text": "And it didn't do a particularly good job at that."
  },
  {
    "startMs": 499000,
    "text": "So I built a new head."
  },
  {
    "startMs": 500000,
    "text": "I wrote a fresh set of specs."
  },
  {
    "startMs": 503000,
    "text": "Again, I'm a product manager."
  },
  {
    "startMs": 505000,
    "text": "I'm not about to go write this myself."
  },
  {
    "startMs": 507000,
    "text": "I'm still going to use a coding agent, but I'm going to start fresh with hopefully better specs, some rubrics in terms of giving the agents better goalposts of where it should aim for."
  },
  {
    "startMs": 520000,
    "text": "And I wrote, you know, I created this new agent runtime."
  },
  {
    "startMs": 524000,
    "text": "Everything is an agent."
  },
  {
    "startMs": 525000,
    "text": "There's a registry or different agents."
  },
  {
    "startMs": 527000,
    "text": "And what I ended up with looks, this is like almost three months ago."
  },
  {
    "startMs": 532000,
    "text": "So it's like what all the coding agents look like these days."
  },
  {
    "startMs": 535000,
    "text": "Right."
  },
  {
    "startMs": 536000,
    "text": "So in the middle, I give a coding agent SDK instructions on what I want built."
  },
  {
    "startMs": 543000,
    "text": "That's kind of the middle section."
  },
  {
    "startMs": 545000,
    "text": "So that creates an agent that has a canvas that it can output, you know, what it's doing."
  },
  {
    "startMs": 552000,
    "text": "Right."
  },
  {
    "startMs": 553000,
    "text": "So, and remember the part where I'm like, I don't trust Claude."
  },
  {
    "startMs": 558000,
    "text": "I don't trust Codex."
  },
  {
    "startMs": 560000,
    "text": "So I actually do a bake off."
  },
  {
    "startMs": 562000,
    "text": "Okay."
  },
  {
    "startMs": 563000,
    "text": "Four different frameworks."
  },
  {
    "startMs": 564000,
    "text": "For each one, I tell the coding agent, I want you to create this experience where I type into, for example, the Claude code, I would say, I want you to build an experience where I type into Claude Agent SDK that I interact with."
  },
  {
    "startMs": 579000,
    "text": "It will create an agent for me."
  },
  {
    "startMs": 582000,
    "text": "That agent will output stuff on the right."
  },
  {
    "startMs": 584000,
    "text": "Right."
  },
  {
    "startMs": 585000,
    "text": "So I did that with four different ones."
  },
  {
    "startMs": 587000,
    "text": "Each one differently."
  },
  {
    "startMs": 588000,
    "text": "With Codex, I would say, use Codex itself to create the agents and then run the agents."
  },
  {
    "startMs": 594000,
    "text": "Right."
  },
  {
    "startMs": 595000,
    "text": "So I settled ultimately on Claude code and Codex."
  },
  {
    "startMs": 599000,
    "text": "Codex, I work at Microsoft."
  },
  {
    "startMs": 600000,
    "text": "I use it probably 90% of the time at work."
  },
  {
    "startMs": 605000,
    "text": "The rest, Claude code, it's really good."
  },
  {
    "startMs": 610000,
    "text": "Codex, if you have, you know, we released it about a month ago."
  },
  {
    "startMs": 614000,
    "text": "That's really good."
  },
  {
    "startMs": 615000,
    "text": "I use it all the time."
  },
  {
    "startMs": 617000,
    "text": "I really love it."
  },
  {
    "startMs": 618000,
    "text": "For my personal projects like this one, I use Claude code and Codex."
  },
  {
    "startMs": 622000,
    "text": "Pi is an interesting one."
  },
  {
    "startMs": 624000,
    "text": "Pi is the framework."
  },
  {
    "startMs": 626000,
    "text": "Who knows what Pi is?"
  },
  {
    "startMs": 628000,
    "text": "Yeah, Pi is the agent framework that was written by Mario Zechner."
  },
  {
    "startMs": 634000,
    "text": "The guy who said, slow the fuck down."
  },
  {
    "startMs": 638000,
    "text": "Right."
  },
  {
    "startMs": 639000,
    "text": "And so it actually is."
  },
  {
    "startMs": 641000,
    "text": "This is the screenshot of the one created by Pi."
  },
  {
    "startMs": 645000,
    "text": "Pi did a really nice job, really nice looking experience very, very quickly."
  },
  {
    "startMs": 651000,
    "text": "So I'm really impressed with it."
  },
  {
    "startMs": 654000,
    "text": "One thing, though."
  },
  {
    "startMs": 656000,
    "text": "It asks me questions targeted at an engineer like Mario Zechner."
  },
  {
    "startMs": 662000,
    "text": "It asks me things like, hey, you know the SQLite database that we have to keep track of all the agents?"
  },
  {
    "startMs": 670000,
    "text": "Well, do you want to do this or that?"
  },
  {
    "startMs": 672000,
    "text": "Or like, do you want to architect the TypeScript APIs to look this way or that way?"
  },
  {
    "startMs": 677000,
    "text": "And these are like, I don't know."
  },
  {
    "startMs": 680000,
    "text": "Like, I think this is like, Pi speaks like a love language for Mario Zechner, right?"
  },
  {
    "startMs": 685000,
    "text": "For like, hardcore engineers."
  },
  {
    "startMs": 687000,
    "text": "It's not what a product manager needs to, needs to like, think about."
  },
  {
    "startMs": 690000,
    "text": "So, I love it, but it's not the one I'm using."
  },
  {
    "startMs": 693000,
    "text": "And I don't think it's the one that, you know, millions of people will be using to build agents and run agents, right?"
  },
  {
    "startMs": 700000,
    "text": "So, now that I got going again, I started to build everything back."
  },
  {
    "startMs": 706000,
    "text": "Adding those transcripts, adding the summaries, adding eval harnesses on top."
  },
  {
    "startMs": 712000,
    "text": "I'll get back to this in a second."
  },
  {
    "startMs": 715000,
    "text": "And as I'm writing it, so the YouTube stuff that I wrote previously, that was all text, right?"
  },
  {
    "startMs": 724000,
    "text": "It was just taking a transcript of the YouTube and then summarizing the text."
  },
  {
    "startMs": 729000,
    "text": "I see this lecture float by on YouTube, or by Dwarkesh Patel, his podcaster, YouTube channel."
  },
  {
    "startMs": 736000,
    "text": "There's this guy, Reiner Pope, who did a lecture, a Blackboard lecture on LLM inference."
  },
  {
    "startMs": 743000,
    "text": "Kind of the math and how you think about the workloads of LL inference on GPU clusters."
  },
  {
    "startMs": 751000,
    "text": "And I work, that's my day job."
  },
  {
    "startMs": 754000,
    "text": "So, I'm like, roofline analysis."
  },
  {
    "startMs": 756000,
    "text": "I really want to understand this stuff."
  },
  {
    "startMs": 758000,
    "text": "And so, I decide, hey, this time I'm going to go not just do the text, but I'm going to go dig deep."
  },
  {
    "startMs": 765000,
    "text": "Into the YouTube images themselves."
  },
  {
    "startMs": 769000,
    "text": "So, I build a video detive agent, where now I'm going to go, you know, basically take keyframes out of the video."
  },
  {
    "startMs": 779000,
    "text": "For that video, I had like 440 of them."
  },
  {
    "startMs": 782000,
    "text": "And then I use an AI eval rubric to get the best shots."
  },
  {
    "startMs": 788000,
    "text": "Pull out the Blackboard equations."
  },
  {
    "startMs": 790000,
    "text": "And then put them into a canvas that I can explore that YouTube."
  },
  {
    "startMs": 795000,
    "text": "This is how I do most YouTube these days."
  },
  {
    "startMs": 798000,
    "text": "I don't want the whole YouTube."
  },
  {
    "startMs": 799000,
    "text": "I go to my system and look at the canvas of the detail."
  },
  {
    "startMs": 803000,
    "text": "So, let me just show you what that looks like."
  },
  {
    "startMs": 807000,
    "text": "There's no audio, but basically this is the summary of that YouTube."
  },
  {
    "startMs": 817000,
    "text": "Where it, you know, it breaks it up into chapters."
  },
  {
    "startMs": 820000,
    "text": "It shows the Blackboard equations that's in the lecture."
  },
  {
    "startMs": 824000,
    "text": "It provides explanations for those."
  },
  {
    "startMs": 827000,
    "text": "And then I can, you know, jump to a particular section of the YouTube."
  },
  {
    "startMs": 831000,
    "text": "I can highlight an equation."
  },
  {
    "startMs": 833000,
    "text": "I can start to chat with it."
  },
  {
    "startMs": 835000,
    "text": "Or chat with the agent to say, hey, explain this part a bit better for me."
  },
  {
    "startMs": 839000,
    "text": "Right?"
  },
  {
    "startMs": 840000,
    "text": "And the way I did this is with a rubric."
  },
  {
    "startMs": 847000,
    "text": "So, with an eval framework."
  },
  {
    "startMs": 849000,
    "text": "So, again, I don't, in the old, six months ago what I was doing was I was telling an agent to,"
  },
  {
    "startMs": 856000,
    "text": "this is the spec for your work."
  },
  {
    "startMs": 859000,
    "text": "It creates a plan and I either approve it or I don't."
  },
  {
    "startMs": 861000,
    "text": "Now I also tell it how I want the outcome to be."
  },
  {
    "startMs": 866000,
    "text": "So, the goal rubric for what the agent should work toward."
  },
  {
    "startMs": 871000,
    "text": "So, in the particular case, so I really went deep on this particular YouTube because I wanted to get the system really, really right."
  },
  {
    "startMs": 879000,
    "text": "So, that it does the right thing."
  },
  {
    "startMs": 881000,
    "text": "And it turns out this generalizes to other types of YouTubes."
  },
  {
    "startMs": 885000,
    "text": "For example, AI engineer videos that have, like, you know, slides, not just blackboards."
  },
  {
    "startMs": 892000,
    "text": "Or that have people showing off VS Code or Cursor or something or terminals."
  },
  {
    "startMs": 898000,
    "text": "So, it works on all of those."
  },
  {
    "startMs": 899000,
    "text": "But to get there, I went deep with this one video."
  },
  {
    "startMs": 903000,
    "text": "Worked on the rubric to say, hey, here's what I want from this canvas output."
  },
  {
    "startMs": 909000,
    "text": "I want everything well explained."
  },
  {
    "startMs": 911000,
    "text": "I want everything simple."
  },
  {
    "startMs": 913000,
    "text": "I want to be able to jump to the right places in the video, right."
  },
  {
    "startMs": 916000,
    "text": "And this process itself was something that I iterated on."
  },
  {
    "startMs": 921000,
    "text": "Now, these rubrics, guess who wrote them?"
  },
  {
    "startMs": 926000,
    "text": "AI agents, right."
  },
  {
    "startMs": 928000,
    "text": "I did not write these rubrics."
  },
  {
    "startMs": 930000,
    "text": "I directed the agent to say, hey, this is how I want the rubrics to be created."
  },
  {
    "startMs": 934000,
    "text": "And it works great."
  },
  {
    "startMs": 937000,
    "text": "Now, you have to iterate quite a bit."
  },
  {
    "startMs": 940000,
    "text": "But then what I do is I hill climb, right."
  },
  {
    "startMs": 943000,
    "text": "I set the rubric."
  },
  {
    "startMs": 944000,
    "text": "I let the agents hill climb to optimize toward that rubric."
  },
  {
    "startMs": 949000,
    "text": "Once it's done a pass to generate the canvas based on that, then I go back and I see, look, for example, if there's something I don't like in the result canvas, I go back and tell the agent, hey, let's change the canvas."
  },
  {
    "startMs": 963000,
    "text": "Let's change the rubric in a particular way because I want it to look like this."
  },
  {
    "startMs": 969000,
    "text": "So in the very beginning, what happens is all those blackboard equations, they were just clumped together at the beginning."
  },
  {
    "startMs": 976000,
    "text": "And then I would see like a big wall of text."
  },
  {
    "startMs": 979000,
    "text": "Not what you want, right."
  },
  {
    "startMs": 981000,
    "text": "So you want the each equation accompanied by the explanation interleave in the right order."
  },
  {
    "startMs": 988000,
    "text": "So that's what I worked."
  },
  {
    "startMs": 989000,
    "text": "You know, I basically told the agents to update the rubric to represent."
  },
  {
    "startMs": 993000,
    "text": "That's what I wanted."
  },
  {
    "startMs": 995000,
    "text": "And guess what else?"
  },
  {
    "startMs": 998000,
    "text": "I caught the agents cheating again."
  },
  {
    "startMs": 1001000,
    "text": "So what happened was I wasn't reviewing the rubrics completely either."
  },
  {
    "startMs": 1006000,
    "text": "It turned out the agents created one rubric that was really, really just completely set up to succeed on that one Ryan or Pope video."
  },
  {
    "startMs": 1016000,
    "text": "So it was basically kind of bench maxing, you know, kind of trying to game itself to succeed at that one video."
  },
  {
    "startMs": 1024000,
    "text": "But when I ran it on like five of the videos, it didn't do well because it was just so tuned to that video that it didn't do well."
  },
  {
    "startMs": 1034000,
    "text": "But actually when I told the agent to go fix it, you know, don't try to just game yourself to be good at that one video."
  },
  {
    "startMs": 1042000,
    "text": "It actually generalized better and it got overall better scores on everything."
  },
  {
    "startMs": 1047000,
    "text": "So yeah, and then so that's basically a loop, right?"
  },
  {
    "startMs": 1054000,
    "text": "It's a loop of creating some rubrics with an agent."
  },
  {
    "startMs": 1060000,
    "text": "It's a loop of now once you've created a rubric, it hill climbs to get to the best possible outcome."
  },
  {
    "startMs": 1069000,
    "text": "And then I loop on top of that to iterate on the rubric itself to improve the rubric so that the outcome gets better."
  },
  {
    "startMs": 1078000,
    "text": "So that's one level of looping."
  },
  {
    "startMs": 1080000,
    "text": "There's also other loopings, Claude and Codex."
  },
  {
    "startMs": 1083000,
    "text": "At this point, remember the part where I'm like, I don't always trust what they're doing?"
  },
  {
    "startMs": 1089000,
    "text": "I have them check each other, okay?"
  },
  {
    "startMs": 1091000,
    "text": "Like in these adversarial code reviews where I have them deeply look at what they're doing and then suggest fixes."
  },
  {
    "startMs": 1098000,
    "text": "And this goes for like several rounds."
  },
  {
    "startMs": 1101000,
    "text": "So another loop."
  },
  {
    "startMs": 1102000,
    "text": "So basically where I've landed at this point, I was all the way at L."
  },
  {
    "startMs": 1110000,
    "text": "I looked at the code, but I'm back, double down on L, right?"
  },
  {
    "startMs": 1117000,
    "text": "I own the goal."
  },
  {
    "startMs": 1118000,
    "text": "I tell the agents what I want."
  },
  {
    "startMs": 1120000,
    "text": "I tell the agents how they should output, how they should work."
  },
  {
    "startMs": 1125000,
    "text": "But I still don't read the code for the most part, right?"
  },
  {
    "startMs": 1130000,
    "text": "Definitely not every line."
  },
  {
    "startMs": 1132000,
    "text": "So I'm never going to get to Z."
  },
  {
    "startMs": 1135000,
    "text": "I'm always going to be on the L spectrum, right?"
  },
  {
    "startMs": 1138000,
    "text": "And that's the right place for me."
  },
  {
    "startMs": 1140000,
    "text": "So token boxing is what makes a product manager an AI engineer."
  },
  {
    "startMs": 1152000,
    "text": "And I want to thank you for being here."
  },
  {
    "startMs": 1155000,
    "text": "I want to thank everybody who creates the AIs and the tools that makes it possible for a product manager like me to create these experiences that I've wanted to build for years."
  },
  {
    "startMs": 1168000,
    "text": "So thank you again."
  }
]