# Prototyping as Leadership: How a CTO Ships with AI Agents Hursh Agrawal AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/bdHaOXZOhcM-prototyping-as-leadership-cto-ships-ai-agents) · [Video](https://www.youtube.com/watch?v=bdHaOXZOhcM) ## Prototyping as Leadership: How a CTO Ships with AI Agents A crowded management calendar can still accommodate building when agents work between check-ins—but useful autonomy depends on context, verification and personal review. Familiarity with pull requests, continuous integration and basic LLM evaluation will help; no machine-learning training experience is required. ### Building inside a crowded calendar [0:16](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=16s) Fifteen-plus recurring meetings, seven direct reports, stand-ups, reviews and recruiting: Hursh Agrawal opens with his actual calendar. As CTO and co-founder of The Browser Company, makers of Arc and Dia, he has an organization to run. Yet something has changed in the gaps between appointments. Agrawal reports shipping two to ten PRs a week with coding agents while working a 40–50-hour week. With a toddler at home, extending the workday indefinitely is not the solution. ![Slide titled “This is my actual week” with four columns showing recurring meetings, direct reports, PRs per week, and a toddler at home.](https://ai.engineer/api/talk-essay-media/7f32b5c2f29db258babf864465603d07bbb737068b93397b4853fae1f245d5fa) An actual week: 15+ recurring meetings, seven direct reports, 2–10 PRs per week, and one toddler at home. Traditionally, growing as a leader meant influencing more work through roadmaps, documents and meetings. You supplied context to engineers who could spend sustained time implementing it. As coding agents become capable of longer autonomous tasks, the fragmented day described in Paul Graham’s [Maker’s Schedule, Manager’s Schedule](https://www.paulgraham.com/makersschedule.html) becomes usable for building, too. The manager does not need to remain at the keyboard throughout the implementation. ### Learn the capabilities, then make them tangible [1:48](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=108s) **Building becomes a way to acquire the judgment needed to lead.** Agrawal describes frontier model families changing on roughly three-month cycles: their capabilities shift, their responses to prompting change, and previously impractical product ideas become plausible. Opinions about each release are plentiful; firsthand intuition is harder to acquire. Using the models while building helps a leader set engineering expectations and assess implications for the product, business and strategy. It also makes questions such as what might happen if open-source models catch up in three to six months concrete enough to explore, rather than treating that possible timeline as a certainty. That intuition is difficult to transfer through enthusiasm alone. An engineer who has not tried the new model may reasonably remain unconvinced that it changes what the team can build. A working prototype moves the conversation forward: the capability is now inside the product, and colleagues can try it themselves. The artifact carries information that another persuasive meeting cannot. Leaders also bring useful inputs to agent work: business priorities, strategic trade-offs, past decisions and upcoming imperatives. Agrawal argues that this wider context makes their steering especially consequential. Delegation skills transfer directly—set a goal, supply context, check progress and coach when necessary. In his experience with the models of the period, execution is stronger than judgment. When an agent declares an algorithm impossible or says it cannot finish a task, suggesting another approach can restart useful work. That assessment is explicitly time-dependent; it is a reason to keep testing the models’ limits. ### Choose work that can tolerate interruptions [4:52](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=292s) Agrawal attributes four useful categories to Julie Zhuo’s poll of technical leaders: - **Internal efficiency:** tools that make the organization work better. - **Quality of life:** improvements to the codebase or product—the gardening that otherwise waits behind larger commitments. - **Celebration:** artifacts that recognize people and their contributions. - **Vision:** prototypes that connect new model capabilities to opportunities for the product and business. The vision category is especially valuable because it combines learning with demonstration: discover what a model can do, then put that possibility in colleagues’ hands. **Keep this work off the critical path.** Fires, recruiting calls and meetings still interrupt a leader’s day. A prototype can wait while those responsibilities take precedence; another engineer’s blocked delivery should not have to. The choice of task makes the schedule workable before any prompting technique does. ### Arrange the day around an overnight run [5:50](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=350s) Agrawal’s example routine allocates two to three hours a day to coding, distributed around management responsibilities. | Window | Human work | | --- | --- | | Morning, about an hour | Review the previous night’s code | | Gaps between meetings | Steer ongoing work | | End of day, around 5 PM | Prepare the next unattended run | The final block is the crucial handoff. It turns a short period of attention into a task that can continue after the workday ends. The loop starts by gathering context and answering clarifying questions. The coding agent then works unattended; Agrawal describes runs lasting four, six or eight hours. In the morning, its report gives him something concrete to assess and decide whether to ship. He applies this same structure to three kinds of work: implementing features, improving AI features through evaluations, and training custom machine-learning models. ### Give the agent enough context to make decisions [7:13](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=433s) For feature implementation, the useful question changes from how to break the work into individually prompted steps to **what the model needs to know to make the decisions you would make**. An agent working overnight cannot rely on continuous steering. It needs the business goals and the reasons behind the request, not just an implementation checklist. Agrawal uses an agent connected to organizational information—Slack, Jira, Confluence, Notion and the repository—to prepare that context. Before a meeting, he asks it to research the proposed work and produce a prompt for the coding agent. The request covers prior attempts, trade-offs, what worked, what failed and why the feature or model matters to the business. His example is a roughly thirty-second dictation through Wispr Flow into Dia, requesting about twenty minutes of research. The result is a substantial prompt that can be pasted into Claude Code, Cursor or Codex. The evening instruction can then be broad: implement the whole feature, using the gathered context. The research pass supplies the decision background that would otherwise remain in the leader’s head or scattered across internal systems. ### Specify verification and delivery, not just implementation [8:54](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=534s) An unattended implementation needs ways to check itself while it proceeds. Agrawal’s instructions establish a sequence: 1. **Write tests first.** In his experience, agents write less rigorous tests when they add them after the implementation. 2. **Exercise the complete flow with computer use.** Click through the interface and check whether the feature solves the business problem described in the context. 3. **Prepare reviewer-friendly PRs.** Split the work into readable changes with clear descriptions, monitor CI and get the whole stack passing. The distinction between tests and end-to-end interaction matters: passing checks and a coherent product flow answer different questions. The run should also include AI code review. The Browser Company has internal reviewers; Agrawal mentions Qodo as an external example and suggests running a review skill in a clean sub-agent when appropriate. The coding agent should then fix findings, watch PRs for bot comments and failing checks, and resolve review threads. This expands the assignment from producing code to preparing a reviewable stack. Once setup questions are settled, he asks the agent to continue without overnight interruptions and leave a morning report. He also adds encouragement as a personal habit, while saying he does not know whether it helps. The required output is substantive: the completed stack, results and an explanation of trade-offs. Agrawal says models such as **[Claude Opus 4.8](https://www.anthropic.com/news/claude-opus-4-8)** and newer **GPT** models have handled work he previously regarded as taking weeks in a single overnight run. That is his experience, not a controlled productivity benchmark. The practical benefit is the package waiting in the morning: something to inspect and improve, rather than a task still waiting to begin. ### Turn everyday feedback into an evaluation set [10:54](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=654s) The first version of an LLM feature usually leaves quality, latency and cost to optimize. To create material for that work, Agrawal adds a feedback button and text box after the LLM or agent run. During a thirty- or sixty-minute usage block, he tries the feature and records qualitative judgments: good, bad, or what should have been better. Each run produces a JSON dump in his downloads folder containing the system prompt, inputs and feedback. For a summarization feature, the same record structure could look like this: ```json { "system_prompt": "Summarize the meeting notes, preserving decisions and unresolved questions.", "inputs": { "notes": "We agreed to test the prototype with employees. Public release remains undecided." }, "feedback": "The summary treated public release as approved. Preserve the distinction between an agreed employee test and an undecided public release." } ``` The useful information is the connection between the instructions, a concrete input and the user’s judgment. That gives the next agent something more specific to work with than a request to improve quality. Agrawal starts small: four, five or ten examples can begin the process, with coworkers helping collect twenty or thirty. At evening setup, he gives the agent those files and asks it to turn them into a local eval set, using something as simple as SQLite or Markdown. Before handing over execution, he discusses the evaluation design and scoring functions interactively. The human judgment in the feedback has to become an agreed criterion the harness can evaluate. ### Let the harness drive improvement [12:19](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=739s) With scoring agreed, the agent builds a harness that runs the target call against the evals and iterates to improve the score. This is **hill climbing**: evaluate a candidate, make changes and evaluate again. A small dataset creates an obvious overfitting risk. Agrawal reports explicitly asking the agent to avoid overfitting and keep improvements general; that instruction expresses the goal, but does not itself establish generalization. The overnight assignment ends with a report describing the work and results. He also asks the agent to save the workflow as a reusable skill. Subsequent runs can improve both the feature and the process used to optimize it. Agrawal reports that improvements persisted as features moved into employee dogfooding and production. The accompanying slide reports an evaluation result moving from 2/7 to 7/7 and code-review recall rising from 10% to 55%. These are internal reported results; the supplied explanation does not specify enough evaluation conditions to treat them as general benchmarks or evidence that overfitting is impossible. ![Slide titled “One end-of-day prompt turns feedback into a tuned feature,” with a Claude Code prompt on the left and reported overnight results on the right.](https://ai.engineer/api/talk-essay-media/9b54b478926ddf344cbf2db46e7fd37342e254ac5ecde086a7206ddcbed3ca8f) An overnight evaluation prompt alongside reported results of 2/7 → 7/7 and code-review recall rising from 10% to 55%. An internal AI code reviewer is one concrete application. The team repeatedly used overnight runs to improve its quality, applying the same feedback, evaluation and iteration loop to a tool that supports future development work. ### Train custom models with a bounded infrastructure assignment [13:23](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=803s) The third application extends the overnight loop to custom model training. Agrawal describes a **[ModernBERT](https://github.com/AnswerDotAI/ModernBERT)** classifier for personally identifiable information, or PII. Earlier attempts using **Opus** and **Haiku** had unsatisfactory cost, latency and precision-recall for the task. The starting point for a different approach was collected training data plus business context. The agent’s assignment covered the training workflow, not just a training command: 1. **Prepare the data.** Clean the collected examples and augment them with synthetic data as needed, using access to OpenAI and Anthropic models as an ensemble to improve data quality. 2. **Choose candidate models.** Propose suitable model classes and train two candidates rather than committing to one immediately. 3. **Use isolated infrastructure.** Work within an AWS sandbox, explicitly without production access. 4. **Complete the resource lifecycle.** Select GPU and EC2 resources, train the candidates, test them against evaluation examples and deprovision the infrastructure. 5. **Return deployable artifacts and a plan.** Deliver the trained models and a report, including how inference could be hosted within the existing codebase. The sandbox boundary is part of the task definition. The agent needs enough access to carry out the experiment without gaining the ability to disrupt production. After answering clarifying questions, Agrawal leaves the run to proceed. He reports receiving two trained models and a full report the next morning, and says the team has moved multiple models from this kind of workflow into production. He supplies no numerical precision, recall, latency or cost measurements for this classifier, so the example demonstrates the scope of delegated work rather than a measured comparison between model families. ### Expand scope within organizational safeguards [14:52](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=892s) Across the three examples, the experiment is to keep increasing task scope. Agrawal cites Notion co-founder Simon Last’s observation that modern models can do more than people tend to assume. Testing those boundaries teaches a leader both how to direct engineering work and what new utility the product might offer. His invitation to try work on the scale of weeks or months in an overnight run is an ambition to explore, not an established universal speedup. That ambition depends on existing organizational support. AI code reviewers, maintained `CLAUDE.md` and `AGENTS.md` instructions, and CI that the team can trust give autonomous work a framework for checking correctness. Feature flags and staged distribution limit where an experiment can have an effect. The Browser Company also has a prototype branch that distributes changes to employees without releasing them to production. This lets colleagues try the artifacts while preserving a separate production boundary. ### Own the code before asking others to review it [16:14](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=974s) The safeguards do not make agents infallible. Agrawal acknowledges that his code has annoyed engineers and caused severity incidents, or SEVs. He still considers building worthwhile for what it teaches and for the example it sets, but that example includes engineering hygiene as well as ambitious delegation. The morning block therefore includes personal testing of what the agent produced. Agrawal recommends testing everything before putting the PR up; automated computer interaction does not replace trying the feature yourself. Small, readable PRs matter for the same reason: a leader establishes what others see as acceptable. His negative example is submitting three 5,000-line PRs and encouraging the team to imitate that behavior. **Read and review the generated code before requesting a colleague’s review.** A review-ready stack from an agent is still your responsibility. Handing unread code to another engineer transfers the burden of discovering obvious mistakes—and undermines the standard you are trying to establish. Agrawal’s warning comes from having made that mistake himself. ![Slide reading “Code hygiene matters” and “My code has annoyed my engineers. It has caused SEVs,” followed by a warning that coding agents are not perfect.](https://ai.engineer/api/talk-essay-media/895f2655224730cc6a16169e56e6b7ed82161b5695bbb3ae28531f2f39e59223) “Code hygiene matters”: the speaker’s slide acknowledges that his code annoyed engineers and caused SEVs. His closing recommendation is to make building part of leadership, even with only one or two hours a day—less than the earlier example routine. Repeated practice develops a feel for each new model family and for the amount of scope it can handle. Working artifacts then show the organization what is possible, while the experience of making them generates ideas for the next overnight run. ## Resources - [Maker's Schedule, Manager's Schedule](https://www.paulgraham.com/makersschedule.html) — Paul Graham's July 2009 essay explains why meetings affect managers and makers differently. - [ModernBERT](https://github.com/AnswerDotAI/ModernBERT) — Official research code with training configuration, evaluation instructions and links to model checkpoints. - [ModernBERT: Smarter, Better, Faster, Longer](https://arxiv.org/abs/2412.13663) — Original paper describing ModernBERT's encoder architecture and classification and retrieval evaluations. - [Introducing Claude Opus 4.8](https://www.anthropic.com/news/claude-opus-4-8) — Anthropic's release announcement describes Opus 4.8 and accompanying Claude Code capabilities. - [Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) — Practical guidance for defining task-specific metrics, constructing evaluation cases and choosing grading methods. - [Best practices for Claude Code](https://code.claude.com/docs/en/best-practices) — Current guidance on context, verification, independent review and maintaining project instructions. ## Slides ### Prototyping as leadership Hursh Agrawal introduces how he ships with AI agents, as coding agents have become more autonomous and able to handle longer tasks in the last few months. [1:24](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=84s) [Title slide introducing Hursh Agrawal’s talk on how a CTO ships with AI agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b00c7f25f5b9ca6297a2411429d59afae8407351eb7a26c439fadb4b06b5994.webp) Prototyping as leadership How a CTO ships with AI agents Hursh Agrawal CTO and co-founder · The Browser Company Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 1 / 20 [illegible] Made with Dia ### This is my actual week Hursh Agrawal’s week includes 15+ recurring meetings, seven direct reports, 2–10 PRs per week, and a toddler at home. [1:34](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=94s) [The speaker lists over 15 recurring meetings, seven direct reports, two to ten pull requests per week, and one toddler at home.](https://assets.ai.engineer/sites/aiecode2025/sha256/332c154d0c05160ea71003c54f384bb73f83082e85e79bc352b5a441a6b09963.webp) This is my actual week 15+ Recurring meetings 7 Direct reports 2–10 PRs per week 1 Toddler at home Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 3 / 20 [illegible] Made with Dia ### The wall every engineering leader hits Engineering leaders can use scattered coding blocks to ship, even as meetings and documents pull their attention toward describing the future. [2:08](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=128s) [Two panels contrast leaders describing work through documents and meetings with using fragmented time to build.](https://assets.ai.engineer/sites/aiecode2025/sha256/99ea99de16c16d9cf1cb6f7de8e4024a41c6f2f64a80f0ca4ad291126412998c.webp) The wall every engineering leader hits THE OLD TRADE You stop shipping and start describing. Calendar tetris wins. Your influence narrows to roadmaps, docs, and meetings. You communicate the future instead of building it. WHAT CHANGED Fragmented time became buildable. A coding block at the start of the day, one at the end, a few in between. That is now enough to ship real things. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 4 / 20 [illegible] Made with Dia ### So building is part of the job now Leaders build to learn what new models can do and where they fit their product. A working prototype lets colleagues try those capabilities instead of debating them. [2:47](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=167s) [Two reasons to build: firsthand experience informs leadership, and prototypes demonstrate what is possible.](https://assets.ai.engineer/sites/aiecode2025/sha256/1331d0048301ece6eee1cd9c0cab8198f6861c5ce4cdde37cd0cbf212337a547.webp) So building is part of the job now You can finally fit it in — and it’s becoming a necessary part of the job because... 1 You can’t lead what you can’t feel New frontier models reset what’s possible every few months. If you’re not building, you only know their capabilities secondhand — and you’ll set the wrong expectations and miss where they fit your product. 2 It’s better to show, not tell A working prototype moves the org more than any doc or roadmap. You stop arguing about what’s possible and just demonstrate it. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 5 / 20 [illegible] | Made with Dia ### And leaders are unusually well suited for it Leaders bring context and delegation experience to coding agents. When an agent cannot find a solution, a leader can suggest another approach, supplying judgment while the agent executes. [4:36](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=276s) [The slide identifies context, delegation experience, and judgment as leadership strengths for working with coding agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/034daccc77f761125c4ff1cb5296849434bfe0cf0356a24de5b856892505ce2a.webp) And leaders are unusually well suited for it 1 Context is the unfair advantage You have the most context of anyone — business, strategy, people, tech. Your steering is more impactful than an IC’s. 2 You already delegate Hand off a mission, set the goal, review the output. Agents respond to the same muscle you’ve used with people for years. 3 Coding agents need judgement to wield well Great at execution at volume, still weak at judgment. You supply the judgment; it supplies the throughput. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 6 / 20 [illegible] Made with Dia ### Manager IC work: four categories Julie Zhuo suggests managers work on team efficiency, product fixes, celebration stories, and vision pieces. She cautions against critical-path product work because it can compete with managing. [5:49](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=349s) [A post lists internal efficiency, quality-of-life improvements, celebration stories, and vision pieces as manager IC work, and advises against critical-path work.](https://assets.ai.engineer/sites/aiecode2025/sha256/25ddcc29a7db79d71dc00c2472ce61d04403b67acb96419d82e698a54439101b.webp) Julie Zhuo @joulee What kind of IC work can you do if you are a manager? Attended two dinners with dozens of senior Bay Area tech managers last week. Every single one of them said they were expected to be hands-on building with AI these days. But what to work on? 4 categories emerged: 1. Internal efficiency -- anything to make your team more productive, from communication tools to digital brain to team skills. 2. Quality-of-life improvement -- got something that bugs the heck out of you about the product but you can't convince anyone else to work on it? Just fix it yourself. 3. Celebration story -- artifact (usually video, image narrative, etc) that hypes up the work done by your team. 4. Vision piece -- describes a really freakin' cool future that your team can move towards. What NOT to do: take on any critical-path product work. (Because then you either do a poor job delivering on that, or managing.) Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 7 / 20 Made with Dia ### What three hours of coding a day looks like Coding blocks bookend the day at 9 AM and 5 PM. Two 30-minute steers redirect the agent between meetings, while it continues working through meetings and overnight. [6:48](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=408s) [A daily schedule places coding blocks at 9 AM and 5 PM, with two 30-minute steering sessions between meetings.](https://assets.ai.engineer/sites/aiecode2025/sha256/63f3f5944c5e9b63734e710abd82fbc2cf7e92fb0936aefba8fa6ab99dfe7784.webp) What ~3 hours of coding a day actually looks like 9 AM — Coding block Kick off the day’s run ↑ Review overnight + night-shift output 10 AM — Meetings Standups, 1:1s, interviews 11 AM — 30-min steer Give feedback, re-steer the run → Course-correct toward the goal 12 PM — Lunch + meetings Midday break, calls 1 PM — Meetings 1:1s, reviews 2 PM — Meetings Syncs, interviews 3 PM — 30-min steer Check progress, redirect → Second course-correction 4 PM — Meetings Reviews, syncs 5 PM — Coding block Wrap the day ↓ Set up the overnight run, let it go Two coding blocks bookend the day. Two 30-minute steers in between. The agent works through the meetings, and overnight. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 8 / 20 [illegible] Made with Dia ### Three tasks, one overnight loop At 5 PM, Dia/Cowork gathers context and writes a prompt, and the user sets up the run. Claude Code works overnight; at 9 AM, the user reviews and ships. The loop covers features, evals, and model training. [7:25](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=445s) [Building features, tuning AI features, and training models share a loop of gathering context, setting up a run, overnight work, and morning review.](https://assets.ai.engineer/sites/aiecode2025/sha256/06f3339f586d03b95b787c6b2e39f27ab361dccee2c0326ce9012706b9bcc87d.webp) Three examples of tasks — one overnight loop 1 · Building features Real production code. 2 · Evals & hill-climbing AI features tuned to high quality. 3 · Training models Custom models, trained overnight. 1 DIA / COWORK · 5 PM Gather context, write the prompt → 2 YOU · 5 PM Set up the run, answer questions ↓ 3 CLAUDE CODE · OVERNIGHT ☽ Do the work (build, train, hill-climb, etc.) ← 4 YOU · 9 AM ☀ Read the report, review & ship Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 9 / 20 [illegible] | 1782164052 Made with Dia ### Building features: give context, not tasks The speaker gives Dia a 30-second voice request to gather project context and trade-offs. The co-work agent can ask clarifying questions and turn that context into a prompt for a coding agent. [8:10](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=490s) [A comparison contrasts breaking a feature into prompted tasks with giving an agent context to make decisions and prepare a coding prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/ceeea1da57885de0b25a68db6432e43db5021ee2d537f27d61edc52ca15e691c.webp) Building features: give context, not tasks BUILDING FEATURES OLD WAY “What are the pieces, and how do I prompt each one?” You decompose the task yourself, then babysit the model step by step. NEW WAY “What’s the context the model needs to decide like I would, without me?” Let your co-work agent (I use Dia) read Slack, Notion, Jira, and the repo, ask you clarifying questions, then write the coding-agent prompt. It comes back richer than what I’d write by hand — and costs a 30-second voice dump. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 10 / 20 [illegible] | Made with Dia ### One prompt, the whole job The overnight prompt asks Claude Code to build and test a feature, prepare reviewer-friendly PRs, address review feedback, and leave a morning report. [9:38](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=578s) [An example overnight prompt asks a coding agent to implement and test a feature, prepare pull requests, review feedback, and report in the morning.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a45c609bbf754adb12b9fe99514e1160630339722703627f990932c9b12e993.webp) One prompt, the whole job — while I sleep BUILDING FEATURES · THE OVERNIGHT PROMPT CLAUDE CODE · 5 PM › Implement this whole feature: [Pasted prompt from Dia / Cowork agent] › Write the tests first, so they capture what it should do. › Test it end-to-end with computer use — click around, confirm it hits the goal. › Split it into reviewer-friendly PRs with clear descriptions; push; get CI green. › Run an AI code review in a clean subagent using the [skill] skill, then watch the PRs — read every bot comment, fix what's valid, resolve the threads. › Run this autonomously. Don't ask me questions. I'm going to bed — leave me a report in the morning. Modern models handle weeks of work in one run. Push the scope. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 11 / 20 Made with Dia ### Make an AI feature actually good Capture feedback alongside each run’s prompt, inputs and output, then turn a handful of JSON records into an evaluation set. The slide suggests 20 or 30 examples, though even 5 or 10 can work. [11:46](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=706s) [The slide recommends capturing real feedback with each AI feature use and collecting a handful of examples for evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/85fda5d35f6caea3d3ce95ca7147fa371a32f46803dd70ae5f0f9822d3f1ca5c.webp) Make an AI feature actually good JUN 30, 2026 PROTOTYPING AS LEADERSHIP 12 / 20 EVALS & HILL-CLIMBING Every AI feature has an LLM call to tune for quality, cost, and latency. 1 Capture real feedback Drop a “give feedback” button in the feature. Each use saves a JSON with the system prompt, the inputs, the output, and your feedback. 2 Collect a handful 20 or 30 is plenty. Even 5 or 10 examples work. You can knock this out in one 30-minute block during the day. Interface text: [illegible] | Made with Dia ### One end-of-day prompt turns feedback into a tuned feature A prompt asks Claude Code to turn 20–30 feedback JSONs into evals and hill-climb overnight. One recall eval rose from 2/7 to 7/7 in five hours; a teammate’s code reviewer went from 10% to 55% recall. [12:54](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=774s) [An overnight evaluation prompt and reported recall improvements.](https://assets.ai.engineer/sites/aiecode2025/sha256/010cbcf1c5161a74cf965afe731a11104079a211328980acab173e54f7cd6d90.webp) One end-of-day prompt turns feedback into a tuned feature EVALS & HILL-CLIMBING · THE OVERNIGHT PROMPT CLAUDE CODE · 5 PM › Here are 20–30 feedback JSONs — each has the system prompt, the inputs, the output, and my feedback. › Turn them into an eval set (store them in SQLite). › Talk me through it: how would you design evals that optimize for this feedback? › Build a harness that runs the call against the evals and hill-climbs until the scores go up. › Don't overfit — keep it general. Run overnight autonomously once we align on a scoring rubric. Deliver a full report alongside the updated prompts and harness. › Then save this flow as a generic /generate-evals-and-hill-climb skill so I can reuse it. WHAT I WOKE UP TO 2/7 → 7/7 On one of our recall evals, climbed to a perfect score over a 5-hour overnight run. A teammate's code reviewer rode the same loop from 10% to 55% recall. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 13 / 20 [illegible] | Made with Dia ### Train a custom model overnight ModernBERT-large without regex reaches 0.871 F1, versus 0.722 for the previous Opus-based ceiling. Adding regex raises recall from 0.784 to 0.854 but lowers precision and F1. [14:25](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=865s) [A table compares precision, recall and F1 for custom and frontier-model detectors.](https://assets.ai.engineer/sites/aiecode2025/sha256/664d403635f3eacdb0e64a02ed905181f44d21786aa4090920c730e4aee4a585.webp) TRAINING MODELS Train a custom model that beats the frontier – overnight A small fine-tuned model can replace a frontier call — cheaper, faster, and yours to control. Detector | Precision | Recall | F1 ModernBERT-large (no regex) | 0.980 | 0.784 | 0.871 ModernBERT-large + regex | 0.767 | 0.854 | 0.808 Opus fanout + regex (prev ceiling) | 0.744 | 0.701 | 0.722 Haiku fanout + regex (production today) | 0.701 | 0.594 | 0.643 Modern models can train SFT custom models with minimal oversight. Start by collecting some training data, then running an overnight prompt. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 14 / 20 Made with Dia ### Collect data as you go, then one prompt does the rest The prompt asks Claude Code to prepare data, train and evaluate models in a sandbox, and report cost, latency and quality against the LLM version. The speaker says it returned two trained models and a report by morning. [14:45](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=885s) [An overnight prompt outlines data preparation, model training, evaluation and reporting.](https://assets.ai.engineer/sites/aiecode2025/sha256/acc910857d238db46509c68b5e839ff00cadb1685d52aaa0489af34959e2ec72.webp) Collect data as you go, then one prompt does the rest TRAINING MODELS · THE OVERNIGHT PROMPT CLAUDE CODE · 5 PM › Here's the business context and the training data I collected off the existing LLM flow. › Clean up the training data; bolster it with synthetic data if it's thin (OpenAI + Anthropic keys attached). Use an ensemble of frontier models to push quality. › Suggest the best model class — give me a few options. Pick one or two best options to train. › Provision a sandboxed AWS box (not prod), pick the right GPU, train, test against the eval examples, then deprovision. › Report back: cost, latency, and quality vs. the LLM version, plus a plan to host it for inference. › Ask me any clarifying questions before you work autonomously. In the morning, have a report and the trained models ready for me. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 15 / 20 [illegible] Made with Dia ### Those are three examples. Try your own! As model capabilities shift every few months, try larger tasks to find their edge. Simon Last suggests work that would take a good engineer multiple weeks. [15:50](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=950s) [The slide encourages expanding task scope and includes a post about thinking bigger.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe23cd6e59ce0d31c3ec509a179ba28eef81500d14bb19f5a0ec895e4199696c.webp) Those are three examples. Try your own! The main thing to push on is task scope Modern models are capable of far more than feels intuitive, and the boundary keeps moving every few months. Pushing the scope of each run is how you find that edge — and how you build a real feel for the model curve and where it’s headed. Simon Last @simonlast 2/ Think bigger. This is the most common mistake I see: tasks scoped too small. At this point you want to be aiming for work that would take a good engineer multiple weeks. 8:13 PM · May 22, 2026 · 76.5K Views Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 16 / 20 [illegible] | Made with Dia ### This works because of organizational scaffolding AI reviewers check PRs, agents.md and CI gates set shared standards, and feature flags plus a prototype branch provide a place to share work without touching production. [16:32](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=992s) [Three cards describe AI code review, shared standards and CI gates, and feature flags with a prototype branch.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4411d82efef4d2f1c51f69e019f1176f1e454bb79e649f7f0f8400f6c591134.webp) This works because of organizational scaffolding You can only push this much code safely because the structure around the agent makes it safe. Build that first. AI code reviewer Bots review every PR, so you’re not burdening engineers. agents.md + CI gates Shared standards and green-CI requirements the agent must meet. Feature flags and prototype branch A safe place to share the vision without touching production. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 17 / 20 [illegible] Made with Dia ### Code hygiene matters The speaker’s own code has annoyed engineers and caused SEVs. From experience, they urge reviewing code before adding other reviewers. [17:11](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1031s) [The speaker says their code has annoyed engineers and caused incidents, emphasizing code hygiene.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f9a5cf6ae1bf6e2464b6f1f92610ac10e979bb4210286c591b3c787bdca86ca.webp) Code hygiene matters My code has annoyed my engineers. It has caused SEVs. Coding agents are not perfect (yet). You will be humbled. Hygiene matters. JUN 30, 2026 PROTOTYPING AS LEADERSHIP 18 / 20 Interface text: Made with Dia ### Some tactical tips on hygiene Manually test features, keep PRs small, and review your own code before assigning reviewers. The speaker warns that skipping self-review can leave obvious problems for engineers to catch. [17:31](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1051s) [Three code hygiene tips: manually test features, keep PRs small and readable, and review your own code before requesting review.](https://assets.ai.engineer/sites/aiecode2025/sha256/3cb0404caf970153515dee594b1d01e5b6f7dc24e7a186afdf126a6f404c79b5.webp) Some tactical tips on hygiene 1. Test everything before the PR goes up Computer use and tests are great, but still manually test the feature yourself. That morning block is good for this. 2. Small PRs, readable descriptions Break the work into reviewable pieces. You’re modeling what good looks like for the whole org. 3. Never put up code you haven’t reviewed yourself It’s tempting to pass it straight to another reviewer. Don’t. Read your own PR first. Interface text: JUN 30, 2026 PROTOTYPING AS LEADERSHIP 19 / 20 [illegible] | 1782164052 Made with Dia ## Full transcript [0:00](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=0s) [on-hold music] Hi, everyone. Thanks for coming. [0:16](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=16s) Uh, I'm Hursh Agrawal. I'm the CTO and co-founder of The Browser Company, makers of the Arc and Dia browsers. Uh, and I'm gonna talk about prototyping as leadership, as you get to a manager's schedule, how you can keep building. [0:28](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=28s) So this is my actual calendar from last week. Uh, I imagine this is ki- familiar to some of you in leadership. Um, that was my actual week. I, uh, have a whole org that reports up into me, so I have fifteen plus recurring meetings a week, uh, stand-ups, reviews, recruiting meetings, et cetera, uh, and seven direct reports. [0:48](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=48s) And I've noticed something over the last several months, which is I've started to consistently ship, you know, two to ten PRs a week, uh, and this is new. This was not possible several months ago. [0:58](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=58s) It's really because of these new AI agents. Uh, and notably, I also have a toddler at home, so I, like, cannot work nine-nine-six. You know, I'm working forty, fifty hours a week, so I really like to fit all this in into a regular week. [1:11](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=71s) And pre-AI agents, uh, as you grew as a leader, you had more of the org reporting to you, uh, you'd sort of-- the way you would influence the org was through communicating to people. [1:23](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=83s) Uh, so you'd, uh, write roadmaps, docs, meetings, and you'd sort of incept, uh, your context and what you wanted to build with your employees, uh, and your engineers. And now, interestingly, in the last few months, as coding agents have become more autonomous and able to handle longer tasks, the manager's schedule, as Paul Graham put it, is suddenly [1:44](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=104s) usable as building time. You can actually ship stuff. [1:48](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=108s) And so I think building is part of the job now. You can do it, first of all, as a leader, but it's-- I think it's also becoming necessary as part of the job for two reasons. [2:00](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=120s) First, uh, the technology world is changing, where suddenly the core technology that's part of our products are these frontier models that change every three months, which is a new dynamic that's come out. [2:11](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=131s) And as each new frontier model comes out, its capabilities change, the contours of what it's useful for change. It's, you know, how it actually reacts to prompting changes. And there's so much noise on Twitter, even internally for us, and so many opinions with each new model release on what's good and what's bad. [2:30](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=150s) And I found it is impossible to tell what a new model is good for unless you have your hands in it and you're using it all day long. And so having an ability to slot in building time into your schedule means you can figure out the contours of what is this new model family capable of, and both [2:46](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=166s) how do I direct my engineers in terms of, uh, setting expectations on how they should be building. But also getting a sense of how does this fit into our product, how does this fit into our business, you know, how is this-- how is our strategy gonna change because of this? [2:59](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=179s) What's gonna happen in three to six months when the open source models catch up? All of that intuition comes from actually using the models and building. [3:07](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=187s) I've also found even if you've built the intuition, it's actually tough to communicate that to other people who haven't played with the models. Uh, and so you're like, "Oh, this new thing is possible. [3:17](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=197s) It's gonna be amazing." And your engineer's like, "Okay. Yeah, sure." Um, so it's really helpful to have some s- time to actually build stuff so you can show them. [3:25](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=205s) You know, you can be like, "Hey, I built this prototype with this new model family. It works in the product. Here's an actual prototype you can play with." And this is just so much faster and more efficient than trying to convince people every three months when a new model family comes out. [3:39](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=219s) I also think leaders are really well suited for it. You know, you all have, uh, context, more context than anybody else, uh, in the organization about the business, the strategy, the trade-off, the decisions to be made, what needs to be done, what's, uh, what's coming up, what are imperatives. [3:54](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=234s) And so the steering of a leader, uh, as you are prompting these models is to- per token, uh, more impactful than an IC's. And so you can really fit this into a little bit of time. [4:06](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=246s) The delegation skill also transfers. Like, you delegate to people. That's sort of one of the core skills of being a leader, and that transfers to agents pretty well. You know, it's setting goals, giving context, checking in, even coaching the agent. [4:18](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=258s) You know, "What if you tried this?" Um, and then I've also found, I, I think this is maybe specific to now, uh, maybe this will change in several months as these models get better, but the models are really good at execution, but still not unbelievable at judgment. [4:32](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=272s) You know, often the model will come back and be like, "Hey, that algorithm idea is not possible," or like, "I can't do this." And so you have a, uh, as a leader, have to be like, "Oh, have you tried this thing?" [4:40](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=280s) And the model's like, "Oh, okay. Cool, cool, cool. I'll try that." And so it's a really nice dynamic you can have with these models if you have time slotted out for building, uh, that really leverages all the context you have. [4:52](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=292s) So the great Julie Zhuo had a great tweet about this, uh, a month ago. She polled some Bay Area technical leaders on what can you actually build, what should you be building in this new world. [5:02](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=302s) And four categories emerged. Uh, you could build internal tools or internal efficiency, uh, quality-of-life improvements, just like gardening around the code base and the product. This is really helpful. [5:11](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=311s) Um, I really like this. I learned from this, the celebration story. You can build artifacts to celebrate people on your team. And then arguably, I think the most important is the vision piece, is really playing with the new model families, understanding what's possible viscerally yourself, and then trying to fit that into the business and building products that [5:29](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=329s) can show that off to your engineers so you can really push, uh, the boundaries of what's possible with the product and business. [5:36](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=336s) I-- She's right, I would not take any critical path work. The last thing you wanna do is to have that dependent on you, especially you're gonna be dragged into fires and recruiting calls and meetings, et cetera. [5:45](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=345s) Uh, so really trying to do s- one of these four categories is, is optimal. [5:50](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=350s) So this is, this is my how I work, uh, and what two to three hours a day of coding can actually look like. Uh, so I usually have a morning coding block about an hour, and this is when I look over last night's code, what the, uh, agent did last night, review it. [6:06](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=366s) I'll talk a little bit more about this. Uh, and then a few maybe steering blocks throughout the day interspersed between one-on-ones and reviews and meetings and everything else you need to do. [6:14](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=374s) And then the most important block is that 5:00 PM block at the end of the day where you really set up whatever overnight run it is you want to run, and I'll talk about that, whether that's coding or training models or whatever you want the agent to do. [6:28](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=388s) I'll go over sort of three examples of tasks you can do, but really what I've found work for me is this one overnight loop. Uh, you first at-- in that 5:00 PM block gather context. [6:38](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=398s) I'll talk about that. Uh, you set up the run. You know, you get-- ask-- answer any clarifying questions. Uh, and then Claude Code or whatever your coding agent is overnight does the thing, you know, does the work for four, six, eight hours. [6:49](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=409s) And in the morning, you get this beautiful report, uh, and you figure out what to do with it, and then you ship whatever it is. So I'll talk about building features, which is sort of the most obvious one. [6:58](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=418s) Uh, I found this worked really well with evals and hill climbing, so optimizing AI features. Uh, and then a new thing actually in the maybe more recent, uh, generations of models is you can have these models train other ML models overnight too, and this works really, really well. [7:13](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=433s) So building features. Um, the big mindset shift for me that really helped was starting to think about building features not in terms of how do I build this feature and break it up and then give it prompting on how to do the individual tasks, but rather what is all the context this frontier model needs to be able [7:34](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=454s) to make decisions like I would make. So trying to give it as much context as possible 'cause if it's working for six hours, eight hours overnight, you're not gonna be there to steer it. [7:42](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=462s) And so you want it to have as much context as you do about the business goals or whatever you're trying to get it to do. So, uh, a little tip I found that has worked well for me, um, if you have a cowork agent, whatever your, uh, Claude Cowork or Codex or I recommend Dia, I feel like [7:56](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=476s) that's the best one, uh, not biased at all. Um, whatever it is that's connected to your Slack and your Jira, Confluence, Notion, the repo, et cetera, uh, just I-- before a meeting or something, I'll be like at like three, four PM, uh, be like, "Hey, I really wanna build this." [8:11](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=491s) Go and do like twenty minutes of research, and go dig through all of, uh, Slack and Notion and everything else, and come up with a Claude Code prompt that I can post-- paste into Claude Code. [8:21](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=501s) And just give me as much context as possible, the trade-offs, what we tried before, what works, what doesn't, why we're doing it, what is like the business context of this feature I wanna build or this model I wanna train. [8:31](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=511s) And that takes like thirty seconds to Wispr Flow into Dia or whatever your cowork agent is. And the nice thing is this comes back with a giant prompt that then you can paste into Claude Code or Cursor or Codex or whatever your coding agent is. [8:43](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=523s) So this is an example of a, a 5:00 PM prompt I'll have that I give it before I go to bed. I'll say, "Implement this whole feature," and then just paste all of that context I got from my cowork agent. [8:54](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=534s) Uh, and then it's important to think about verification. So as this, uh, agent is running overnight, you want it to verify how it's doing as it's building and testing the feature. [9:03](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=543s) So first I'll be like, "Hey, write the test first so they capture what you do." This is really helpful 'cause I've, I found with the coding agents, if you-- if they write tests afterwards, they're a little sloppish. [9:12](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=552s) Um, and then I'll t-tell it to test the end-to-end flow with computer use. You know, go around, click around, just make sure the flow works and matches the business context I gave you. [9:21](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=561s) And so the modern models are really good at reasoning about, okay, does this flow actually solve the problem I was trying to solve? [9:28](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=568s) And then, uh, I usually prompt it to do as much work as possible so that when I wake up in the morning, the stack is ready. So split this into reviewer-friendly PRs with clear descriptions, m-manage CI and get CI green, and just make sure you're, you're monitoring, uh, just to make sure the, the whole stack is passable. [9:47](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=587s) AI code review is really important. We have a bunch of internal AI code reviewers. Uh, you know, previ-- the previous talk was talking about Qodo, another AI code reviewer. [9:56](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=596s) If you don't have any, I would look on Twitter for a bunch of AI code review skills, and then just prompt it, "Hey, once you're done, run this AI code review skill in a clean sub-agent and then fix those things." [10:05](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=605s) And then watch the PRs. Fix every bot comment, every, you know, CI check that shows up, any sort of, uh, anything you need to do, resolve the threads, run this autonomously. [10:14](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=614s) Uh, don't ask me questions. And then I'll like throw in a little encouraging something, you know, be like, "You'll do great. I believe in you. It's gonna be great." [10:21](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=621s) Um, I don't know if that helps or not, but I, it's, it's a-- I tend to do that. Uh, and then I tell them I'm going to bed. Just leave me a report in the morning on what you do. [10:28](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=628s) I need the full stack ready and a report on what trade-offs you made, how you did. Uh, and it does great. Uh, it actually-- modern models, the new Opus 4.8 or the new GPT, they can handle what used to be, you know, weeks of work, uh, in one overnight run, and you come back in the morning with [10:42](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=642s) this, uh, beautiful package ready for you. It actually makes the mornings kinda nice 'cause you have this little present ready for you. Another example is optimizing AI features. So first, you're at the AI engineering conference. [10:54](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=654s) Uh, I assume you all are familiar with this. When you first build an LLM call or agent into a, into a feature, it's not optimized. You know, you need to optimize it, hill climb on some evals to get the quality, the latency, the cost to where you want it to be. [11:08](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=668s) This is also all doable overnight, um, especially with the modern coding agents. So, uh, usually if I'm prototyping a feature, I will, uh, put a little feedback button on it. [11:18](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=678s) So after the LLM call or the agent runs, I'll be like, "Hey, uh, just add a feedback button and a little text box." And then in one of my one-hour slots or thirty-minute slots throughout the day, I'll just take the time to, uh, collect a bunch of feedback. [11:30](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=690s) So I'll, uh, you know, run it a few times, collect a bit of feedback, be like, "Hey, this was bad," or, "This was good," or, "This could've been better." [11:37](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=697s) And then just have it save JSON dumps on my downloads folder. So it'll save a dump of each of the runs with the system prompt and the inputs and my feedback. [11:46](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=706s) Uh, and you just collect a handful, you know. Like even four, five, ten are fine. If you can get some other, uh, coworkers to run it, that would be great. [11:54](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=714s) You could collect like twenty or thirty. And then similarly, uh, at 5:00 PM when you're setting up an overnight run, uh, just say, "Hey, like here are, you know, ten, twenty, thirty feedback JSONs. [12:05](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=725s) This is all the information they have in them. Turn this into an eval set. Like just do it locally with SQLite or Markdown or whatever you want. And then over the next few minutes, talk me through how would you design evals or scoring functions to optimize for this, and let's do it interactively." [12:19](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=739s) And then I want you to, A, build a harness that runs this call against the evals and hill climbs until the score goes up. Like, just do whatever you need to do to get these scores to go up. [12:29](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=749s) And obviously, with this kind of thing, if you have such little data, overfitting is a problem. But even that I found, these modern models, you can just tell it to not overfit, and it actually does a pretty good job. [12:38](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=758s) Uh, and so just say, "Don't overfit," just keep it general, uh, "Run overnight autonomously until we align on a scoring rubric, and then give me a full report, uh, in the morning." [12:47](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=767s) And then a little tip, it's also nice to just say, "Hey, save this flow as a generic skill so I can reuse it." Uh, and that allows you to improve this flow over time and improve that skill, and this works. [12:58](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=778s) Actually, in the morning, you wake up to ideally something hill climbed. Uh, and actually, when using that feature, when we, we shipped these features to employees to, to further dogfooding to production, it actually does improve the quality. [13:09](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=789s) You know, these are not overfit. Um, and we, we use this flow quite a bit internally. Uh, we have an internal code reviewer similar to, uh, some of the external ones, and same thing there. [13:18](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=798s) We had that hill climb to pretty good quality just o- with these overnight runs. [13:23](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=803s) And then the last example, uh, and this is again pretty new and really interesting, uh, you can train custom models with this mechanism overnight, you know, in one night. [13:31](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=811s) So, uh, this is an example of a ModernBERT PII classifier we trained. We were trying Opus and Haiku for it, and it was expensive and latency was not great, and we just couldn't get the precision recall to be amazing. [13:43](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=823s) And so, uh, we collected a bunch of training data, and then overnight one night I was like, "Here is a bunch of business context and a bunch of training data we collected. [13:53](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=833s) Uh, A, clean up the training data, Claude. Just bolster it with synthetic data, whatever you need to. Here's a bunch of OpenAI and Anthropic keys. Uh, use an ensemble of frontier models to push the quality. [14:04](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=844s) I don't know anything about what ML model would be good here, so, like, you decide. You know? Just give me the best model class. Give me a few options, and in fact, train two. [14:11](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=851s) Like, don't even train one. Train two separate ones." Uh, and then I gave it AWS access. I was like, "Give me a provision sandbox. Do not give it prod." [14:18](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=858s) That's not a good idea. That's how you take prod down. Um, pick the right GPU and EC2 cluster, like, train it, test against eval examples, deprovision it, whatever you need to do, just give me back the trained models, uh, and then give me a report in the morning. [14:30](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=870s) And in fact, even put in how I would host this with my code base on inference once this model's ready. And again, ask me any clarifying questions. You can do it. [14:38](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=878s) I believe in you. You're gonna do great. Uh, and in the morning, just have this ready." And of course, it works beautifully. In the morning you have, uh, two ML models trained, a full report, great results, uh, and we've, uh, pushed a bunch of these to production. [14:52](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=892s) So those are three examples. Um, I would try, try your own. Uh, but I think the main takeaway for me in doing these is really to push on task scope. [15:01](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=901s) So this was a, a tweet by Simon Last, one of the Notion founders, uh, and he, he called out these modern models are just capable of so much more than we think. [15:10](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=910s) Uh, and I think it's our jobs as leaders to really understand the contours of how much we can push these models, 'cause it affects both how we lead and how we influence our teams, but also how they fit into our products and what these, our new products are capable of in terms of utility. [15:23](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=923s) So I'd really push you overnight, try to think of how do I give it more and more, uh, scope so that we can do, you know, weeks of work, months of work overnight, and see what these models are really capable of. [15:35](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=935s) I will say a caveat, a lot of this works because of existing organizational scaffolding, so you'll have to build this up in your organization. You know, AI code reviewers, this really helps. [15:44](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=944s) We have an internal one we trained, but also just external ones, whatever you need. Um, AGENTS.md hygiene, you know, CLAUDE.md, AGENTS.md, proper CI that you can trust. And then we have a bunch of other things like, uh, really sophisticated feature flags. [15:59](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=959s) Uh, we have a prototype branch that you can push to that goes to employees but it doesn't go to production. Just layers like that so you're not taking prod down, basically, as you're prototyping and showing these things off. [16:10](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=970s) Not amazing for the CTO to take prod down. [16:14](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=974s) I will say again, code hygiene really matter, really matters. I've been humbled a lot. You know, my code has annoyed engineers. It's caused SEVs. Uh, coding agents are not perfect yet, so just, uh, your mileage may vary. [16:25](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=985s) You will get humbled, but I, I still think it's worth it, uh, 'cause you learn so much and you can model what taking on more and more scope, even with a busy calendar, looks like. [16:34](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=994s) Some tactical tips on hygiene. Uh, I would test everything before the PR goes up. That morning slot, that's a lot of what I'm doing is just testing what the overnight run did. [16:43](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1003s) Uh, even though there's computer use, et cetera, it's really important you test it yourself. Um, small, readable PRs are really helpful. I think they, you're modeling to the rest of the team what good looks like. [16:54](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1014s) So if you're putting up three 5,000-line PRs, uh, other engineers on the team are gonna start doing that, and that's not amazing. So, uh, it's really important your hygiene is really good, 'cause it's modeling for the rest of the team what good looks like. [17:05](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1025s) Um, and then, ooh, this is so tempting. It's so tempting to put other reviewers on code you haven't read yet. Don't do it. It's, uh, mostly 'cause you're gonna look like an ass, 'cause there's gonna be something in that code that is gonna be so obviously bad, and then some senior engineer is gonna call you and be [17:20](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1040s) like, "Yo, why didn't you read this?" So, uh, I speak from experience. Read the code, review it properly before you add anybody else to it. [17:29](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1049s) So hopefully this convinces you building is part of the job now. Uh, you can do it, uh, even with just one or two hours a day, and you should. [17:37](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1057s) Uh, you'll start to feel the models every three months as the new model families come out, and you'll learn the skill on how to push scope as you are, uh, instructing them. [17:46](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1066s) And you'll be able to show the, the, the team and the org what's possible as well. You know, just because you h- as you have this capability, you'll automatically think of ideas on what to slot in for these overnight runs, and then you can model for the team, "Hey, this is what the future looks like." [17:58](https://www.youtube.com/watch?v=bdHaOXZOhcM&t=1078s) Thank you. [audience applauding] [upbeat music]