# How Software Factories Improve Themselves — Suraj Gupta, Warp Suraj Gupta, Warp AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/TN3mj92oZ8I-software-factories-improve-themselves) · [Video](https://www.youtube.com/watch?v=TN3mj92oZ8I) ## How Software Factories Improve Themselves Suraj Gupta explains how Warp improves recurring agent work through human-reviewed skill changes, reusable memories, and model routing informed by task-specific evaluations. ### The factory needs its own improvement loop [0:12](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=12s) A coding agent can finish an assignment while leaving the next run no better prepared. That becomes a maintenance problem when agents repeatedly handle work from issue triage through production. Suraj Gupta, who leads harness development at Warp, frames the next engineering responsibility as building and maintaining these software factories—and improving how the factory works over time. [0:12](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=12s) ![Source frame: The factory needs its own improvement loop](https://ai.engineer/api/talk-essay-media/cef2607cf4ce88e768c0424e0d399bd669366c8a4c4855ed68fb451c544e1215) Source frame: The factory needs its own improvement loop Warp began with a modern terminal and an agentic development environment; the work described here extends into a cloud agent platform for teams running factories. Three mechanisms organize the talk: skills retain procedures, persistent memory retains facts from previous work, and model routing changes which model handles a task. Each gives the factory something it can carry forward into later runs. ### Turn triage feedback into a reviewed skill change [2:04](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=124s) A skill gives an agent procedural memory: instructions for a task it will perform repeatedly. A triage skill might explain how to reproduce an issue. But the procedure can become dated as agents discover useful approaches and humans correct their decisions. Unless that experience changes the skill, later runs keep starting from the old instructions. [2:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=126s) ![Source frame: Turn triage feedback into a reviewed skill change](https://ai.engineer/api/talk-essay-media/71a59c1463e6b0fedf453a4edabc68e365316835382e42a104508e30efb54291) Source frame: Turn triage feedback into a reviewed skill change Warp separates execution from improvement. The inner-loop agent applies the skill and does the triage. An outer-loop agent examines those runs, looks for mistakes, and collects human feedback. Its output is a proposed change to the procedure that future triage runs will use. The demonstration uses the triage workflow in Warp’s open-source client repository. A newly submitted GitHub issue triggers the agent. It identifies missing information that would help decide whether the request should become new work, be deduplicated against an existing issue, or receive another disposition. The outer loop then examines those decisions alongside feedback from the issue thread. - **Reactions:** Thumbs-up and thumbs-down provide feedback signals on the issue. - **User comments:** Follow-up discussion supplies additional feedback about the triage. - **Employee comments:** Warp employees can directly correct or guide the agent’s work. How does feedback become a change without immediately rewriting the running agent’s behavior? The loop below puts a pull request between the outer-loop proposal and the skill used in subsequent runs. Git records how the procedure changes, and a human reviews the update before accepting it. That review matters because an improvement agent can also make a mistake and produce a worse triage procedure. A skill improves through a separate review loop Execution produces evidence; the outer loop proposes a procedural change. Human review separates that proposal from its adoption. - Triage skill — Procedure used for recurring issue triage. - Inner-loop agent — Applies the skill to incoming GitHub issues. - Outer-loop agent — Examines decisions, mistakes, reactions, and comments. - Pull request and human review — Proposed skill edits remain visible in Git. - Triage skill → Inner-loop agent (Guides execution) - Inner-loop agent → Outer-loop agent (Runs and feedback) - Outer-loop agent → Pull request and human review (Proposes skill update) - Pull request and human review → Triage skill (Reviews updates before adoption) ### Let the next investigation use what the last one learned [5:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=332s) Procedures do not capture everything worth remembering. Consider a Sentry agent that investigates an issue, gathers context, finds the root cause, and fixes it. When a similar issue appears later, a fresh run may fail to find the same cause. Even if it succeeds, it spends tokens gathering context the factory already gathered once. [5:39](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=339s) Persistent memory carries that investigation forward. Gupta describes it as a fact store scoped to an agent. An outer-loop agent extracts facts, learnings, and outcomes from the inner-loop run, making them available to future runs. The retained material answers what previous work discovered; the skill still describes how to do the work. In Oz, Warp’s cloud agent platform, the Sentry agent has accumulated memories from earlier root-cause analyses. Reuse also creates a curation problem: an agent can retain an unimportant observation or a conclusion that worked locally but should not guide future work. Oz exposes versioning and human controls to create, edit, or delete memories, along with links to their source runs. Opening a source run fails during this demonstration; Gupta suspects that changing IP addresses and staging IP restrictions explain the failure. That provenance navigation is described rather than completed on screen. Its practical purpose is clear: a questionable memory can be examined in the context that produced it and removed if it should not persist. The Sentry example then develops into a visible record of reuse. A later run performs another investigation and reports which memories it used—five in the demonstrated store. Gupta attributes faster work to that reuse. The concrete change is that earlier investigation results now participate in a later run; the demonstration does not quantify the speed or token savings. The memory layer is presented as working across harnesses running in Oz. Automatic memory creation therefore belongs to the platform’s surrounding workflow, while humans retain the ability to review, version, delete, and edit what it remembers. Changing the harness need not mean giving up this shared mechanism for retaining previous work. ### Choose models for recurring classes of work [8:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=526s) A factory multiplies the cost of a model choice across recurring work. Using **Opus** for every triage task or simple CI fix can become prohibitively expensive. Model routing makes that choice depend on the task, so relatively simple work does not automatically inherit the cost of the model selected for harder work. [8:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=526s) ![Source frame: Choose models for recurring classes of work](https://ai.engineer/api/talk-essay-media/a2a71970fbc9a654fa5e78503e3d5351d9cfc869ab6032e8367784c86b6d31fd) Source frame: Choose models for recurring classes of work Warp offers two ways to make that decision: - **Auto models:** Warp maintains routers and evaluates new models for Pareto efficiency—the tradeoff between useful performance and cost, rather than assuming one model should handle everything. Gupta recommends this as a starting point over pinning every task to Opus or **Haiku**. - **Custom routing rules:** A configuration assigns classes of tasks to models. The demonstration includes database migrations assigned to **GLM**, plus a separate assignment for runbooks and API documentation. These rules express the operator’s judgment about which model suits each kind of work. Custom rules alone leave the choice “more of an art than a science.” The next step described in the talk is customer-facing evaluation: define what matters in your workflow, vary routing choices, and compare models on a specific class of issues. A generic benchmark cannot settle whether Haiku, GLM, or Opus handles your particular work best. At the time of the recording, these customer-facing evaluation tools are planned work. ### Use evaluations to improve the routing rules [11:19](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=679s) Warp already uses an internal evaluation sidecar alongside its routing rules. A prompt is run through multiple agents in Oz using different models, and the comparisons help determine which models work better for which tasks. This supplies the missing improvement mechanism. Routing rules begin as choices; repeated comparisons give the team evidence for changing those choices. [11:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=702s) What relationship does this evaluation make visible? The diagram separates the rules used to choose a model from the comparisons used to inform those rules. The useful output is knowledge about a task class, which can guide later routing rather than remaining an isolated result from one run. One internal finding changes a concrete routing decision: Gupta reports that GLM handles UI tasks well enough that Warp can use it more efficiently than Opus for that work. The talk supplies no quantitative scores, scoring procedure, or task distribution, so this supports Warp’s reported routing choice rather than a general ranking of the models. The closing product direction is to bring these evaluations into customers’ own workflows, allowing each factory to improve its routing with evidence from the work it actually performs. Comparing models informs future routing The internal sidecar compares agents using different models on a prompt. Task-specific findings inform model choices; customer-facing evaluations are the proposed next step. - Prompt — Work submitted for an agent to perform. - Agents across models — Oz runs multiple agents using different models. - Evaluation sidecar — Determines which models work better for task classes. - Routing choices — For example, Warp reports using GLM efficiently for UI tasks. - Prompt → Agents across models (Run across models) - Agents across models → Evaluation sidecar (Compare task performance) - Evaluation sidecar → Routing choices (Inform model selection) ## Key takeaways - Separate the agent doing recurring work from the agent improving its procedure. Propose skill changes through pull requests so humans can review them and Git can preserve their history. - Retain investigation facts for later runs, and keep memories editable and traceable so a mistaken or narrowly useful conclusion does not become permanent guidance. - Improve model routing with comparisons on your task classes. Warp’s internal UI-task finding illustrates this approach; customer-facing evaluations were still planned at the time of the talk. ## Resources - [Warp](https://www.warp.dev/) — The development environment and cloud agent platform used in the demonstrations of triage skills, persistent memory, and model routing. ## Slides ### Suraj Gupta Suraj Gupta leads harness development at Warp, which is building a cloud agent platform for teams to build software factories. He previously worked at Snowflake. [0:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=28s) [Suraj Gupta’s introduction slide, with his Warp role, previous Snowflake role, and portrait.](https://assets.ai.engineer/sites/aiecode2025/sha256/cdf2a5eca25f6d514a7dac6c40a347ed33b6a6c8f285be759c3fa58552a5a5c6.webp) Suraj Gupta AI Engineering Lead @ Warp Previously @ Snowflake Interface text: Figma File Edit View Object Text Arrange Plugins Widgets Window Help Untitled AI Engineer Expo Talk Audience · AI Engineer Expo Talk Growth New slide Building Self-Improving Agent Software Factories Self-improvement How do factory agents get better over time? Skills Agent Memory Design Animate 63% Slide 3 Template style Custom Style Matter Background Color 1 Selection colors Color 2 Color 1 Add presenter notes... Start with a draft Share ### Self-improvement The diagram places each factory agent run inside a larger loop: agent results are consolidated and the agent configuration is updated for future runs. [0:36](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=36s) [Diagram of a factory agent inside a self-improvement loop that consolidates results and updates its configuration.](https://assets.ai.engineer/sites/aiecode2025/sha256/0736d082f82e76609b785e15e545faff19e49409105c1c3afe4ad5fa8e14a501.webp) Self-improvement Consolidation of agent results Self-improvement loop New agent run started Agent completes execution Factory Agent Context surfaced New learnings / outcomes Agent config updated 4 Interface text: Figma File Edit View Object Text Arrange Plugins Widgets Window Help Cooking · in 28m Wed Jul 1 5:01 PM Untitled AI Engineer Expo Talk Audience · AI Engineer Expo Talk - Fi[illegible] AI Engineer Expo Talk Growth New slide Design Animate 63% Share Slide 4 Template style Custom Style Matter Background Color 1 Selection colors Color 2 Color 2 Color 2 Accent 1 Color 1 000000 100% FFFFFF 100% Add presenter notes... Start with a draft ### How do factory agents get better over time? Gupta outlines three ways agent factories can improve: skills, persistent memory, and model routing. [1:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=113s) [The slide lists skills, memory, and model routing as ways factory agents improve.](https://assets.ai.engineer/sites/aiecode2025/sha256/cffd90389631da2cf2c0686c0523f64b48d79ed07294914c2e695f1556bc0e95.webp) How do factory agents get better over time? Skills Memory Model routing 5 Interface text: Figma File Edit View Object Text Arrange Plugins Widgets Window Help Cooking · in 28m Wed Jul 1 5:01 PM Untitled AI Engineer Expo Talk Growth New slide Design Animate 63% Slide 5 Template style Custom Style Matter Background Color 1 Selection colors Color 2 Accent 1 Color 1 0B5AFF 100% FFFFFF 100% Add presenter notes... Start with a draft ### Skills A triage agent classifies new issues in the inner loop. An outer-loop agent reviews comments and creates PRs to improve the skill, drawing on feedback and observations from triage runs. [2:04](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=124s) [An inner issue-triage loop connects to an outer loop that reviews feedback and creates pull requests to improve skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/010259806b1fb4b3a4ae56c0f0a4c8579ea4939dd459fbb5513d13d77b170b95.webp) Skills Inner loop issue-triage/SKILL.md New issues get created Triage agent classifies issues Outer loop improve-issue-triage/SKILL.md Skill updater reviews comments Agent creates PRs to improve skills 6 Interface text: Figma File Edit View Object Text Arrange Plugins Widgets Window Help AI Engineer Expo Talk Growth New slide Design Animate 63% Slide 6 Template style Custom Style Matter Background Color 1 Selection colors Color 2 Color 2 Accent 1 Color 1 000000 100% 0B5AFF 100% FFFFFF 100% Add presenter notes... ### Issue-triage skill Warp’s issue-triage skill classifies new GitHub issues, applies a matching label and requests feedback. A separate agent updates the skill’s version when it revises the instructions. [3:17](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=197s) [A GitHub page shows the issue-triage skill’s description, version, and initial instructions.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c7b8646c4ef115222eec59449284ba2a6f0424fe7b605c62527d73782ffbae2.webp) name: triage-issue description: Triage a newly opened GitHub issue into one of three buckets (ready-to-implement, needs-info, duplicate), apply the matching label, and post a triage comment requesting feedback. Use when a new issue needs triage. version: 1 Triage Issue You are triaging a single GitHub issue. You will be given the repository and issue number. Use the gh CLI for all GitHub interactions. Skill version This skill is at version 1. Always embed the current version number in the triage comment marker (see below). This version is bumped automatically by the improve-triage-skill agent when it revises this skill. Steps 1. Read the issue. gh issue view --repo / --json title,body,author,labels,createdAt Interface text: github.com/warpdotdev-demos/issue-triage-loop/blob/main/.agents/skills/triage-issue/SKILL.md warpdotdev-demos / issue-triage-loop Code Issues 5 Pull requests 1 Discussions Actions Projects Wiki Security and quality Insights Settings main issue-triage-loop / .agents / skills / triage-issue / SKILL.md zachlloyd and oz-agent Add issue triage self-improvement loop: skills, workflow, seed script 09fd53d · 3 weeks ago History Preview Code Blame 80 lines (60 loc) · 3.96 KB Raw ### Issue-triage workflow When an issue is opened, a GitHub Actions workflow runs the triage agent with the issue number and title, then prints its output. [4:00](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=240s) [A GitHub Actions workflow runs an agent to triage newly opened issues.](https://assets.ai.engineer/sites/aiecode2025/sha256/14ad9da1162341bcbabc04d0b99b6d40539f1ef7b46d1ddee171b5eefde25608.webp) name: Triage new issues on: issues: types: [opened] jobs: triage: runs-on: ubuntu-latest permissions: contents: read issues: write steps: - uses: actions/checkout@v6 - uses: warpdotdev/oz-agent-action@v1 id: agent env: GH_TOKEN: ${{ github.token }} with: skill: 'triage-issue' prompt: | Triage issue #${{ github.event.issue.number }} in the repository ${{ github.repository }}. Issue title: ${{ github.event.issue.title }} warp_api_key: ${{ secrets.WARP_API_KEY }} - name: Print agent output env: AGENT_OUTPUT: ${{ steps.agent.outputs.agent_output }} run: printf '%s\n' "$AGENT_OUTPUT" Interface text: github.com/warpdotdev-demos/issue-triage-loop/blob/main/.github/workflows/triage-new-issues.yml Files main Go to file .agents .github/workflows triage-new-issues.yml scripts README.md issue-triage-loop / .github / workflows / triage-new-issues.yml View Runs zachlloyd and oz-agent: Print agent output via env var to avoid shell interpolation failures b9aa228 · 3 weeks ago History Code Blame 28 lines (26 loc) · 752 Bytes Raw ### Improve Triage Skill The improvement skill reviews the triage agent’s past decisions and proposes edits in a pull request. It starts with issues updated in the last 14 days and looks for versioned triage comments. [4:13](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=253s) [GitHub preview introducing the triage improvement skill and finding past decisions.](https://assets.ai.engineer/sites/aiecode2025/sha256/b2ad52904042968382eb9463a6d350d1a6e71cf6cc8ca86aabc4ae55a8083178.webp) name: improve-triage-skill description: Observe how humans responded to past automated issue triage and propose improvements to the triage-issue skill via a pull request. Use on a schedule or on demand to evolve the triage skill from real feedback. Improve Triage Skill You are the self-improvement half of an issue-triage loop. Your job is to study how maintainers and issue authors reacted to past triage decisions made by the triage-issue skill (.agents/skills/triage-issue/SKILL.md), extract generalizable lessons, and propose edits to that skill in a pull request. Use the gh CLI for all GitHub interactions. Work in the repository this skill lives in. Step 1: Find past triage decisions Find issues updated in the last 14 days that carry a triage comment. Triage comments always begin with the marker , where is the skill version that produced them. gh issue list --repo / --state all --search "updated:>=" \ --json number --limit 100 For each issue, fetch its comments and look for the marker: gh api repos///issues//comments Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Cooking · in 26m Wed Jul 1 5:03 PM Sentry Demo | Memories - Oz Runs - Oz Warp Ask Gemini https://github.com/warpdotdev-demos/issue-triage-loop/blob/main/.agents/skills/improve-triage-skill/SKILL.md Code Issues 5 Pull requests 1 Discussions Actions Projects Wiki Security and quality Insights Settings Files main Go to file .agents/skills improve-triage-skill SKILL.md triage-issue .github scripts README.md issue-triage-loop / .agents / skills / improve-triage-skill / SKILL.md zachlloyd and oz-agent Add issue triage self-improvement loop: skills, workflow, seed script 09fd53d · 3 weeks ago History Preview Code Blame 89 lines (61 loc) · 5.15 KB Raw ### Find Decisions and Collect Feedback The improvement skill searches for triage decisions on issues updated in the last 14 days, then checks reactions, human replies and label changes for feedback on the bot’s choices. [4:22](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=262s) [GitHub preview showing instructions to find triage comments and collect feedback signals.](https://assets.ai.engineer/sites/aiecode2025/sha256/52cc9be0a4d685586e5fb48e79c72d7467864c1c657d90fac8244aaa1af124c3.webp) decisions made by the triage-issue skill (.agents/skills/triage-issue/SKILL.md), extract generalizable lessons, and propose edits to that skill in a pull request. Use the gh CLI for all GitHub interactions. Work in the repository this skill lives in. Step 1: Find past triage decisions Find issues updated in the last 14 days that carry a triage comment. Triage comments always begin with the marker , where is the skill version that produced them. gh issue list --repo / --state all --search "updated:>=" \ --json number --limit 100 For each issue, fetch its comments and look for the marker: gh api repos///issues//comments Skip issues with no triage comment. Record for each triaged issue: the triage comment ID, the skill version from the marker, and the bucket the bot chose (from the ## Triage: heading). Step 2: Collect feedback signals For each triaged issue, gather: 1. Reactions on the triage comment — 👍 means correct, 👎 means incorrect: gh api repos///issues/comments//reactions 2. Human replies after the triage comment — corrections ("this isn't a duplicate", "this is actually actionable"), answered follow-up questions, or complaints. Ignore other bot comments. 3. Bucket drift — compare the bucket the bot applied with the issue's current labels: [illegible] Interface text: github.com/warpdotdev-demos/issue-triage-loop/blob/main/.agents/skills/improve-triage-skill/SKILL.md issue-triage-loop / .agents / skills / improve-triage-skill / SKILL.md Files main Go to file .agents/skills improve-triage-skill SKILL.md triage-issue .github scripts README.md Preview Code Blame 89 lines (61 loc) · 5.15 KB Raw Top ### Feedback Signals and Lessons The improvement skill compares reactions, human replies, label changes and duplicate decisions, then turns those signals into generalizable lessons for the triage skill. [4:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=275s) [GitHub preview listing feedback signals and examples of generalizable lessons.](https://assets.ai.engineer/sites/aiecode2025/sha256/52d1f8e44e2f5119a9affa2200bcc6f6b0725648d8d189f33b5d4427053a4858.webp) Skip issues with no triage comment. Record for each triaged issue: the triage comment ID, the skill version from the marker, and the bucket the bot chose (from the ## Triage: heading). Step 2: Collect feedback signals For each triaged issue, gather: 1. Reactions on the triage comment — 👍 means correct, 👎 means incorrect: gh api repos///issues/comments//reactions 2. Human replies after the triage comment — corrections ("this isn't a duplicate", "this is actually actionable"), answered follow-up questions, or complaints. Ignore other bot comments. 3. Bucket drift — compare the bucket the bot applied with the issue's current labels: gh issue view --repo / --json labels,state,stateReason A maintainer replacing the bot's bucket label is ground truth that the bot was wrong. 4. Duplicate accuracy — issues closed as duplicate (stateReason: "not_planned" with a duplicate reference, or maintainer comments saying so) that the bot did NOT bucket as duplicate, and issues the bot called duplicate that maintainers kept open. Step 3: Synthesize lessons Turn the signals into a small number of generalizable lessons. A good lesson describes a category of issue and the correct handling, e.g.: • "Crash reports without OS/version info were marked ready-to-implement twice and corrected to needs-info — require environment details for crash reports." • "Feature requests that name a desired UI change but no motivation were accepted by maintainers — do not demand acceptance criteria for small UI tweaks." Bad lessons are one-off fixes ("issue #14 was mislabeled") — never write those. Weight the signals: maintainer relabels and explicit correction replies are strong; reactions are moderate; absence of feedback is weak positive (silence usually means the triage was acceptable). Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Cooking · in 26m Wed Jul 1 5:04 PM Sentry Demo | Memories - Oz Runs - Oz Warp Ask Gemini Work github.com/warpdotdev-demos/issue-triage-loop/blob/main/.agents/skills/improve-triage-skill/SKILL.md Files main Go to file .agents/skills improve-triage-skill SKILL.md triage-issue .github scripts README.md issue-triage-loop / .agents / skills / improve-triage-skill / SKILL.md Top Preview Code Blame 89 lines (61 loc) · 5.15 KB Raw ### Synthesize Lessons and Edit the Skill The outer-loop agent turns feedback into generalizable lessons and updates the triage skill only when a lesson is well supported. The instructions also call for incrementing the skill version. [4:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=284s) [GitHub preview explaining how to synthesize lessons and update the triage skill.](https://assets.ai.engineer/sites/aiecode2025/sha256/14379395d5f6786ff41670eedba0b39310c219f831adede1254bd89786107128.webp) gh issue view --repo / --json labels,state,stateReason A maintainer replacing the bot's bucket label is ground truth that the bot was wrong. 4. Duplicate accuracy — issues closed as duplicate (stateReason: "not_planned" with a duplicate reference, or maintainer comments saying so) that the bot did NOT bucket as duplicate, and issues the bot called duplicate that maintainers kept open. Step 3: Synthesize lessons Turn the signals into a small number of generalizable lessons. A good lesson describes a category of issue and the correct handling, e.g.: • "Crash reports without OS/version info were marked ready-to-implement twice and corrected to needs-info — require environment details for crash reports." • "Feature requests that name a desired UI change but no motivation were accepted by maintainers — do not demand acceptance criteria for small UI tweaks." Bad lessons are one-off fixes ("issue #14 was mislabeled") — never write those. Weight the signals: maintainer relabels and explicit correction replies are strong; reactions are moderate; absence of feedback is weak positive (silence usually means the triage was acceptable). Step 4: Edit the triage skill If (and only if) you found at least one well-supported lesson, edit .agents/skills/triage-issue/SKILL.md: • Append each lesson as a bullet under ## Learned guidelines (replace the "(None yet.)" placeholder if present). Each bullet should be a single imperative guideline, optionally with a parenthetical note of the evidence, e.g. (observed: 2 maintainer relabels). • If a lesson contradicts an existing learned guideline, revise or remove the old one rather than adding a conflicting bullet. • If a lesson warrants it, adjust the bucket definitions in the classification step directly — but keep the three buckets exactly as they are (ready-to-implement, needs-info, duplicate). • Bump the version: increment version: in the frontmatter, the "version N" text in the ## Skill version section, and the v: in the comment template. Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Cooking · in 25m Wed Jul 1 5:04 PM Sentry Demo | Memories - Oz Runs - Oz Warp Ask Gemini https://github.com/warpdotdev-demos/issue-triage-loop/blob/main/.agents/skills/improve-triage-skill/SKILL.md Files main Go to file .agents/skills improve-triage-skill SKILL.md triage-issue .github scripts README.md issue-triage-loop / .agents / skills / improve-triage-skill / SKILL.md Preview Code Blame 89 lines (61 loc) · 5.15 KB Raw Top ### Guardrails and Pull Request The improvement skill changes triage only when lessons are well supported, then opens a pull request so humans can review changes tracked in Git. Weak or conflicting feedback leaves the skill unchanged. [5:03](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=303s) [GitHub preview showing skill-editing guardrails and instructions for opening a pull request.](https://assets.ai.engineer/sites/aiecode2025/sha256/a79d4775547cc2f866de78faa302afddaec482b70104b3a82110f20c140c2839.webp) Step 4: Edit the triage skill If (and only if) you found at least one well-supported lesson, edit .agents/skills/triage-issue/SKILL.md: • Append each lesson as a bullet under ## Learned guidelines (replace the "(None yet.)" placeholder if present). Each bullet should be a single imperative guideline, optionally with a parenthetical note of the evidence, e.g. (observed: 2 maintainer relabels). • If a lesson contradicts an existing learned guideline, revise or remove the old one rather than adding a conflicting bullet. • If a lesson warrants it, adjust the bucket definitions in the classification step directly — but keep the three buckets exactly as they are (ready-to-implement, needs-info, duplicate). • Bump the version: increment version: in the frontmatter, the "version N" text in the ## Skill version section, and the v: in the comment template. Guardrails • Never change the marker convention, the feedback footer, the three bucket names, or remove the ## Learned guidelines section. • Keep ## Learned guidelines to at most 15 bullets. When at the limit, consolidate overlapping guidelines or prune the weakest-evidenced ones. • If feedback signals are weak or conflicting, make no change. An empty run is a valid outcome — report "no changes warranted" and stop without opening a PR. Step 5: Open a pull request Never commit to the default branch. Create a branch and PR: git checkout -b skill-improvement/ git add .agents/skills/triage-issue/SKILL.md git commit -m "Improve triage-issue skill from feedback (v -> v)" git push -u origin skill-improvement/ gh pr create --title "Triage skill v: " --body-file The PR body must include: Interface text: github.com/warpdotdev-demos/issue-triage-loop/blob/main/.agents/skills/improve-triage-skill/SKILL.md Files; main; Go to file .agents/skills; improve-triage-skill; SKILL.md; triage-issue; .github; scripts; README.md issue-triage-loop / .agents / skills / improve-triage-skill / SKILL.md Preview; Code; Blame; 89 lines (61 loc) · 5.15 KB; Raw; Top ### Agent Memory After an agent session, an outer-loop agent extracts facts and updates memory so future runs can use what earlier runs learned. [5:37](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=337s) [Slide showing an agent memory cycle connecting an agent session, post-session extraction, facts, and memory updates.](https://assets.ai.engineer/sites/aiecode2025/sha256/eeea9f4273be3a890521f50f931d9bedddd4bff1e23f3bee4d2223c9d39c43b6.webp) Agent Memory Agent Session Automatic Post-Session Extraction Extract Facts Update Memories Agent Memory 7 Interface text: Figma File Edit View Object Text Arrange Plugins Widgets Window Help AI Engineer Expo Talk Growth New slide Design Animate 63% Slide 7 Template style Custom Style Matter Background Color 1 Selection colors Color 2 Color 2 Accent 1 Accent 1 Color 1 000000 100% FFFFFF 100% 085AFF 100% FFFFFF 15% Add presenter notes... Start with a draft ### Agent memory store The Sentry demo shows a memory store attached to an agent, with saved debugging lessons and controls to create, edit, delete, and view versions of memories. [6:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=402s) [A memory store page showing an agent and saved troubleshooting memories.](https://assets.ai.engineer/sites/aiecode2025/sha256/638d3ed52b868ff71d11a0192447adfb35a82a7f19ce331c43a2a7f7b3b61eda.webp) Sentry Demo Store UID [illegible] Owner Team Created Jun 1, 2026, 11:02 AM Updated Jun 1, 2026, 11:02 AM Agents 1 total Saleor Sentry Demo Use this to save known noise patterns to skip, saleor codebase architecture notes, past fix patterns that worked, previous triage decisions and their outcomes, root cause analysis, etc. Anything that will help future triage fix runs. READ WRITE Memories 5 total New memory Memory ID [illegible] View Run Updated 28 days ago Saleor payment webhook vulnerability: _validate_transaction_action_data and _validate_transaction_session_action_data both call .get() on response_data without type-checking, causing AttributeError when webhooks return JSON arrays instead of objects. Correct defensive pattern: validate that response_data is a dict before calling .get(), then raise ValueError early for type mismatches (which callers already catch) rather than letting AttributeError propagate. This prevents unexpected exception types from bubbling up. Versions Edit Delete Memory ID [illegible] View Run Updated 28 days ago Error handling principle for React render utilities: When a utility function is called during component render and may encounter malformed data, prefer graceful degradation (log the error + skip/return safe default) over throwing exceptions. This prevents production page crashes while still capturing errors via errorTracker.captureException() for Sentry observability. Apply even when the issue appears to be a schema bug. Example: Dashboard getGiftCardsAmountUsed should never throw from within a .reduce() callback consumed by a render path—log the error and skip the problematic entry instead. Versions Edit Delete Memory ID [illegible] View Run Updated 28 days ago GraphQL null response handling: When GraphQL APIs return { ok: true, data: null }, TypeScript optional chaining must be applied at the root property level. Specifically, executePublicGraphQL can return { ok: true, data: null } at HTTP 200 status; callers must guard against null Interface text: Back Sentry Demo Delete Chrome File Edit View History Bookmarks Profiles Tab Window Help Sentry Demo | Memories - Oz Runs - Oz Warp Ask Gemini Work oz.staging.warp.dev/memory/d52f8149-93ed-4810-b48d-0b23dbbb709d ### Saved troubleshooting memories The Sentry agent’s memory store records past root-cause findings and fix patterns. Each entry has controls to view its source run, inspect versions, edit or delete it. [6:59](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=419s) [A scrolled list of saved memories about webhook validation, error handling, and GraphQL null responses.](https://assets.ai.engineer/sites/aiecode2025/sha256/50c00627844236e82634ec648bd91cbc88dc98905f2ec8fed3301ea64b5d8c4b.webp) Saleor Sentry Demo Use this to save known noise patterns to skip, saleor codebase architecture notes, past fix patterns that worked, previous triage decisions and their outcomes, root cause analysis, etc. Anything that will help future triage fix runs. READ WRITE Memories 5 total New memory Memory ID [illegible] View Run Updated 28 days ago Saleor payment webhook vulnerability: _validate_transaction_action_data and _validate_transaction_session_action_data both call .get() on response_data without type-checking, causing AttributeError when webhooks return JSON arrays instead of objects. Correct defensive pattern: validate that response_data is a dict before calling .get(), then raise ValueError early for type mismatches (which callers already catch) rather than letting AttributeError propagate. This prevents unexpected exception types from bubbling up. Versions Edit Delete Memory ID [illegible] View Run Updated 28 days ago Error handling principle for React render utilities: When a utility function is called during component render and may encounter malformed data, prefer graceful degradation (log the error + skip/return safe default) over throwing exceptions. This prevents production page crashes while still capturing errors via errorTracker.captureException() for Sentry observability. Apply even when the issue appears to be a schema bug. Example: Dashboard getGiftCardsAmountUsed should never throw from within a .reduce() callback consumed by a render path—log the error and skip the problematic entry instead. Versions Edit Delete Memory ID [illegible] View Run Updated 28 days ago GraphQL null response handling: When GraphQL APIs return { ok: true, data: null }, TypeScript optional chaining must be applied at the root property level. Specifically, executePublicGraphQL can return { ok: true, data: null } at HTTP 200 status; callers must guard against null result.data with optional chaining or explicit null checks before accessing properties. Incorrect: result.data.collection?.products (crashes if result.data is null). Correct: result.data?.collection?.products (safe guard at each level). Versions Edit Delete Memory ID [illegible] View Run Updated 29 days ago Sentry error-seeding classification pattern: When seed-generated error classes (e.g., TransactionActionDataError, StorefrontDataIntegrityError) don't exist in the product codebase, this indicates the underlying code path handles that scenario gracefully via fallback. Classify such cases as noise when searching the codebase yields no results, as the absence itself proves robustness. Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Cooking · in 23m Wed Jul 1 5:06 PM issue-triage-loop/.agents/ski… issue-triage-loop/.github/wor… issue-triage-loop/.agents/ski… Sentry Demo | Memories - Oz Runs - Oz Warp Ask Gemini https://oz.staging.warp.dev/memory/d52f8149-93ed-4810-b48d-0b23dbbb709d Work https://oz.staging.warp.dev/runs/[illegible] ### Completed Sentry investigation The completed Sentry run reports fixes for three issues: webhook validation, a null GraphQL response, and missing gift card balance data. It also identifies the saved memories used to guide the fixes. [7:40](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=460s) [A completed agent run summarizes three Sentry fixes and the memories used.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3b77cb853d2fd675c03a754291b0027e1ce1fa166b4292ca511d2b308b85be0.webp) Agent finished Your agent has successfully completed your task. Investigate And Classify Sentry Issues Quick run Oz Web Investigate and fix these Sentry issues: SALEOR-DEMO-D, SALEOR-DEMO-A, and SALEOR-DEMO-7. Check memory for relevant past fixes and other information that can aid in this triage run. Any memories you use, make sure to call them out being [illegible] Show more Status Done Investigated and fixed all three Sentry issues: SALEOR-DEMO-7 (fix_now → saleor backend): AttributeError: 'list' object has no attribute 'get' — Added dict type check to _validate_transaction_action_data and _validate_transaction_session_action_data in saleor/payment/utils.py. When webhook returns a JSON array instead of object, raises ValueError early (callers already catch it). Validated with direct Python execution and ruff. New tests added. SALEOR-DEMO-A (fix_now → saleor-storefront): TypeError: Cannot read properties of null (reading 'collection') — Added optional chaining result.data?.collection in getFeaturedProducts to guard against null GraphQL response data. The page now gracefully falls back to an empty product list. SALEOR-DEMO-D (fix_now → saleor-dashboard): Error: [extractOrderGiftCardUsedAmount] Missing balance — Replaced throw with console.error() + skip in getGiftCardsAmountUsed so the OrderSummary view degrades gracefully. Updated 3 tests. All 52 tests pass. Memories used: Memory from e8b5924e (payment webhook vulnerability pattern) and df3c6489 (prior fix patterns for null collection and gift card balance). Sentry remained read-only — no writes, no events emitted, no secrets printed. Artifacts warpdotdev-demos/saleor#99 sentry/fix-transaction-action-dict-check Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help Cooking · in 22m Wed Jul 1 5:07 PM issue-triage-loop/.agents/ski… issue-triage-loop/.github/wo… issue-triage-loop/.agents/ski… Runs - Oz Runs - Oz Warp Ask Gemini oz.staging.warp.dev/runs/019e8dd3-9baf-71fa-a374-a93443ea11a5 Warp staging.warp.dev Memory usage: 465 MB Work View session ### Sentry triage summary The Sentry agent reports three PRs, identifies prior fix patterns it used, and lists two candidate memories for future triage. The full backend pytest suite was skipped because Docker was unavailable. [7:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=468s) [A completed agent session lists triage decisions, pull requests, memories used, and candidate memories.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc156c3acd990fd0a81d0f7a3d1bdcc23a20e0a54f4bce09839c317ad9f7f110.webp) All three PRs created. Here is the final triage summary: Sentry sweep: • Query: is:unresolved environment:saleor-sentry-demo • Candidate count: 3 (targeted) • Scope limits: SALEOR-DEMO-D, SALEOR-DEMO-A, SALEOR-DEMO-7 • API gaps: Docker not available in sandbox; full pytest suite skipped for backend Triage decisions: • SALEOR-DEMO-7 (id=[illegible]): fix_now — saleor — Webhook returns JSON array; .get() on list raises AttributeError — PR created • SALEOR-DEMO-A (id=[illegible]): fix_now — saleor-storefront — GraphQL returns data: null with 200; accessing .collection on null crashes page — PR created • SALEOR-DEMO-D (id=[illegible]): fix_now — saleor-dashboard — Gift card event missing balance data; explicit throw crashes OrderSummary view — PR created Delegated investigations: All three investigated inline (single-agent run). Implementation: • Mode: fix-capable • SALEOR-DEMO-7: saleor / sentry/fix-transaction-action-dict-check / PR #99 • SALEOR-DEMO-A: saleor-storefront / sentry/fix-featured-products-null-data / PR #21 • SALEOR-DEMO-D: saleor-dashboard / sentry/fix-giftcard-balance-graceful / PR #90 Sentry safety: • API writes performed: none • Sentry events emitted: none • Secrets printed: none Memories used: • e8b5924e: Confirmed _validate_transaction_action_data and _validate_transaction_session_action_data vulnerability pattern and correct defensive pattern (raise ValueError early) • df3c6489: Confirmed prior fix patterns — optional chaining for SALEOR-DEMO-A, graceful error handling for gift card balance issues Memory candidates: • Storefront executePublicGraphQL can return { ok: true, data: null } when GraphQL responds with { data: null } at 200 status — callers must guard with optional chaining on result.data • Dashboard getGiftCardsAmountUsed should never throw from within a .reduce() callback consumed by a render path — log and skip instead Interface text: Completed Output final triage summary (5/5) Investigate And Classify Sentry Issues View in Oz Status: Done Agent: Saleor Sentry Demo Harness: Warp Artifacts: sentry/fix-transaction-action-dict-... saleor #99 sentry/fix-featured-products-null-d... saleor-storefront #21 sentry/fix-giftcard-balance-graceful saleor-dashboard #90 Directory: /workspace Run ID: [illegible] Credits used: 752.6 Run time: 36.6 min Created on: 10:11 AM on 6/3/2026 Environment details: Name: Saleor Sentry Triage CI ID: [illegible] Image: warpdotdev/dev-web:latest Initial query: Investigate and fix these Sentry issues: SALEOR-DEMO-D, SALEOR-DEMO-A, and SALEOR-DEMO-7. 721.6 credits Ask a follow up staging.warp.dev/conversation/00039952-7e15-45f0-978f-e72af0d4ec7b [7:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=475s) [A completed agent run summarizes three issue fixes and lists memories used and proposed.](https://assets.ai.engineer/sites/aiecode2025/sha256/87e9e8983281997687d2235568831e8a90028f129bbc52f5d72cd1b25200df52.webp) All three PRs created. Here is the final triage summary: Sentry sweep: • Query: is:unresolved environment:saleor-sentry-demo • Candidate count: 3 (targeted) • Scope limits: SALEOR-DEMO-D, SALEOR-DEMO-A, SALEOR-DEMO-7 • API gaps: Docker not available in sandbox; full pytest suite skipped for backend Triage decisions: • SALEOR-DEMO-7 (id=[illegible]): fix_now — saleor — Webhook returns JSON array; .get() on list raises AttributeError — PR created • SALEOR-DEMO-A (id=[illegible]): fix_now — saleor-storefront — GraphQL returns data: null with 200; accessing .collection on null crashes page — PR created • SALEOR-DEMO-D (id=[illegible]): fix_now — saleor-dashboard — Gift card event missing balance data; explicit throw crashes OrderSummary view — PR created Delegated investigations: All three investigated inline (single-agent run). Implementation: • Mode: fix-capable • SALEOR-DEMO-7: saleor / sentry/fix-transaction-action-dict-check / PR #99 • SALEOR-DEMO-A: saleor-storefront / sentry/fix-featured-products-null-data / PR #21 • SALEOR-DEMO-D: saleor-dashboard / sentry/fix-giftcard-balance-graceful / PR #90 Sentry safety: • API writes performed: none • Sentry events emitted: none • Secrets printed: none Memories used: • e8b5924e: Confirmed _validate_transaction_action_data and _validate_transaction_session_action_data vulnerability pattern and correct defensive pattern (raise ValueError early) • df3c6489: Confirmed prior fix patterns — optional chaining for SALEOR-DEMO-A, graceful error handling for gift card balance issues Memory candidates: • Storefront executePublicGraphQL can return { ok: true, data: null } when GraphQL responds with { data: null } at 200 status — callers must guard with optional chaining on result.data • Dashboard getGiftCardsAmountUsed should never throw from within a .reduce() callback consumed by a render path — log and skip instead Interface text: Completed Output final triage summary (5/5) Investigate And Classify Sentry Issues View in Oz Status: Done Agent: Saleor Sentry Demo Harness: Warp Artifacts: sentry/fix-transaction-action-dict-... saleor #99 sentry/fix-featured-products-null-d... saleor-storefront #21 sentry/fix-giftcard-balance-graceful saleor-dashboard #90 Directory: /workspace Run ID: [illegible] Credits used: 752.6 Run time: 36.6 min Created on: 10:11 AM on 6/3/2026 Environment details: Name: Saleor Sentry Triage CI ID: [illegible] Image: warpdotdev/dev-web:latest Initial query: Investigate and fix these Sentry issues: SALEOR-DEMO-D, SALEOR-DEMO-A, and SALEOR-DEMO-7. 721.6 credits Ask a follow up staging.warp.dev/conversation/00039952-7e15-45f0-978f-e72af0d4ec7b ### Model routing and eval sidecar A router selects a model for each request. An eval sidecar samples and scores outcomes, then feeds policy updates back to the router. Gupta says Warp uses this internally to compare models across tasks. [8:45](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=525s) [A router sends an incoming request to a selected model, while an evaluation sidecar scores outcomes and updates the routing policy.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9337c691df6d53920801ce31dc73f272723cefabc01c408ef1b4e106d222a59.webp) Model Routing Incoming request Router Selected model Response Eval Sidecar Eval sampler Outcome scorer Policy update 8 Interface text: Figma AI Engineer Expo Talk New slide Building Self-Improving Agent Software Factories Suraj Gupta Self-improvement How do factory agents get better over time? Skills Memory Model routing Skills Agent Memory Design Animate 63% Slide 8 Template style Custom Style Matter Background Color 1 Selection colors Color 2 Color 2 Accent 1 Color 1 000000 100% FFFFFF 100% 085AFF 100% 1783FF 100% D87756 100% Add presenter notes... Start with a draft Share ### Custom model routing configuration Gupta shows a Warp configuration that routes acme-payments database migrations to GLM and API documentation to Qwen, with separate models for incident debugging and scaffolding. [9:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=597s) [A terminal displays a configuration assigning different models to different categories of work.](https://assets.ai.engineer/sites/aiecode2025/sha256/b67ab64d9baa27b1d50362ccab96dfcadc992cac287e1b6ca5a70aa8ed91ebe6.webp) ~/.warp-dev/custom_model_routers (0.062s) cat orchestrator.yaml name: Orchestrator type: prompt default: claude-4-6-sonnet-high routing: - description: writing or reviewing database migrations for the acme-payments service model: glm-5.2-fireworks - description: debugging production incidents or fixing failing CI tests model: claude-4-8-opus-high - description: scaffolding new services or boilerplate from our internal templates model: kimi-k27-code-fireworks - description: drafting release notes, runbooks, or API documentation model: qwen-3.7-plus-fireworks Interface text: WarpDev File Edit View Tab Blocks AI Drive Window Help New agent conversation Settings ESC for terminal ? for help / for commands & send task to the cloud ⌘ Y open conversation Warp anything e.g. Set up SSL certificates and configure HTTPS for my domain Default Orchestrator /remote-control Choose an agent model AI Engineer Expo Talk New slide Design Animate 63% Slide 8 Template style Custom Style Matter Background Color 1 Selection colors Color 2 Color 2 Accent 1 Color 1 000000 100% FFFFFF 100% 085AFF 100% 1783FF 100% D87756 100% Add presenter notes Start with a draft ### Selecting a model router Gupta selects a custom Orchestrator alongside Warp’s automatic model options. Its configuration maps different kinds of work to specific models, giving Warp rules for routing requests. [10:05](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=605s) [A model selection menu shows a custom orchestrator selected alongside automatic model options.](https://assets.ai.engineer/sites/aiecode2025/sha256/38dd82cb82b02286cd30d8a74a52b3acd3720b29d8e568fe65d56cec1e6a59e3.webp) ~/.warp-dev/custom_model_routers (0.062s) cat orchestrator.yaml routing: - description: writing or reviewing database migrations for the acme-payments service model: glm-5.2-fireworks - description: debugging production incidents or fixing failing CI tests model: claude-4-8-opus-high - description: scaffolding new services or boilerplate from our internal templates model: kimi-k27-code-fireworks - description: drafting release notes, runbooks, or API documentation model: qwen-3.7-plus-fireworks /MODEL Base Full Terminal Use Manage defaults auto (responsive) auto (cost-efficient) auto (genius) auto (open-weights) Pareto Orchestrator (selected) test claude 4.5 haiku claude 4.5 opus claude 4.5 opus (thinking) Model Specs Warp's benchmarks for how well a model performs in our harness, the rate at which it consumes credits, and task speed. Intelligence Speed Cost ↵ to select ⌘ ↵ select and save to profile ⇧ tab to cycle tabs esc to dismiss Search models Interface text: WarpDev File Edit View Tab Blocks AI Drive Window Help Cooking · in 20m Wed Jul 1 5:09 PM Untitled AI Engineer Expo Talk Audience · AI Engineer Expo Talk New agent conversation Settings ESC for terminal Default Orchestrator /remote-control AI Engineer Expo Talk Growth New slide Building Self-Improving Agent Software Factories Suraj Gupta Self-improvement How do factory agents get better over time? Skills Agent Memory Design Animate 63% Slide 8 Template style Custom Style Matter Background Color 1 Selection colors Color 2 Color 2 Accent 1 Color 1 000000 100% FFFFFF 100% 085AFF 100% 1783FF 100% D87756 100% Share Add presenter notes... Start with a draft ### Warp Platform Gupta invites attendees to talk at Warp’s booth, U-G20. [12:25](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=745s) [A Warp closing slide shows the platform name, website and booth information over a laptop photograph.](https://assets.ai.engineer/sites/aiecode2025/sha256/f68bcfd1b7172e23c01699f5b0dafb0d36d9eb72c96204fecf40ab50a7e78803.webp) warp Warp Platform | warp.dev Booth U-G20 Interface text: Figma File Edit View Object Text Arrange Plugins Widgets Window Help AI Engineer Expo Talk Growth New slide Suraj Gupta Self-improvement How do factory agents get better over time? Skills Agent Memory Design Animate 63% Slide 9 Template style Custom Style Matter Background Color 2 Selection colors Color 2 Color 1 FFFFFF 100% Linear 20% Add presenter notes... Start with a draft ### AI Engineer World's Fair [12:40](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=760s) [AI Engineer World's Fair logo in white on a black background, with the event website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6768ac2a52b8f15547969f194514988544be7500c9e5c18d840205558158e71.webp) AI Engineer World’s Fair HTTPS://AI.ENGINEER ## Full transcript [0:12](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=12s) Um my name is Siraj Gupta. I lead [0:14](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=14s) Harness development at Warp. Um in case [0:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=16s) you haven't heard of Warp before, um we [0:18](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=18s) started off by building a [0:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=20s) state-of-the-art agentic development [0:22](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=22s) environment. Um we have about a million [0:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=24s) active users that use that. Um it was [0:26](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=26s) born out of a modern-day terminal. [0:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=28s) Um and nowadays we're building a um [0:31](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=31s) cloud agent platform to enable teams to [0:33](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=33s) build software factories. [0:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=35s) Um today I'm going to talk about um [0:38](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=38s) self-improving software factories, which [0:40](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=40s) I know might sound kind of buzzy, um [0:43](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=43s) but I'm going to break it down and [0:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=44s) hopefully you can walk out of here with [0:45](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=45s) something concrete that you can [0:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=46s) implement yourselves. [0:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=48s) Um you've probably heard of both of [0:50](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=50s) these terms, self-improvement and [0:52](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=52s) software factories, used throughout this [0:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=53s) conference, but I'm sure you've heard of [0:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=55s) them independently. [0:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=57s) Um [0:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=57s) you've heard about how [0:59](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=59s) self-improvement is about agents being [1:01](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=61s) able to [1:02](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=62s) um [1:03](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=63s) improve over time and agents and models [1:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=66s) um [1:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=66s) and, you know, phasing out humans from [1:08](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=68s) that loop, so that way the the [1:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=70s) improvement loop can be automatic. Um [1:13](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=73s) and I'm sure you've heard about software [1:14](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=74s) factories being [1:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=76s) um the way that like the software [1:18](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=78s) development paradigm is shifting from [1:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=80s) individual coding agents to building out [1:22](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=82s) automations that take work from triage [1:25](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=85s) up until production. Um but less has [1:27](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=87s) been said about how software factories [1:30](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=90s) themselves can self-improve over time, [1:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=92s) so that the the factory can get better, [1:34](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=94s) more efficient, faster. Um and I think [1:37](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=97s) this is going to become an extremely [1:38](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=98s) important part of our jobs as software [1:40](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=100s) engineers as we transition from building [1:43](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=103s) products to building factories and and [1:45](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=105s) maintaining them. [1:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=106s) Um so let's dive in. Um [1:49](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=109s) self-improvement is a pretty broad [1:50](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=110s) umbrella, but I'm going to focus on [1:52](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=112s) three concrete ways that agents can [1:54](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=114s) improve over time, agent factories can [1:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=117s) improve over time. I'm going to start [1:58](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=118s) with um skills, then we'll talk about [2:00](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=120s) persistent memory, and then I'll finish [2:02](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=122s) off by talking about model routing. [2:04](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=124s) Um so [2:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=126s) skills are a great way to give your [2:07](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=127s) agents procedural memory. I don't think [2:09](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=129s) I have to kind of spell that out for [2:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=130s) you. You describe a a procedure to your [2:13](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=133s) agent that you expect it to have to do a [2:15](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=135s) task over and over. Um for example, [2:18](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=138s) let's say you're building a triage [2:19](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=139s) agent, which is a pretty common thing [2:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=140s) you might build in a software factory. [2:23](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=143s) Um [2:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=144s) and you give it a skill on how you want [2:26](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=146s) it to go about reproducing issues. Um [2:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=148s) those skills get stale over time. Um [2:30](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=150s) humans give feedback to the agents. [2:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=152s) Agents learn things in their own um runs [2:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=155s) and trajectories. [2:36](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=156s) Um and [2:38](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=158s) the like the skill that you previously [2:40](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=160s) gave it becomes dated. So, how do we fix [2:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=162s) this and let the agent improve over time [2:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=164s) using that skill? [2:45](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=165s) Well, one way to do it is to deploy an [2:47](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=167s) outer loop agent. Um and this is what [2:49](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=169s) we've done in our own internal software [2:51](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=171s) factory that's been really productive. [2:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=173s) The idea is that your um [2:56](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=176s) your inner loop agent is the thing [2:58](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=178s) that's actually applying the skill. It's [3:00](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=180s) like doing the triage. And then you have [3:02](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=182s) this other outer loop agent that is [3:03](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=183s) observing your inner loop agent's runs [3:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=186s) and improving its skill over time by [3:08](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=188s) looking for [3:09](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=189s) um [3:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=190s) for mistakes it made or looking at [3:12](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=192s) feedback that the human has given it. [3:14](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=194s) Um so, I'm going to show an example of [3:15](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=195s) that. [3:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=196s) Um let me swap here. Cool. Um so, this [3:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=200s) is actually a skill that we've used [3:22](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=202s) internally at Warp. We've open-sourced [3:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=204s) our client repository and it's fully run [3:26](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=206s) as as a factory. And so, this is an [3:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=208s) example of an of a [3:31](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=211s) of a skill that's that's backing a [3:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=212s) triage agent that we have today. [3:34](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=214s) The idea is that um [3:37](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=217s) you you get an issue that comes in on [3:39](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=219s) GitHub, and you want to automatically [3:41](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=221s) run an agent to figure out like what [3:43](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=223s) sorts of um [3:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=224s) like it what sort of information did the [3:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=226s) user not provide that is going to help [3:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=228s) us figure out if this is something that [3:49](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=229s) we should build, an existing issue that [3:51](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=231s) we might actually just want to dedupe [3:52](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=232s) against or something else. [3:54](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=234s) Um [3:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=235s) and then we have a Sorry, let me just [3:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=237s) switch over here. [3:59](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=239s) And then we have a workflow that [4:01](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=241s) actually like runs that that triage um [4:05](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=245s) agent. [4:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=246s) Um this is a pretty simple workflow. It [4:07](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=247s) just Every time an an an issue gets [4:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=250s) submitted on the repo, this [4:11](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=251s) um this workflow runs. And then here's [4:14](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=254s) the actual outer loop agent, which is [4:15](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=255s) about looking at the the work that the [4:18](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=258s) triage agent has done and improving it [4:19](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=259s) over time. So, the idea is that the [4:22](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=262s) agent looks at the decisions that the [4:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=264s) triage the inner loop triage agent made, [4:26](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=266s) and it um collects feedback signals, [4:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=268s) which might be uh you know, thumbs up or [4:30](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=270s) thumbs down on the issues. Um it might [4:33](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=273s) be users commenting on the issue. It [4:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=275s) might be someone from Warp um an [4:37](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=277s) employee of Warp commenting on the [4:38](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=278s) issue, uh giving the agent feedback on [4:40](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=280s) what it did. [4:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=282s) Um and then ultimately that leads to [4:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=284s) like some synthesis of of the of that um [4:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=288s) iteration. And so, we we end up [4:50](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=290s) basically updating our inner loop skill [4:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=293s) with this outer loop agents. Um and [4:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=295s) that's what leads to this [4:56](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=296s) self-improvement loop with skills. [4:58](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=298s) What's really nice about this is that um [5:00](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=300s) the [5:01](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=301s) So, in step four, you'll see we we added [5:03](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=303s) the triage skill, and then we open a [5:04](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=304s) pull request. So, that means that um all [5:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=306s) of the improvements to the inner loop [5:08](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=308s) skill are going to be tracked through [5:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=310s) get. So, you get like full observability [5:12](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=312s) into like how that skill is transformed [5:14](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=314s) over time. And that it also means that [5:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=316s) um [5:17](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=317s) a human is actually going to review [5:18](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=318s) those updates to the skill. So, that [5:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=320s) way, you know, this this outer loop [5:21](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=321s) agent doesn't make a mistake and [5:23](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=323s) ultimately cause your triage agent to to [5:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=324s) actually perform worse. [5:26](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=326s) Um [5:27](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=327s) yeah, so that's that's kind of what what [5:29](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=329s) I want to talk about for skills. Um [5:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=332s) let me talk about persistent memory now. [5:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=335s) So, I'm going to go back to my deck. [5:37](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=337s) Um [5:39](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=339s) cool. So, skills are great for [5:41](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=341s) remembering procedures, but what about [5:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=342s) everything else? So, like what what [5:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=344s) happens when your Sentry agent [5:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=346s) um looks for [5:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=348s) an issue and [5:50](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=350s) and finds an issue, does some work to [5:52](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=352s) like gather context about it, and fixes [5:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=355s) it, but then now you have you encounter [5:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=357s) a similar issue in the future. [6:00](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=360s) Your century your agent might get lucky [6:02](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=362s) and like be able to figure out the same [6:05](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=365s) root cause that it did the first time [6:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=366s) around, but that's not guaranteed. And [6:08](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=368s) and even if it does, you're you've [6:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=370s) probably wasted a lot of tokens [6:11](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=371s) re-gathering context on an issue that [6:13](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=373s) you've already fixed before. So, that's [6:15](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=375s) where [6:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=376s) something like persistent memory comes [6:17](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=377s) in. [6:18](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=378s) You can think of this as a fact store [6:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=380s) scoped to an agent. Your agent runs and [6:23](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=383s) similar to the skill improvement loop, [6:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=384s) you have an outer loop agent that's [6:26](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=386s) going to extract facts, learnings, [6:27](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=387s) outcomes [6:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=388s) from your from your inner loop agent. [6:30](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=390s) So, that way future runs of the agent [6:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=392s) can lean on what's already been done in [6:33](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=393s) the past. [6:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=395s) So, I'm going to show an example of [6:36](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=396s) this. Come back to the deck here. [6:39](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=399s) So, in warp or in Oz rather, like I [6:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=402s) mentioned, we built out a a cloud agent [6:43](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=403s) platform where you can run agents in the [6:45](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=405s) cloud. And [6:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=408s) for all for all these agents, you can [6:49](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=409s) attach memory stores to them. Again, a [6:51](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=411s) memory a memory store is a collection of [6:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=413s) facts. In this case, I have a century [6:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=415s) agent that [6:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=417s) has collected some memories over time [7:00](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=420s) about root cause analysis that it's done [7:02](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=422s) in the past. What's cool is like you can [7:04](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=424s) actually version this memory, you can [7:05](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=425s) update it as a human, create new [7:07](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=427s) memories or delete them yourselves, [7:09](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=429s) but primarily this this memory creation [7:11](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=431s) loop is driven by agents. You can also [7:14](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=434s) see where these memories were sourced [7:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=436s) from. So, [7:17](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=437s) I can actually like open up the run here [7:19](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=439s) to see [7:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=440s) Oh, well. [7:22](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=442s) I think this my IP address here keeps [7:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=444s) changing and this my our staging [7:25](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=445s) platform is gated to an IP address, but [7:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=448s) anyways, [7:29](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=449s) you should like [7:30](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=450s) you can see where those memories were [7:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=452s) sourced from. So, that way you can be [7:34](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=454s) like, oh, like maybe that wasn't [7:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=455s) actually an important memory or that was [7:36](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=456s) kind of like a local maxima that I don't [7:38](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=458s) want to include in my memory store. Um [7:41](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=461s) and then when your agents continue to [7:43](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=463s) run, so in this case I have the sentry [7:45](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=465s) agent that's [7:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=466s) um that's done a bunch of runs after [7:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=468s) it's created those memories, it actually [7:49](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=469s) can lean on those memories in the future [7:51](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=471s) um when it's making um [7:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=473s) when it's triaging new issues. So in [7:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=475s) this case, you'll notice that um [7:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=477s) this agent ran, it did a bunch of work [7:59](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=479s) to do some root cause analysis, and then [8:01](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=481s) it ultimately reported back on which [8:02](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=482s) memories it used [8:04](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=484s) um [8:05](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=485s) uh in this case it it it it was able to [8:07](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=487s) use those um five memories that I had in [8:09](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=489s) that memory sort to to speed up the work [8:11](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=491s) that I did this time around. [8:13](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=493s) Um, [8:15](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=495s) cool. Uh [8:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=496s) let me go back to the deck. [8:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=500s) So, [8:21](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=501s) what's cool in Oz is that persistent [8:23](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=503s) memory works across all harnesses, so [8:25](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=505s) Warp Zone, proprietary harness, plot [8:27](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=507s) code, Kodak, it doesn't really matter [8:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=508s) what you're running on Oz, um memories [8:30](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=510s) will automatically be created for you. [8:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=512s) Um, but you also have the control to to [8:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=515s) review those memories, version them, [8:37](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=517s) delete them, or edit them as you please. [8:39](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=519s) Um, all that being fully traceable. [8:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=522s) Um, cool. That's what I wanted to say [8:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=524s) about memory. Um, [8:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=526s) last thing I want to talk about here is [8:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=528s) model routing. So, um you may be [8:50](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=530s) wondering how this plays into [8:51](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=531s) self-improvement loops for um factories. [8:54](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=534s) Well, first I want to touch on why model [8:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=535s) routing is important in a factory in the [8:56](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=536s) first place. [8:57](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=537s) Um, it can become prohibitively [8:59](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=539s) expensive to run all of your agents that [9:01](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=541s) are doing simple things like triage or [9:03](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=543s) fixing simple CI failures with Opus. [9:05](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=545s) That's just like going to be really [9:07](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=547s) expensive over time, and I'm sure some [9:08](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=548s) of you have maybe been burnt by this or [9:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=550s) have heard stories of of or [9:12](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=552s) organizations being burnt by this. Um, [9:14](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=554s) so model like choosing which models you [9:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=556s) use for which tasks that is actually [9:18](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=558s) very important um and something that [9:19](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=559s) we're we've been working on at work. [9:21](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=561s) Um, so in Warp, you can actually use uh [9:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=564s) our out-of-the-box model routers, we [9:25](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=565s) call them our auto models. Um, over like [9:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=568s) as new models come out, we're all we're [9:29](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=569s) always evaluating like what are the best [9:31](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=571s) models when it comes to Pareto [9:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=572s) efficiency. And so, like if you're a new [9:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=575s) user and you just want to get started, [9:36](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=576s) it's much better than like pinning your [9:38](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=578s) model to like Opus or or or Haiku, for [9:41](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=581s) example. Um [9:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=584s) let me show you all um that experience [9:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=586s) in Warp. [9:47](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=587s) So, let me go here. [9:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=593s) Give me 1 sec. [9:54](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=594s) Got to drag this over here. [9:56](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=596s) So, here I have my Warp desktop um and [9:59](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=599s) you can see that um alongside being able [10:01](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=601s) to define models um [10:04](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=604s) being able to use Warp's auto models is [10:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=606s) one option, like these models here, but [10:08](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=608s) you can also define your own model [10:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=610s) routed um [10:12](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=612s) uh models, which uh basically boil down [10:15](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=615s) to like a set of rules that you can give [10:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=616s) um to us and we will like make decisions [10:19](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=619s) about model routing based on those [10:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=620s) rules. So, here I have an example of a [10:23](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=623s) configuration file that I've defined um [10:25](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=625s) where I've defined a few different rules [10:26](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=626s) about like I want database migrations [10:29](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=629s) done with with GLM fire with GLM or I [10:31](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=631s) want um you know, uh runbooks and API [10:34](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=634s) documentation done with Quen. Um [10:38](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=638s) So, this really allows you to like like [10:39](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=639s) to define classes of tasks and be able [10:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=642s) to say, "Okay, I want this this class of [10:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=644s) task done by this model because that's [10:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=646s) kind of what you've seen." [10:47](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=647s) At this point though, that's kind of [10:49](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=649s) more of an art than a science um and so [10:52](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=652s) uh what we're working on next is being [10:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=653s) able to define evals for um [10:56](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=656s) customer-facing evals where you can [10:58](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=658s) actually define what sorts of things you [11:00](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=660s) care about and what knobs you want to [11:02](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=662s) turn on these configurations and [11:04](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=664s) actually see, "Okay, is is Haiku better [11:07](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=667s) than fi- than GLM than Opus in a [11:10](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=670s) specific class of issues?" That's like [11:11](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=671s) specific to your workflows and not just [11:13](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=673s) some like generic benchmark that um that [11:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=676s) someone posted online. [11:19](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=679s) Um let me go back here. Give me a sec. [11:24](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=684s) Cool. [11:30](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=690s) Cool. Um [11:33](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=693s) and yeah, so let me uh I'm in my [11:35](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=695s) slideshow. [11:42](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=702s) So, here is kind of like a a high-level [11:44](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=704s) overview of how this works in Warp [11:46](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=706s) today. So, um internally we have our [11:48](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=708s) model routing rules, um but then we have [11:51](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=711s) like an eval sidecar that's determining [11:53](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=713s) um what sorts of tasks are actually [11:55](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=715s) better with which models. And we do this [11:56](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=716s) with a simple like best-at-k um approach [11:59](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=719s) where when you type a prompt, we run a [12:01](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=721s) bunch of agents in Oz uh across a [12:03](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=723s) different set of models. Um and we found [12:05](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=725s) success in like being able to determine, [12:06](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=726s) "Oh, like UI tasks are really well done [12:09](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=729s) with GLM. Um we don't really need to run [12:11](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=731s) those with Opus. It's much more [12:12](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=732s) efficient to run those with GLM." [12:14](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=734s) Um so, that's how we've been using it, [12:15](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=735s) but um sooner enough we're going to be [12:16](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=736s) building this into our product so that [12:18](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=738s) way our our customers can actually use [12:20](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=740s) it in their own workflows. Um [12:23](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=743s) Yeah, that's uh that's mostly what I had [12:25](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=745s) today. Um [12:26](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=746s) we're around in a booth around that [12:28](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=748s) side. So, if you're interested in this [12:29](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=749s) kind of stuff, come talk to me. I'm [12:31](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=751s) always down to talk about this kind of [12:32](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=752s) stuff. But yeah, thanks for your time [12:34](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=754s) today. That's all I had. [12:49](https://www.youtube.com/watch?v=TN3mj92oZ8I&t=769s) >> [music]