# How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp

Zach Lloyd, Warp

[Talk page](https://ai.engineer/talks/nKjPBAw_N_0-build-software-factory-that-improves-itself) · [Video](https://www.youtube.com/watch?v=nKjPBAw_N_0)

## How Warp Measures Whether Its Coding Agents Improve

Zach Lloyd opens Warp’s own engineering dashboard to show how a team can turn repeated mistakes into a pull request, then replay its past tasks to decide which model earns its cost.

![A circular workflow connects bug reports and new ideas with triage, specification, implementation, verification and release. People review the specification, code and product.](https://ai.engineer/api/talk-essay-media/18efecdf8df1d126ab9a5536e69a4c767db8b5ebc9d466c37cdd3260f3b94b7a)

A product idea or bug report moves through a loop with explicit points for people to review the specification, code and result.

### What disappears when the laptop closes?

[1:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=67s)

An engineer opens several coding agents, assigns each one a task and switches between them when they ask for input. It can be a productive afternoon. At the end of it, a team lead may still struggle to answer three questions: Which changes reached users? What did they cost? And what did one agent learn that the next engineer can reuse?

Zach Lloyd, Warp’s founder and a former engineering lead on Google Docs, describes that arrangement as interactive prompting. Each laptop has its own tools, permissions and configuration. Someone has to start each job and keep it moving. A local agent may also inherit access its particular assignment never required. Issuing more subscriptions gives the company more of these sessions, with little insight into how the work adds up.

In the alternative Lloyd calls a software factory, a person submits a feature or reports a bug. The work can pass through triage, a specification, implementation, review and verification. Some steps need a person. Others suit an agent running in an isolated cloud environment. A reviewer can see what went into a step and what came out after the person who started it has gone home.

### Put the workflow in a repository

[4:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=241s)

Lloyd wants the engineering team to own the definition of this process. Its repository records which agent handles a step, which tools it may use, the instructions it gets, the model it calls and where it runs. The same repository can hold benchmarks and the events that start a job. Someone reviewing a change can see precisely which part of tomorrow’s workflow will differ from today’s.

There is a business reason to spell these things out. A team might want to keep its data on its own infrastructure, pay for inference through an existing account or replace a model when another proves better for its work. Those decisions should be visible to the people responsible for the result. A stable record also gives an engineer two versions of a workflow they can test against each other.

Warp gives its product engineers familiar entry points. A task can arrive through Slack, a Linear ticket or a GitHub review comment. In the demo, an orchestrator selects the next step. A dashboard brings the runs and their artifacts together. Lloyd distinguishes the engineer shipping a customer feature from the engineer who improves the prompts, skills and routing that many such features will use. On a small team, those can be two jobs for the same person.

### When the same mistake shows up 37 times

[12:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=721s)

The dashboard starts with ordinary operating questions: cycle time, spend, pull requests and the number of times a person had to intervene. The interesting part lies behind the charts. A team can inspect what an agent received, the conversation as it worked and the change it left behind. That record gives a reviewer much more to go on than a green check beside a pull request.

Warp runs model-based evaluators on these traces. One looks for redundant tests. A task can produce passing code and an unnecessary test, so a passing test suite alone would miss the behavior that reviewer cares about. Storing the evaluator’s judgment with the run lets an engineer inspect the cases where it claims to have found the problem.

In a separate example, the improvement system found 37 instances of agents beginning implementation without reading the relevant skills already in the repository. Its suggested fix was a pull request against the factory definition. Lloyd opened the proposal onstage. He showed how a recurring mistake can lead to a specific edit someone can review. Whether that edit reduces the failure rate is a question for the runs that follow.

![One expanded improvement proposal recommends looking up relevant repository skills before an implementation starts.](https://ai.engineer/api/talk-essay-media/fab28ffd12a7397bf02c1b7cc8fe44195806a8d98d5e36729944dd77befd0606)

Warp’s improvement view links the proposal to read repository skills with 37 prior failures.

### A benchmark drawn from yesterday’s tickets

[14:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=888s)

How would the team know whether to buy a more expensive model for front-end work? A public leaderboard says little about its own repository or conventions. Lloyd wants to take representative work the team has already completed, preserve the starting conditions and replay it against competing configurations.

The live demo supplied a mundane but important detail. Lloyd tried to add a past run to a benchmark and could not do it on the example he had selected. He switched to an existing suite to show the result. In his spoken comparison, one configuration scored about 93 percent for roughly two dollars. Another reached about 95 percent and cost around three times as much. A team can decide what those last points are worth once it knows which tasks the cheaper choice misses.

A question from the audience established what “correct” meant on that chart. The scores came from an LLM judge. Human grading was something Lloyd recommended, and Warp had yet to add it to that workflow. His advice for building the first suite was equally practical: pick a few easy, medium and difficult tasks by hand. With a baseline in place, use the archive of previous sessions to look for cases it leaves out.

![The tooltip on the cost–correctness graph reports the average cost and correctness for the high configuration.](https://ai.engineer/api/talk-essay-media/2c2c63edc15bd67ebcb87f251ad1706702b4265c0b1668985dd5f4c32490b658)

The tooltip on Warp’s internal benchmark shows 94.8% correctness at $6.23 average cost for one configuration. An LLM judge supplied the scores.

### What did the user actually get?

[18:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1082s)

More agent throughput can leave humans staring at a longer queue of pull requests. Asked about verification, Lloyd suggests writing a specification the result can be checked against and asking an agent to produce evidence from the user’s point of view. For an interface change, a reviewer who can see the changed interaction has a more concrete starting point than a summary saying the implementation is complete. The talk describes that practice; the demonstration focused on scoring and benchmarking.

Verification costs money too. Lloyd’s suggested unit is the cost per pull request, with scoring and benchmark expenditure broken out. That lets a team estimate a project from work it recognizes and decide where another evaluation is worth paying for. It also reveals when a seemingly cheap model needs expensive supervision.

Warp’s terminal is open source. Lloyd clarified in the Q&A that the factory product he demonstrated is proprietary and designed to accommodate a team’s own tools. An individual with one interactive agent may have little need for it. The problem becomes pressing when one engineer’s change to a shared workflow can improve the next hundred tasks, and the team needs evidence to decide whether to keep it.

## Resources

- [Warp documentation](https://docs.warp.dev/) — Warp documents its terminal, cloud automation and software-factory workflows.

## Slides

### The software factory stack

[0:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=11s) [Warp’s software factory stack appears on Zach Lloyd’s title slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8cc4a5b04a869bbf964d134ec9e0c2cc09f2022e52a4bf693bedb4ed27549eb.webp)

The Software Factory Stack. Zach Lloyd | Warp

### Zach Lloyd’s background

[0:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=22s) [Lloyd introduces his work at Google and Warp beside his portrait.](https://assets.ai.engineer/sites/aiecode2025/sha256/07764508ece8e1c2f057b9d94c74dca94b2852e495bca1e8570c14832c8ad01f.webp)

Zach Lloyd. Principal engineer at Google, lead on GDocs. 20 years of engineering. Founder of Warp.

### The evolution of agentic development

[1:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=70s) [Chat and autocomplete precede interactive prompting and automated development on a rising curve.](https://assets.ai.engineer/sites/aiecode2025/sha256/97c83f6410b1c4072fb58ad608a877d032a49e649b818110a9a63de0810e500e.webp)

The evolution of agentic development. Chat. Autocomplete. Interactive Prompting. Automated development.

### The limits of individual agent sessions

[1:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=107s) [Three limitations of ad hoc AI use sit beside coverage of corporate AI spending and governance.](https://assets.ai.engineer/sites/aiecode2025/sha256/723806d401fb5d7ec967188fda0ed45b254b5485f9af639fd8f744f3c2545b32.webp)

More AI usage != more business value. Ad hoc interactive usage, no automation. No systematic way to optimize cost and quality. No central layer for governance and control.

### Governing local agents

[2:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=139s) [Lloyd presents the governance problem for agents running on individual machines.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d3643f39525e449ccb4d3ec6e2f294101b54340c16215761131fd7519c806a6.webp)

Local agents are ungovernable agents.

### Isolate agent work

[2:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=172s) [A single instruction sums up Lloyd’s proposed change to the execution environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/85a97f6f72cbb2a536811d4e1fca0a70641417d66facf8472327666273a4fd30.webp)

Sandbox your agents.

### The cloud software factory loop

[2:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=178s) [A circular workflow connects bug reports and new ideas with triage, specification, implementation, verification and release. People review the specification, code and product.](https://assets.ai.engineer/sites/aiecode2025/sha256/18efecdf8df1d126ab9a5536e69a4c767db8b5ebc9d466c37cdd3260f3b94b7a.webp)

A cloud software factory. New product idea or bug reported. Triage. Spec. Reviewing the spec. Implement. Reviewing the code. Verify. Reviewing the product. Ship. Monitor.

### A factory defined in code

[4:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=240s) [Lloyd lists three desirable properties of the underlying infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/428386da948f0cc2903e2eee12739eb1e2775177166bbdd939290d3a83aa1e00.webp)

Software factories should be built on an infrastructure stack that is open, adaptable, and defined in code.

### Control over AI infrastructure

[5:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=312s) [Lloyd introduces the ownership requirements that the following slide breaks down.](https://assets.ai.engineer/sites/aiecode2025/sha256/0742256ebe91a394c4f2b60da91a578d2ed9cfb0a2a882b59184657fa5db977b.webp)

AI Sovereignty matters.

### Control over data, compute and inference

[5:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=344s) [The completed slide lists the infrastructure and agent choices an organization can own.](https://assets.ai.engineer/sites/aiecode2025/sha256/191bbf51ca0c56c6a1c70823f083e7426b5a92e900201b7206d5584d39bb677d.webp)

Your data. Your compute. Your inference gateway. Any model, any harness.

### Three ways to measure the system

[5:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=359s) [Scoring, self-improvement and benchmarking are listed as distinct measurement activities.](https://assets.ai.engineer/sites/aiecode2025/sha256/6118696a075350c02076293864f2c5fb8caeb0b26aa7fdcc91befed68b2ae924.webp)

Measurement > Vibes. Scoring vs. Self-improvement vs. Benchmarking.

### Product and factory engineers

[6:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=413s) [The diagram branches from a shared factory to the people shipping product changes and those improving the workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/632f4b6667d8bc49df9a1f9d28196a45677046534aeee980e958ccbf695393df.webp)

Two types of engineers will use your factory: Product Engineer. Factory Engineer.

### Measuring productivity across runs

[8:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=503s) [The dashboard plots human interactions, cycle time and pull requests alongside the total number of runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/7925091ed357b382534c9c6ff8afe5246e8dc7e9969db5f95a97f128b82df3b7.webp)

Productivity. Spending. Scorers. Human interactions per PR. Median cycle time. Pull requests. Total runs.

### Tracking cost per pull request

[8:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=531s) [The spending view charts cost per PR and shows how spending varies by source.](https://assets.ai.engineer/sites/aiecode2025/sha256/848b476607cf6c7de69a851fb7ac24fc7f77184139317f28c07d118259604dd5.webp)

Spending. Cost per PR. $17.63. Cost breakdown. Spend by source. $5,251.16.

### The agents behind the workflow

[10:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=632s) [An orchestration view places the foreman over agents responsible for specification, triage, implementation and code review.](https://assets.ai.engineer/sites/aiecode2025/sha256/4a5be47f3abe9149012186e93426bc5fd2c467fd479814bcf0da3dc773475a93.webp)

wilson Foreman. Orchestrates the factory workflow and dispatches each gated step. code-review. implementation. spec. triage.

### Scoring redundant tests

[12:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=766s) [A grader’s detail view describes how it judges tests delivered in the implementation and charts its historical pass rate.](https://assets.ai.engineer/sites/aiecode2025/sha256/306d12c2f0fa90461e6e257de6b490beb5b317784a58d61365d88addbcc2052c.webp)

Redundant Tests. Whether the implementation agent’s final delivered test diff includes redundant, low-signal, or trivial tests. All-time scorer cost. Pass rate. Daily cost.

### Inspecting a grader’s individual decisions

[12:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=779s) [A table shows passes, failures and insufficient evidence for the redundant-tests evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/85c38f5829668e1ec61a6c58c8a192c111b9d05e3cac29461137e53fcf23dece.webp)

Redundant Tests. Pass. Fail. No surplus tests. Some surplus tests. Insufficient evidence.

### A recurring failure becomes a proposed change

[13:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=839s) [One expanded improvement proposal recommends looking up relevant repository skills before an implementation starts.](https://assets.ai.engineer/sites/aiecode2025/sha256/fab28ffd12a7397bf02c1b7cc8fe44195806a8d98d5e36729944dd77befd0606.webp)

Read matching repository skills before implementing. PROBLEM. SOLUTION. 37 failed runs. Self-improvement run.

### Comparing correctness and cost on internal tasks

[16:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=969s) [A scatter plot compares five model configurations on WarpBench and names a lowest-cost option.](https://assets.ai.engineer/sites/aiecode2025/sha256/d07d8011b0bec0d3943db8aff3012a84539f564fafb3c30e441568ee0f4ddb30.webp)

WarpBench 0.1. Correctness vs. Average cost. Most efficient: glm 5.3 flash. Overall 0.93. Lowest cost option: glm 5.3 flash. $1.99/trial.

### The more expensive configuration

[16:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=978s) [The tooltip on the cost–correctness graph reports the average cost and correctness for the high configuration.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c2c63edc15bd67ebcb87f251ad1706702b4265c0b1668985dd5f4c32490b658.webp)

Correctness vs Average cost. gpt-5.6-sol (high). Average cost: $6.23. Correctness: 94.8%.

### The factories stack

[17:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1035s) [The closing diagram builds from factories-as-code and data up through compute, inference, improvement, orchestration and access.](https://assets.ai.engineer/sites/aiecode2025/sha256/863c85335e006b32a3d99461acc59be4ae83dcab6715162aac70c5eaab26781b.webp)

The Factories Stack. 00 Factories-as-code. 01 Data & Context. 02 Compute. 03 Inference. 04 Improvement. 05 Orchestration. 06 Access.

## Full transcript

[0:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=12s) So, I'm going to be talking today about

[0:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=14s) So, I'm going to be talking today about

[0:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=14s) So, I'm going to be talking today about software factories and uh specifically

[0:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=18s) software factories and uh specifically

[0:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=18s) software factories and uh specifically like an infrastructure approach to

[0:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=20s) like an infrastructure approach to

[0:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=20s) like an infrastructure approach to software factories. Um, a little bit

[0:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=23s) software factories. Um, a little bit

[0:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=23s) software factories. Um, a little bit about me though before we start. So, I'm

[0:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=25s) about me though before we start. So, I'm

[0:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=25s) about me though before we start. So, I'm Zach. Uh, I've been an engineer for

[0:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=28s) Zach. Uh, I've been an engineer for

[0:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=28s) Zach. Uh, I've been an engineer for really for a long time at this point. I

[0:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=30s) really for a long time at this point. I

[0:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=30s) really for a long time at this point. I was a principal engineer at Google. I

[0:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=31s) was a principal engineer at Google. I

[0:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=31s) was a principal engineer at Google. I used to lead engineering on the Google

[0:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=32s) used to lead engineering on the Google

[0:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=32s) used to lead engineering on the Google Doc Suite. Helped scale that up to

[0:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=35s) Doc Suite. Helped scale that up to

[0:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=35s) Doc Suite. Helped scale that up to hundreds of millions and I think maybe

[0:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=37s) hundreds of millions and I think maybe

[0:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=37s) hundreds of millions and I think maybe now over a billion people who use it.

[0:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=39s) now over a billion people who use it.

[0:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=39s) now over a billion people who use it. I'm currently the founder and CEO at

[0:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=41s) I'm currently the founder and CEO at

[0:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=41s) I'm currently the founder and CEO at Warp. Uh, Warp is a company in the

[0:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=44s) Warp. Uh, Warp is a company in the

[0:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=44s) Warp. Uh, Warp is a company in the agentic development space. Our current

[0:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=46s) agentic development space. Our current

[0:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=46s) agentic development space. Our current focus is on helping companies build

[0:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=49s) focus is on helping companies build

[0:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=49s) focus is on helping companies build their own internal software factories.

[0:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=51s) their own internal software factories.

[0:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=51s) their own internal software factories. So, we have a platform called Warp

[0:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=53s) So, we have a platform called Warp

[0:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=53s) So, we have a platform called Warp Factories which we do that with. We're

[0:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=55s) Factories which we do that with. We're

[0:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=55s) Factories which we do that with. We're also uh pretty well known for Warp

[0:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=57s) also uh pretty well known for Warp

[0:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=57s) also uh pretty well known for Warp Terminal, which is an Agentic terminal.

[0:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=59s) Terminal, which is an Agentic terminal.

[0:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=59s) Terminal, which is an Agentic terminal. It's open source, around 65,000 GitHub

[1:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=61s) It's open source, around 65,000 GitHub

[1:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=62s) It's open source, around 65,000 GitHub stars, almost a million active

[1:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=63s) stars, almost a million active

[1:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=63s) stars, almost a million active developers. Um, our focus as a company

[1:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=66s) developers. Um, our focus as a company

[1:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=66s) developers. Um, our focus as a company has always been helping developers ship

[1:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=69s) has always been helping developers ship

[1:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=69s) has always been helping developers ship better software more quickly.

[1:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=72s) better software more quickly.

[1:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=72s) better software more quickly. Just to set some context on where we're

[1:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=75s) Just to set some context on where we're

[1:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=75s) Just to set some context on where we're at uh in terms of agentic development.

[1:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=78s) at uh in terms of agentic development.

[1:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=78s) at uh in terms of agentic development. So most companies that I speak with uh

[1:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=82s) So most companies that I speak with uh

[1:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=82s) So most companies that I speak with uh are at this like phase of what I think

[1:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=84s) are at this like phase of what I think

[1:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=84s) are at this like phase of what I think of as interactive prompting. And so

[1:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=86s) of as interactive prompting. And so

[1:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=86s) of as interactive prompting. And so they've deployed tools like cloud code

[1:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=88s) they've deployed tools like cloud code

[1:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=88s) they've deployed tools like cloud code or codeex to their engineers. Uh

[1:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=90s) or codeex to their engineers. Uh

[1:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=90s) or codeex to their engineers. Uh engineers are working on their laptops.

[1:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=92s) engineers are working on their laptops.

[1:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=92s) engineers are working on their laptops. They are they might have a bunch of

[1:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=93s) They are they might have a bunch of

[1:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=93s) They are they might have a bunch of these going but generally they're like

[1:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=95s) these going but generally they're like

[1:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=95s) these going but generally they're like interactively prompting agents to build

[1:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=97s) interactively prompting agents to build

[1:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=97s) interactively prompting agents to build features and fix bugs. And where uh most

[1:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=102s) features and fix bugs. And where uh most

[1:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=102s) features and fix bugs. And where uh most companies I speak with want to get to is

[1:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=104s) companies I speak with want to get to is

[1:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=104s) companies I speak with want to get to is something that is more like automated

[1:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=106s) something that is more like automated

[1:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=106s) something that is more like automated development. So

[1:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=108s) development. So

[1:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=108s) development. So the reason for that is really there's

[1:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=112s) the reason for that is really there's

[1:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=112s) the reason for that is really there's this um you know they want to get to a

[1:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=114s) this um you know they want to get to a

[1:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=114s) this um you know they want to get to a world where they are deploying this

[1:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=116s) world where they are deploying this

[1:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=116s) world where they are deploying this transformational technology which is

[1:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=118s) transformational technology which is

[1:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=118s) transformational technology which is agents and LLMs to get the most possible

[2:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=121s) agents and LLMs to get the most possible

[2:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=121s) agents and LLMs to get the most possible business value at the lowest possible

[2:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=124s) business value at the lowest possible

[2:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=124s) business value at the lowest possible cost in a way that is governed and

[2:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=127s) cost in a way that is governed and

[2:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=127s) cost in a way that is governed and secure and they're running up against

[2:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=129s) secure and they're running up against

[2:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=129s) secure and they're running up against the limits of interactive agents which

[2:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=131s) the limits of interactive agents which

[2:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=131s) the limits of interactive agents which are like you you're bottlenecked by a

[2:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=134s) are like you you're bottlenecked by a

[2:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=134s) are like you you're bottlenecked by a person running them. Um it's hard to

[2:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=137s) person running them. Um it's hard to

[2:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=137s) person running them. Um it's hard to control costs if everyone has their own

[2:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=138s) control costs if everyone has their own

[2:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=138s) control costs if everyone has their own setup and it's hard to govern. And so

[2:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=141s) setup and it's hard to govern. And so

[2:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=141s) setup and it's hard to govern. And so just to double click on that I think for

[2:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=143s) just to double click on that I think for

[2:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=143s) just to double click on that I think for this audience like local agents have

[2:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=145s) this audience like local agents have

[2:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=145s) this audience like local agents have this security problem where uh if there

[2:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=149s) this security problem where uh if there

[2:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=149s) this security problem where uh if there is a problem or compromise of a local

[2:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=151s) is a problem or compromise of a local

[2:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=151s) is a problem or compromise of a local agent then it's basically has the

[2:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=154s) agent then it's basically has the

[2:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=154s) agent then it's basically has the ability to take over whatever the user

[2:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=156s) ability to take over whatever the user

[2:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=156s) ability to take over whatever the user who signed in on their laptop is. And so

[2:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=158s) who signed in on their laptop is. And so

[2:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=158s) who signed in on their laptop is. And so that's not a good thing. Um local agents

[2:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=161s) that's not a good thing. Um local agents

[2:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=161s) that's not a good thing. Um local agents are also hard to govern. And so right

[2:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=162s) are also hard to govern. And so right

[2:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=162s) are also hard to govern. And so right now we're in this situation where you

[2:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=165s) now we're in this situation where you

[2:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=165s) now we're in this situation where you know everyone is kind of configuring

[2:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=166s) know everyone is kind of configuring

[2:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=166s) know everyone is kind of configuring their own MCPs, their own skills, their

[2:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=168s) their own MCPs, their own skills, their

[2:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=168s) their own MCPs, their own skills, their own tools. And it's like you can't see

[2:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=170s) own tools. And it's like you can't see

[2:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=170s) own tools. And it's like you can't see what's going on. You can't audit it. And

[2:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=172s) what's going on. You can't audit it. And

[2:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=172s) what's going on. You can't audit it. And so the solution uh to both these

[2:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=175s) so the solution uh to both these

[2:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=175s) so the solution uh to both these problems is to sandbox your agents and

[2:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=178s) problems is to sandbox your agents and

[2:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=178s) problems is to sandbox your agents and move them to the cloud. And the product

[3:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=180s) move them to the cloud. And the product

[3:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=180s) move them to the cloud. And the product category that is emerging to support

[3:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=183s) category that is emerging to support

[3:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=183s) category that is emerging to support this uh is what um you know we call a

[3:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=185s) this uh is what um you know we call a

[3:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=186s) this uh is what um you know we call a cloud software factory. [snorts] And so

[3:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=188s) cloud software factory. [snorts] And so

[3:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=188s) cloud software factory. [snorts] And so this this term is kind of overloaded,

[3:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=190s) this this term is kind of overloaded,

[3:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=190s) this this term is kind of overloaded, but what it means to me is basically

[3:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=195s) but what it means to me is basically

[3:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=195s) but what it means to me is basically every company is going to set up a loop

[3:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=197s) every company is going to set up a loop

[3:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=197s) every company is going to set up a loop in the cloud where

[3:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=199s) in the cloud where

[3:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=199s) in the cloud where um a combination of humans and agents do

[3:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=203s) um a combination of humans and agents do

[3:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=203s) um a combination of humans and agents do work. And so humans are responsible for

[3:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=206s) work. And so humans are responsible for

[3:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=206s) work. And so humans are responsible for putting work in, agents will do

[3:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=209s) putting work in, agents will do

[3:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=209s) putting work in, agents will do different parts of the software life

[3:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=211s) different parts of the software life

[3:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=211s) different parts of the software life cycle. also a triage. They'll work with

[3:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=213s) cycle. also a triage. They'll work with

[3:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=213s) cycle. also a triage. They'll work with the human on a spec. They'll do

[3:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=215s) the human on a spec. They'll do

[3:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=215s) the human on a spec. They'll do implementation. They will do computer

[3:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=218s) implementation. They will do computer

[3:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=218s) implementation. They will do computer use verification. And as a as a

[3:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=221s) use verification. And as a as a

[3:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=221s) use verification. And as a as a business, what you're going to want to

[3:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=222s) business, what you're going to want to

[3:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=222s) business, what you're going to want to do is set these up internally. Define

[3:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=226s) do is set these up internally. Define

[3:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=226s) do is set these up internally. Define all the rules for how humans and agents

[3:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=227s) all the rules for how humans and agents

[3:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=227s) all the rules for how humans and agents [music]

[3:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=228s) [music]

[3:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=228s) [music] interact and then get to a spot where

[3:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=230s) interact and then get to a spot where

[3:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=230s) interact and then get to a spot where you can really, really measure and

[3:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=233s) you can really, really measure and

[3:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=233s) you can really, really measure and improve over time. Because getting this

[3:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=235s) improve over time. Because getting this

[3:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=235s) improve over time. Because getting this concept right is going to be the key

[3:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=238s) concept right is going to be the key

[3:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=238s) concept right is going to be the key driver to how your organization ships

[4:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=240s) driver to how your organization ships

[4:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=240s) driver to how your organization ships software.

[4:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=242s) software.

[4:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=242s) software. Um at at warp like our our belief is

[4:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=244s) Um at at warp like our our belief is

[4:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=244s) Um at at warp like our our belief is that you should think of this as

[4:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=246s) that you should think of this as

[4:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=246s) that you should think of this as infrastructure. So not as like a product

[4:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=248s) infrastructure. So not as like a product

[4:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=248s) infrastructure. So not as like a product or an AI teammate but really like an

[4:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=250s) or an AI teammate but really like an

[4:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=250s) or an AI teammate but really like an infrastructure stack that you can build

[4:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=253s) infrastructure stack that you can build

[4:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=253s) infrastructure stack that you can build on top of and scale. And the key aspects

[4:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=256s) on top of and scale. And the key aspects

[4:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=256s) on top of and scale. And the key aspects of this are like it should be open, it

[4:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=259s) of this are like it should be open, it

[4:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=259s) of this are like it should be open, it should be adaptable and you should think

[4:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=260s) should be adaptable and you should think

[4:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=260s) should be adaptable and you should think of your factories as something that you

[4:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=262s) of your factories as something that you

[4:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=262s) of your factories as something that you define in code. And so by open I mean

[4:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=267s) define in code. And so by open I mean

[4:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=267s) define in code. And so by open I mean you want a a factory model that can

[4:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=269s) you want a a factory model that can

[4:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=269s) you want a a factory model that can support different uh different models

[4:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=271s) support different uh different models

[4:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=271s) support different uh different models whether it's open weight or frontier.

[4:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=273s) whether it's open weight or frontier.

[4:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=273s) whether it's open weight or frontier. You want one that can support different

[4:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=275s) You want one that can support different

[4:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=275s) You want one that can support different harnesses and the reason is just like

[4:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=277s) harnesses and the reason is just like

[4:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=277s) harnesses and the reason is just like the best models and harnesses are

[4:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=279s) the best models and harnesses are

[4:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=279s) the best models and harnesses are constantly changing. You want adaptable

[4:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=282s) constantly changing. You want adaptable

[4:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=282s) constantly changing. You want adaptable means like the factory should work with

[4:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=283s) means like the factory should work with

[4:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=283s) means like the factory should work with your existing infrastructure. And then

[4:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=286s) your existing infrastructure. And then

[4:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=286s) your existing infrastructure. And then finally defined in code means like your

[4:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=289s) finally defined in code means like your

[4:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=289s) finally defined in code means like your factory should be a thing uh that has

[4:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=293s) factory should be a thing uh that has

[4:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=293s) factory should be a thing uh that has like a strict definition and that

[4:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=295s) like a strict definition and that

[4:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=295s) like a strict definition and that definition should include like what are

[4:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=297s) definition should include like what are

[4:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=297s) definition should include like what are the workflows, what are the skills, what

[4:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=299s) the workflows, what are the skills, what

[4:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=299s) the workflows, what are the skills, what are the MCPs, where do these run and

[5:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=301s) are the MCPs, where do these run and

[5:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=301s) are the MCPs, where do these run and when you define it in code and version

[5:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=303s) when you define it in code and version

[5:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=303s) when you define it in code and version control it then you get the ability to

[5:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=305s) control it then you get the ability to

[5:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=305s) control it then you get the ability to like test different versions of your

[5:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=307s) like test different versions of your

[5:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=307s) like test different versions of your factory. You get the ability for coding

[5:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=309s) factory. You get the ability for coding

[5:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=309s) factory. You get the ability for coding agents to update your factory. So these

[5:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=311s) agents to update your factory. So these

[5:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=311s) agents to update your factory. So these are some key characteristics.

[5:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=314s) are some key characteristics.

[5:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=314s) are some key characteristics. Uh I'll say another thing that I'm

[5:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=316s) Uh I'll say another thing that I'm

[5:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=316s) Uh I'll say another thing that I'm hearing from from businesses is like AI

[5:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=319s) hearing from from businesses is like AI

[5:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=319s) hearing from from businesses is like AI sovereignty matters. And so the concept

[5:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=321s) sovereignty matters. And so the concept

[5:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=321s) sovereignty matters. And so the concept here is like as you think about

[5:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=323s) here is like as you think about

[5:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=324s) here is like as you think about factories, you should think about uh

[5:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=326s) factories, you should think about uh

[5:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=326s) factories, you should think about uh owning all the data that goes in and all

[5:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=329s) owning all the data that goes in and all

[5:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=329s) owning all the data that goes in and all the data that comes out as well.

[5:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=332s) the data that comes out as well.

[5:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=332s) the data that comes out as well. You should think about how do you have

[5:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=333s) You should think about how do you have

[5:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=333s) You should think about how do you have these agents running on your own

[5:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=335s) these agents running on your own

[5:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=335s) these agents running on your own infrastructure and own compute. Um this

[5:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=337s) infrastructure and own compute. Um this

[5:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=337s) infrastructure and own compute. Um this is what most larger enterprises that we

[5:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=339s) is what most larger enterprises that we

[5:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=339s) is what most larger enterprises that we work with want.

[5:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=341s) work with want.

[5:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=341s) work with want. How do you provide your own inference?

[5:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=344s) How do you provide your own inference?

[5:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=344s) How do you provide your own inference? And then finally, like I said, any

[5:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=345s) And then finally, like I said, any

[5:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=345s) And then finally, like I said, any model, any harness. And so really

[5:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=348s) model, any harness. And so really

[5:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=348s) model, any harness. And so really preserve optionality and treat um treat

[5:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=351s) preserve optionality and treat um treat

[5:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=351s) preserve optionality and treat um treat these software factories as something

[5:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=352s) these software factories as something

[5:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=352s) these software factories as something that's very key to your business and

[5:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=355s) that's very key to your business and

[5:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=355s) that's very key to your business and that you want to, you know, mostly take

[5:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=357s) that you want to, you know, mostly take

[5:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=357s) that you want to, you know, mostly take ownership of the core ingredients of.

[6:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=361s) ownership of the core ingredients of.

[6:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=361s) ownership of the core ingredients of. Um, another really important point here

[6:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=363s) Um, another really important point here

[6:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=363s) Um, another really important point here is like as you think about building your

[6:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=366s) is like as you think about building your

[6:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=366s) is like as you think about building your software factory, like if you were to

[6:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=368s) software factory, like if you were to

[6:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=368s) software factory, like if you were to look on Twitter on any given day, you

[6:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=370s) look on Twitter on any given day, you

[6:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=370s) look on Twitter on any given day, you would probably see different model like

[6:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=373s) would probably see different model like

[6:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=373s) would probably see different model like someone would say like PI is the best or

[6:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=374s) someone would say like PI is the best or

[6:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=374s) someone would say like PI is the best or someone say Grock or like there's all

[6:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=376s) someone say Grock or like there's all

[6:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=376s) someone say Grock or like there's all these different things in the world

[6:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=378s) these different things in the world

[6:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=378s) these different things in the world around like what is best and you might

[6:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=379s) around like what is best and you might

[6:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=379s) around like what is best and you might even see benchmarks like Sweetbench or

[6:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=381s) even see benchmarks like Sweetbench or

[6:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=381s) even see benchmarks like Sweetbench or Terminal Bench. But the world that you

[6:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=383s) Terminal Bench. But the world that you

[6:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=383s) Terminal Bench. But the world that you want to get to um as you automate more

[6:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=386s) want to get to um as you automate more

[6:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=386s) want to get to um as you automate more development is actually being

[6:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=389s) development is actually being

[6:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=389s) development is actually being measurement first on your own data. And

[6:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=391s) measurement first on your own data. And

[6:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=391s) measurement first on your own data. And so you should whatever factory model you

[6:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=393s) so you should whatever factory model you

[6:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=393s) so you should whatever factory model you approach have the ability to score

[6:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=396s) approach have the ability to score

[6:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=396s) approach have the ability to score meaning like eval how your coding agents

[6:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=398s) meaning like eval how your coding agents

[6:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=398s) meaning like eval how your coding agents are doing. uh feed those scores into

[6:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=401s) are doing. uh feed those scores into

[6:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=401s) are doing. uh feed those scores into agents that can improve your factory and

[6:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=404s) agents that can improve your factory and

[6:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=404s) agents that can improve your factory and then also create your own benchmarks.

[6:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=406s) then also create your own benchmarks.

[6:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=406s) then also create your own benchmarks. Meaning like be able to take all of your

[6:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=408s) Meaning like be able to take all of your

[6:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=408s) Meaning like be able to take all of your real coding agent tasks and try them

[6:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=411s) real coding agent tasks and try them

[6:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=411s) real coding agent tasks and try them against different models and measure how

[6:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=412s) against different models and measure how

[6:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=412s) against different models and measure how it goes.

[6:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=415s) it goes.

[6:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=415s) it goes. So I'll also talk a little bit about

[6:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=417s) So I'll also talk a little bit about

[6:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=417s) So I'll also talk a little bit about like um how engineering changes in this

[7:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=421s) like um how engineering changes in this

[7:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=421s) like um how engineering changes in this factory world.

[7:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=422s) [snorts] And so the way that I I conceive of this

[7:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=425s) And so the way that I I conceive of this

[7:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=425s) And so the way that I I conceive of this and talk about it is you will still have

[7:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=427s) and talk about it is you will still have

[7:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=427s) and talk about it is you will still have engineers who are very much responsible

[7:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=429s) engineers who are very much responsible

[7:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=429s) engineers who are very much responsible for building useful things for your

[7:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=432s) for building useful things for your

[7:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=432s) for building useful things for your customers. And so they'll build be

[7:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=433s) customers. And so they'll build be

[7:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=433s) customers. And so they'll build be building the features, they'll be fixing

[7:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=435s) building the features, they'll be fixing

[7:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=435s) building the features, they'll be fixing bugs, they'll be doing product

[7:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=436s) bugs, they'll be doing product

[7:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=436s) bugs, they'll be doing product engineering and they'll be doing it like

[7:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=438s) engineering and they'll be doing it like

[7:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=438s) engineering and they'll be doing it like through the factory. So they put work

[7:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=439s) through the factory. So they put work

[7:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=440s) through the factory. So they put work into the factory. But there's a second

[7:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=442s) into the factory. But there's a second

[7:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=442s) into the factory. But there's a second kind of engineering which I call factory

[7:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=445s) kind of engineering which I call factory

[7:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=445s) kind of engineering which I call factory engineering which is uh either uh your

[7:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=449s) engineering which is uh either uh your

[7:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=449s) engineering which is uh either uh your platform team or just your general

[7:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=451s) platform team or just your general

[7:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=451s) platform team or just your general engineers will have to get good at like

[7:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=454s) engineers will have to get good at like

[7:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=454s) engineers will have to get good at like trying to make the factory better. So

[7:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=457s) trying to make the factory better. So

[7:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=457s) trying to make the factory better. So improving the skills,

[7:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=459s) improving the skills,

[7:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=459s) improving the skills, setting the context, changing the model

[7:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=461s) setting the context, changing the model

[7:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=461s) setting the context, changing the model mix, all of these things are going to

[7:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=463s) mix, all of these things are going to

[7:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=463s) mix, all of these things are going to become more important as engineers are

[7:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=465s) become more important as engineers are

[7:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=465s) become more important as engineers are like building and improving the system

[7:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=468s) like building and improving the system

[7:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=468s) like building and improving the system that actually builds the software.

[7:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=471s) that actually builds the software.

[7:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=471s) that actually builds the software. So, I'm going to I'm going to show what

[7:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=473s) So, I'm going to I'm going to show what

[7:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=473s) So, I'm going to I'm going to show what some of this looks like in practice. And

[7:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=474s) some of this looks like in practice. And

[7:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=474s) some of this looks like in practice. And again, this is going to be in the

[7:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=476s) again, this is going to be in the

[7:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=476s) again, this is going to be in the context of our own product, Warp

[7:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=478s) context of our own product, Warp

[7:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=478s) context of our own product, Warp Factories. But these general concepts, I

[8:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=480s) Factories. But these general concepts, I

[8:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=480s) Factories. But these general concepts, I think, will translate to however you're

[8:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=483s) think, will translate to however you're

[8:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=483s) think, will translate to however you're thinking about building your factories.

[8:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=486s) thinking about building your factories.

[8:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=486s) thinking about building your factories. I'll try to leave a little bit of time

[8:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=487s) I'll try to leave a little bit of time

[8:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=487s) I'll try to leave a little bit of time for questions at the end if I can.

[8:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=490s) for questions at the end if I can.

[8:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=490s) for questions at the end if I can. So

[8:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=492s) So

[8:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=492s) So I'm going to start with um the view of

[8:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=496s) I'm going to start with um the view of

[8:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=496s) I'm going to start with um the view of the factory that the factory engineer or

[8:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=499s) the factory that the factory engineer or

[8:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=499s) the factory that the factory engineer or the manager has. And so this is like

[8:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=503s) the manager has. And so this is like

[8:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=503s) the manager has. And so this is like if you are

[8:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=505s) if you are

[8:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=505s) if you are someone who is trying to get better uh

[8:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=508s) someone who is trying to get better uh

[8:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=508s) someone who is trying to get better uh productivity across your team, you're

[8:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=510s) productivity across your team, you're

[8:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=510s) productivity across your team, you're going to want some view of like what is

[8:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=513s) going to want some view of like what is

[8:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=514s) going to want some view of like what is happening from a productivity standpoint

[8:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=515s) happening from a productivity standpoint

[8:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=515s) happening from a productivity standpoint like how automated is my factory? Uh

[8:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=518s) like how automated is my factory? Uh

[8:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=518s) like how automated is my factory? Uh you're going to want a view of um what's

[8:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=521s) you're going to want a view of um what's

[8:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=521s) you're going to want a view of um what's it what's the cycle time. So like a

[8:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=523s) it what's the cycle time. So like a

[8:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=523s) it what's the cycle time. So like a dorometric view. You're going to want a

[8:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=525s) dorometric view. You're going to want a

[8:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=525s) dorometric view. You're going to want a view of uh spending. This might take a

[8:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=527s) view of uh spending. This might take a

[8:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=527s) view of uh spending. This might take a second to load. Um but basically you

[8:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=530s) second to load. Um but basically you

[8:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=530s) second to load. Um but basically you want all this data tracked. And the way

[8:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=533s) want all this data tracked. And the way

[8:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=533s) want all this data tracked. And the way that you get these views is by having a

[8:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=537s) that you get these views is by having a

[8:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=537s) that you get these views is by having a system where everything that your team

[9:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=541s) system where everything that your team

[9:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=541s) system where everything that your team is doing, I don't know if you all can

[9:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=543s) is doing, I don't know if you all can

[9:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=543s) is doing, I don't know if you all can see this, but everything that's

[9:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=545s) see this, but everything that's

[9:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=545s) see this, but everything that's happening on our team, and this is

[9:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=547s) happening on our team, and this is

[9:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=547s) happening on our team, and this is Warp's internal software development

[9:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=549s) Warp's internal software development

[9:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=549s) Warp's internal software development that I'm leaking here, but it's like I

[9:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=552s) that I'm leaking here, but it's like I

[9:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=552s) that I'm leaking here, but it's like I can see what I can see what every

[9:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=554s) can see what I can see what every

[9:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=554s) can see what I can see what every engineer on our team is doing, and I can

[9:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=556s) engineer on our team is doing, and I can

[9:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=556s) engineer on our team is doing, and I can see what all the automations that we're

[9:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=557s) see what all the automations that we're

[9:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=557s) see what all the automations that we're running are doing. And not just can I

[9:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=560s) running are doing. And not just can I

[9:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=560s) running are doing. And not just can I see them like I can see all the

[9:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=561s) see them like I can see all the

[9:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=561s) see them like I can see all the artifacts that they're creating, but all

[9:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=563s) artifacts that they're creating, but all

[9:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=563s) artifacts that they're creating, but all this is getting stored. And so all these

[9:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=566s) this is getting stored. And so all these

[9:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=566s) this is getting stored. And so all these agent traces, you know, I I could

[9:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=568s) agent traces, you know, I I could

[9:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=568s) agent traces, you know, I I could actually go in I could see what this

[9:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=570s) actually go in I could see what this

[9:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=570s) actually go in I could see what this this one is doing. I could go in and

[9:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=572s) this one is doing. I could go in and

[9:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=572s) this one is doing. I could go in and like see the actual conversation that

[9:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=574s) like see the actual conversation that

[9:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=574s) like see the actual conversation that someone is doing. But the important

[9:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=576s) someone is doing. But the important

[9:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=576s) someone is doing. But the important thing here is that um all this data is

[9:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=579s) thing here is that um all this data is

[9:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=579s) thing here is that um all this data is getting stored and what your coding

[9:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=581s) getting stored and what your coding

[9:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=581s) getting stored and what your coding agents are doing is the raw material for

[9:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=584s) agents are doing is the raw material for

[9:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=584s) agents are doing is the raw material for improving them. And so, uh, you might be

[9:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=586s) improving them. And so, uh, you might be

[9:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=586s) improving them. And so, uh, you might be wondering, well, how do how do engineers

[9:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=589s) wondering, well, how do how do engineers

[9:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=589s) wondering, well, how do how do engineers actually do this work? Like, how are

[9:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=590s) actually do this work? Like, how are

[9:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=590s) actually do this work? Like, how are they putting stuff in? Um, and the

[9:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=592s) they putting stuff in? Um, and the

[9:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=592s) they putting stuff in? Um, and the answer on this is that they put it in

[9:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=595s) answer on this is that they put it in

[9:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=595s) answer on this is that they put it in through the tools that they're already

[9:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=596s) through the tools that they're already

[9:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=596s) through the tools that they're already using. So, on our team, they're using

[9:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=599s) using. So, on our team, they're using

[9:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=599s) using. So, on our team, they're using Slack uh to do a lot of the work. Um,

[10:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=602s) Slack uh to do a lot of the work. Um,

[10:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=602s) Slack uh to do a lot of the work. Um, they put in uh agentic work through our

[10:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=604s) they put in uh agentic work through our

[10:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=604s) they put in uh agentic work through our task tracker, which we use linear, but

[10:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=606s) task tracker, which we use linear, but

[10:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=606s) task tracker, which we use linear, but it could be Jira. They have the same

[10:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=608s) it could be Jira. They have the same

[10:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=608s) it could be Jira. They have the same context whether they're uh responding to

[10:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=611s) context whether they're uh responding to

[10:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=611s) context whether they're uh responding to a GitHub PR comment. And so the the idea

[10:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=614s) a GitHub PR comment. And so the the idea

[10:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=614s) a GitHub PR comment. And so the the idea here is like the product engineers are

[10:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=615s) here is like the product engineers are

[10:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=616s) here is like the product engineers are using the tools that they're already

[10:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=617s) using the tools that they're already

[10:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=617s) using the tools that they're already using, but all the work goes into this

[10:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=619s) using, but all the work goes into this

[10:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=619s) using, but all the work goes into this centralized location where you get a

[10:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=621s) centralized location where you get a

[10:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=621s) centralized location where you get a centralized view and all this stuff gets

[10:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=623s) centralized view and all this stuff gets

[10:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=623s) centralized view and all this stuff gets recorded over time by recording

[10:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=625s) recorded over time by recording

[10:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=625s) recorded over time by recording everything. Uh just to go back to this

[10:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=628s) everything. Uh just to go back to this

[10:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=628s) everything. Uh just to go back to this dashboard for a second. Um actually let

[10:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=632s) dashboard for a second. Um actually let

[10:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=632s) dashboard for a second. Um actually let me let me not do that for one second.

[10:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=633s) me let me not do that for one second.

[10:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=633s) me let me not do that for one second. The other thing that's important here

[10:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=634s) The other thing that's important here

[10:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=634s) The other thing that's important here that I I talked about earlier is like

[10:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=636s) that I I talked about earlier is like

[10:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=636s) that I I talked about earlier is like the factory workflow itself defining it

[10:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=639s) the factory workflow itself defining it

[10:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=640s) the factory workflow itself defining it as code. And so this is um our kind of

[10:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=643s) as code. And so this is um our kind of

[10:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=643s) as code. And so this is um our kind of like default factory workflow where

[10:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=646s) like default factory workflow where

[10:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=646s) like default factory workflow where think of it as like workflows in we have

[10:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=649s) think of it as like workflows in we have

[10:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=649s) think of it as like workflows in we have this orchestrator agent that will figure

[10:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=651s) this orchestrator agent that will figure

[10:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=651s) this orchestrator agent that will figure out okay should this next go to a triage

[10:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=654s) out okay should this next go to a triage

[10:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=654s) out okay should this next go to a triage step should it go to a spec step an

[10:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=655s) step should it go to a spec step an

[10:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=655s) step should it go to a spec step an implementation step code review the

[10:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=658s) implementation step code review the

[10:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=658s) implementation step code review the actual details of this need to be very

[11:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=660s) actual details of this need to be very

[11:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=660s) actual details of this need to be very very flexible. Um if you click into one

[11:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=663s) very flexible. Um if you click into one

[11:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=663s) very flexible. Um if you click into one of these you can see like essentially

[11:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=666s) of these you can see like essentially

[11:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=666s) of these you can see like essentially you are providing an agent definition

[11:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=669s) you are providing an agent definition

[11:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=669s) you are providing an agent definition there's uh which is like what are all

[11:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=671s) there's uh which is like what are all

[11:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=671s) there's uh which is like what are all the secrets the MCPS the agent prompt

[11:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=674s) the secrets the MCPS the agent prompt

[11:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=674s) the secrets the MCPS the agent prompt the model all this stuff the the cool

[11:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=677s) the model all this stuff the the cool

[11:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=677s) the model all this stuff the the cool thing though that I would emphasize that

[11:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=680s) thing though that I would emphasize that

[11:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=680s) thing though that I would emphasize that however you're approaching this you

[11:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=681s) however you're approaching this you

[11:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=681s) however you're approaching this you probably want to do is that there's an

[11:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=684s) probably want to do is that there's an

[11:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=684s) probably want to do is that there's an actual code repo that sits behind this

[11:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=687s) actual code repo that sits behind this

[11:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=687s) actual code repo that sits behind this where if you go and like look in that

[11:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=689s) where if you go and like look in that

[11:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=689s) where if you go and like look in that repo you can see like I that earlier, a

[11:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=692s) repo you can see like I that earlier, a

[11:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=692s) repo you can see like I that earlier, a definition of the entire factory at any

[11:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=695s) definition of the entire factory at any

[11:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=695s) definition of the entire factory at any given point in time. And so that

[11:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=696s) given point in time. And so that

[11:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=696s) given point in time. And so that definition includes, think of it as like

[11:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=699s) definition includes, think of it as like

[11:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=699s) definition includes, think of it as like everything that goes into the process of

[11:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=701s) everything that goes into the process of

[11:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=701s) everything that goes into the process of automating development. Um, it's your

[11:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=704s) automating development. Um, it's your

[11:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=704s) automating development. Um, it's your it's your uh agent definitions, it's

[11:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=707s) it's your uh agent definitions, it's

[11:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=707s) it's your uh agent definitions, it's your automations, your benchmarks, it's

[11:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=709s) your automations, your benchmarks, it's

[11:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=709s) your automations, your benchmarks, it's where these agents run, the skills, the

[11:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=710s) where these agents run, the skills, the

[11:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=710s) where these agents run, the skills, the web hooks, everything. So everything

[11:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=712s) web hooks, everything. So everything

[11:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=712s) web hooks, everything. So everything gets stored in in a repo that it allows

[11:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=715s) gets stored in in a repo that it allows

[11:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=715s) gets stored in in a repo that it allows for automation. Um, and I'll I'll I'll

[11:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=718s) for automation. Um, and I'll I'll I'll

[11:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=718s) for automation. Um, and I'll I'll I'll show a little more why that matters in a

[12:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=720s) show a little more why that matters in a

[12:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=720s) show a little more why that matters in a second.

[12:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=721s) second.

[12:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=721s) second. The other thing uh just to return to

[12:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=723s) The other thing uh just to return to

[12:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=723s) The other thing uh just to return to this view for a second because I think

[12:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=724s) this view for a second because I think

[12:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=724s) this view for a second because I think it's interesting. You get these metrics

[12:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=727s) it's interesting. You get these metrics

[12:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=727s) it's interesting. You get these metrics around productivity and spend but you

[12:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=729s) around productivity and spend but you

[12:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=729s) around productivity and spend but you also want to set up something where

[12:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=731s) also want to set up something where

[12:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=731s) also want to set up something where there are like we call them scores. You

[12:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=734s) there are like we call them scores. You

[12:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=734s) there are like we call them scores. You could think of it as like online evals.

[12:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=737s) could think of it as like online evals.

[12:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=737s) could think of it as like online evals. And what what you're looking at here is

[12:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=740s) And what what you're looking at here is

[12:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=740s) And what what you're looking at here is basically we have a set of agents that

[12:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=743s) basically we have a set of agents that

[12:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=743s) basically we have a set of agents that are looking at all of these runs that

[12:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=745s) are looking at all of these runs that

[12:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=745s) are looking at all of these runs that are going through our system. So

[12:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=747s) are going through our system. So

[12:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=747s) are going through our system. So remember I go back here. It's like you

[12:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=748s) remember I go back here. It's like you

[12:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=748s) remember I go back here. It's like you can see everything that everyone's doing

[12:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=750s) can see everything that everyone's doing

[12:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=750s) can see everything that everyone's doing on our team. You then have a scoring

[12:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=752s) on our team. You then have a scoring

[12:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=752s) on our team. You then have a scoring mechanism that goes and looks at those

[12:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=755s) mechanism that goes and looks at those

[12:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=755s) mechanism that goes and looks at those runs and sort of figures out by using

[12:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=759s) runs and sort of figures out by using

[12:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=759s) runs and sort of figures out by using LLM as a judge whether or not um those

[12:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=763s) LLM as a judge whether or not um those

[12:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=763s) LLM as a judge whether or not um those runs are good or not. And so for

[12:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=765s) runs are good or not. And so for

[12:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=766s) runs are good or not. And so for instance we have a scorer here that is

[12:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=769s) instance we have a scorer here that is

[12:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=769s) instance we have a scorer here that is looking for agents that are writing

[12:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=770s) looking for agents that are writing

[12:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=770s) looking for agents that are writing redundant tests which is a common

[12:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=772s) redundant tests which is a common

[12:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=772s) redundant tests which is a common failure mode with these coding agents.

[12:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=774s) failure mode with these coding agents.

[12:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=774s) failure mode with these coding agents. And so for the most part, our our agents

[12:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=776s) And so for the most part, our our agents

[12:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=776s) And so for the most part, our our agents seem to be doing pretty well here. But

[12:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=778s) seem to be doing pretty well here. But

[12:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=778s) seem to be doing pretty well here. But if you look back in time, you'll see

[12:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=779s) if you look back in time, you'll see

[12:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=779s) if you look back in time, you'll see there's some set of them where an agent

[13:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=781s) there's some set of them where an agent

[13:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=781s) there's some set of them where an agent is deciding, okay, this was this thing

[13:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=784s) is deciding, okay, this was this thing

[13:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=784s) is deciding, okay, this was this thing wrote tests that didn't need to be

[13:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=785s) wrote tests that didn't need to be

[13:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=785s) wrote tests that didn't need to be written. And so every one of these

[13:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=787s) written. And so every one of these

[13:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=787s) written. And so every one of these failures is an opportunity for improving

[13:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=791s) failures is an opportunity for improving

[13:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=791s) failures is an opportunity for improving how your system works. And so as you

[13:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=794s) how your system works. And so as you

[13:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=794s) how your system works. And so as you gather this information, and you know,

[13:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=797s) gather this information, and you know,

[13:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=797s) gather this information, and you know, you could go look at this as a person

[13:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=798s) you could go look at this as a person

[13:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=798s) you could go look at this as a person and grade it, but it's much more

[13:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=799s) and grade it, but it's much more

[13:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=800s) and grade it, but it's much more efficient to have agents do it. As you

[13:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=802s) efficient to have agents do it. As you

[13:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=802s) efficient to have agents do it. As you gather this information, you can then

[13:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=804s) gather this information, you can then

[13:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=804s) gather this information, you can then use it to feed what we call a

[13:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=806s) use it to feed what we call a

[13:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=806s) use it to feed what we call a self-improvement loop. And so the idea

[13:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=809s) self-improvement loop. And so the idea

[13:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=809s) self-improvement loop. And so the idea behind a self-improvement loop is that

[13:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=812s) behind a self-improvement loop is that

[13:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=812s) behind a self-improvement loop is that uh agents will go and they'll look at

[13:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=814s) uh agents will go and they'll look at

[13:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=814s) uh agents will go and they'll look at all these failure modes. They'll

[13:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=816s) all these failure modes. They'll

[13:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=816s) all these failure modes. They'll aggregate them. They will look for

[13:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=818s) aggregate them. They will look for

[13:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=818s) aggregate them. They will look for patterns and then they will come up with

[13:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=820s) patterns and then they will come up with

[13:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=820s) patterns and then they will come up with ideas for how do you improve the

[13:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=823s) ideas for how do you improve the

[13:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=823s) ideas for how do you improve the underlying factory definition so that

[13:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=826s) underlying factory definition so that

[13:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=826s) underlying factory definition so that going forward you don't have this same

[13:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=828s) going forward you don't have this same

[13:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=828s) going forward you don't have this same failure mode. And so you know for

[13:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=831s) failure mode. And so you know for

[13:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=832s) failure mode. And so you know for instance um I don't know I'm going to

[13:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=833s) instance um I don't know I'm going to

[13:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=833s) instance um I don't know I'm going to pick a random one here. Um this is like

[13:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=838s) pick a random one here. Um this is like

[13:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=838s) pick a random one here. Um this is like read matching repository skills before

[14:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=840s) read matching repository skills before

[14:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=840s) read matching repository skills before implementing. The key thing here is like

[14:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=843s) implementing. The key thing here is like

[14:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=843s) implementing. The key thing here is like it's saying I saw this fail 37 times and

[14:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=847s) it's saying I saw this fail 37 times and

[14:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=847s) it's saying I saw this fail 37 times and then it's saying okay we can actually um

[14:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=851s) then it's saying okay we can actually um

[14:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=851s) then it's saying okay we can actually um oops we can actually fix this

[14:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=855s) oops we can actually fix this

[14:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=855s) oops we can actually fix this um with a PR

[14:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=858s) um with a PR

[14:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=858s) um with a PR to the factory definition.

[14:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=861s) to the factory definition.

[14:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=861s) to the factory definition. So this is like closing the loop where

[14:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=864s) So this is like closing the loop where

[14:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=864s) So this is like closing the loop where the agent has identified a problem then

[14:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=865s) the agent has identified a problem then

[14:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=866s) the agent has identified a problem then it's suggesting a change to actually

[14:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=868s) it's suggesting a change to actually

[14:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=868s) it's suggesting a change to actually drive a better result in the factory. So

[14:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=870s) drive a better result in the factory. So

[14:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=870s) drive a better result in the factory. So this is like a a core concept I think as

[14:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=873s) this is like a a core concept I think as

[14:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=873s) this is like a a core concept I think as you think of like how do you really

[14:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=874s) you think of like how do you really

[14:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=874s) you think of like how do you really improve development it's it's about

[14:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=877s) improve development it's it's about

[14:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=877s) improve development it's it's about creating these loops where agents can

[14:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=879s) creating these loops where agents can

[14:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=879s) creating these loops where agents can identify problems and then fix them.

[14:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=881s) identify problems and then fix them.

[14:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=882s) identify problems and then fix them. There's one more thing that I want to

[14:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=883s) There's one more thing that I want to

[14:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=883s) There's one more thing that I want to show and then maybe I'll take a couple

[14:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=885s) show and then maybe I'll take a couple

[14:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=885s) show and then maybe I'll take a couple questions. Um I mentioned early on like

[14:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=889s) questions. Um I mentioned early on like

[14:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=889s) questions. Um I mentioned early on like you can't really trust the public

[14:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=891s) you can't really trust the public

[14:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=892s) you can't really trust the public benchmarks [clears throat]

[14:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=893s) benchmarks [clears throat]

[14:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=893s) benchmarks [clears throat] as like a way of understanding how good

[14:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=895s) as like a way of understanding how good

[14:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=895s) as like a way of understanding how good a certain model or harness is going to

[14:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=897s) a certain model or harness is going to

[14:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=897s) a certain model or harness is going to be on your own workflows. And so what

[14:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=899s) be on your own workflows. And so what

[14:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=899s) be on your own workflows. And so what you want to get to a point where you can

[15:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=901s) you want to get to a point where you can

[15:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=901s) you want to get to a point where you can do is basically take any past run. So we

[15:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=906s) do is basically take any past run. So we

[15:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=906s) do is basically take any past run. So we can do this like with any anything that

[15:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=909s) can do this like with any anything that

[15:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=909s) can do this like with any anything that we've done in our system. I could go

[15:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=910s) we've done in our system. I could go

[15:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=910s) we've done in our system. I could go into it here and I have the um

[15:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=914s) into it here and I have the um

[15:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=914s) into it here and I have the um sorry I picked a bad example. I um

[15:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=920s) uh maybe I'm picking a bad example. The

[15:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=922s) uh maybe I'm picking a bad example. The

[15:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=922s) uh maybe I'm picking a bad example. The the idea here is like you can basically

[15:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=924s) the idea here is like you can basically

[15:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=924s) the idea here is like you can basically take past runs. I don't know why I can't

[15:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=928s) take past runs. I don't know why I can't

[15:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=928s) take past runs. I don't know why I can't do this here and say okay I want to add

[15:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=930s) do this here and say okay I want to add

[15:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=930s) do this here and say okay I want to add this to a benchmark suite and what this

[15:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=933s) this to a benchmark suite and what this

[15:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=933s) this to a benchmark suite and what this is doing is say okay this is like a

[15:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=935s) is doing is say okay this is like a

[15:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=935s) is doing is say okay this is like a representative thing that I did in my

[15:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=936s) representative thing that I did in my

[15:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=936s) representative thing that I did in my factory that could be useful to measure

[15:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=940s) factory that could be useful to measure

[15:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=940s) factory that could be useful to measure by testing it with different model

[15:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=941s) by testing it with different model

[15:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=941s) by testing it with different model configurations and so think of like

[15:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=943s) configurations and so think of like

[15:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=943s) configurations and so think of like building up a a suite of like front-end

[15:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=945s) building up a a suite of like front-end

[15:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=945s) building up a a suite of like front-end tasks and the question that you're

[15:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=947s) tasks and the question that you're

[15:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=947s) tasks and the question that you're trying to answer is like what's the best

[15:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=949s) trying to answer is like what's the best

[15:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=950s) trying to answer is like what's the best model for me to use for front-end tasks

[15:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=951s) model for me to use for front-end tasks

[15:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=951s) model for me to use for front-end tasks and like you might look on Twitter but

[15:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=953s) and like you might look on Twitter but

[15:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=953s) and like you might look on Twitter but you might also just be like let

[15:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=955s) you might also just be like let

[15:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=955s) you might also just be like let actually test. And so what you can do or

[15:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=959s) actually test. And so what you can do or

[15:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=959s) actually test. And so what you can do or should try to set up for your own system

[16:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=962s) should try to set up for your own system

[16:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=962s) should try to set up for your own system and this is built into warp factory. So

[16:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=964s) and this is built into warp factory. So

[16:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=964s) and this is built into warp factory. So I'll show it here is basically curate

[16:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=967s) I'll show it here is basically curate

[16:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=967s) I'll show it here is basically curate one of these benchmarks and what what it

[16:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=971s) one of these benchmarks and what what it

[16:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=971s) one of these benchmarks and what what it does is it will replay your real past

[16:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=975s) does is it will replay your real past

[16:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=975s) does is it will replay your real past coding agent tasks and then generate for

[16:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=978s) coding agent tasks and then generate for

[16:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=978s) coding agent tasks and then generate for you what looks like a basically a cost

[16:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=981s) you what looks like a basically a cost

[16:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=981s) you what looks like a basically a cost versus quality trade-off graph. And so

[16:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=984s) versus quality trade-off graph. And so

[16:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=984s) versus quality trade-off graph. And so this is something you'll see when um you

[16:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=986s) this is something you'll see when um you

[16:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=986s) this is something you'll see when um you know the public benchmarks come out, but

[16:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=988s) know the public benchmarks come out, but

[16:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=988s) know the public benchmarks come out, but this is on your own data. And so in in

[16:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=991s) this is on your own data. And so in in

[16:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=991s) this is on your own data. And so in in this case, for example, like we could

[16:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=993s) this case, for example, like we could

[16:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=993s) this case, for example, like we could see GLM53 flash uh was 93% correct on

[16:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=998s) see GLM53 flash uh was 93% correct on

[16:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=998s) see GLM53 flash uh was 93% correct on these tasks at a cost of $2. GPT56 soul

[16:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1002s) these tasks at a cost of $2. GPT56 soul

[16:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1002s) these tasks at a cost of $2. GPT56 soul was like 95% correct, but it cost three

[16:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1005s) was like 95% correct, but it cost three

[16:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1005s) was like 95% correct, but it cost three times as much. And so you can take this

[16:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1008s) times as much. And so you can take this

[16:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1008s) times as much. And so you can take this information and then build a model

[16:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1009s) information and then build a model

[16:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1009s) information and then build a model routing strategy that you have real

[16:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1012s) routing strategy that you have real

[16:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1012s) routing strategy that you have real confidence is not just a generic one but

[16:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1016s) confidence is not just a generic one but

[16:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1016s) confidence is not just a generic one but for your actual code your stack is

[16:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1019s) for your actual code your stack is

[16:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1019s) for your actual code your stack is actually providing value. And so um this

[17:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1022s) actually providing value. And so um this

[17:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1022s) actually providing value. And so um this is like I think a good example of like

[17:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1026s) is like I think a good example of like

[17:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1026s) is like I think a good example of like closing the loop. And then just to go

[17:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1028s) closing the loop. And then just to go

[17:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1028s) closing the loop. And then just to go back I'm going to finish up here in a

[17:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1030s) back I'm going to finish up here in a

[17:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1030s) back I'm going to finish up here in a second.

[17:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1032s) second.

[17:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1032s) second. The

[17:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1035s) The

[17:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1035s) The the sort of like way that we think about

[17:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1037s) the sort of like way that we think about

[17:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1037s) the sort of like way that we think about this is like at the bottom of your

[17:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1039s) this is like at the bottom of your

[17:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1039s) this is like at the bottom of your stack, it's defined in code. It's think

[17:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1042s) stack, it's defined in code. It's think

[17:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1042s) stack, it's defined in code. It's think of a data layer. What's the data going

[17:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1044s) of a data layer. What's the data going

[17:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1044s) of a data layer. What's the data going in and out? Where do these agents run?

[17:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1046s) in and out? Where do these agents run?

[17:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1046s) in and out? Where do these agents run? You really want to sandbox that. You

[17:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1049s) You really want to sandbox that. You

[17:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1049s) You really want to sandbox that. You want inference that's flexible. You want

[17:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1052s) want inference that's flexible. You want

[17:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1052s) want inference that's flexible. You want a self-improvement layer that gives you

[17:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1054s) a self-improvement layer that gives you

[17:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1054s) a self-improvement layer that gives you something like benchmark scoring. You

[17:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1056s) something like benchmark scoring. You

[17:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1056s) something like benchmark scoring. You want orchestration so you can have

[17:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1058s) want orchestration so you can have

[17:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1058s) want orchestration so you can have multi-step workflows. And then you want

[17:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1060s) multi-step workflows. And then you want

[17:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1060s) multi-step workflows. And then you want access from all these existing tools.

[17:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1062s) access from all these existing tools.

[17:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1062s) access from all these existing tools. And so that's like the infrastructure

[17:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1064s) And so that's like the infrastructure

[17:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1064s) And so that's like the infrastructure that I think is going to it's going to

[17:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1066s) that I think is going to it's going to

[17:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1066s) that I think is going to it's going to look something like this. Um no matter

[17:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1069s) look something like this. Um no matter

[17:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1069s) look something like this. Um no matter how you approach this problem uh if

[17:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1072s) how you approach this problem uh if

[17:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1072s) how you approach this problem uh if you're interested in our approach

[17:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1073s) you're interested in our approach

[17:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1073s) you're interested in our approach obviously we're we're we would love to

[17:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1075s) obviously we're we're we would love to

[17:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1075s) obviously we're we're we would love to talk to you. But let me pause there.

[17:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1078s) talk to you. But let me pause there.

[17:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1078s) talk to you. But let me pause there. Hopefully this was interesting to folks.

[17:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1079s) Hopefully this was interesting to folks.

[17:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1079s) Hopefully this was interesting to folks. And I might have time for one question.

[18:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1082s) And I might have time for one question.

[18:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1082s) And I might have time for one question. I don't know. Or I might have to go.

[18:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1086s) I don't know. Or I might have to go.

[18:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1086s) I don't know. Or I might have to go. Yes. What what uh what advice do you

[18:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1088s) Yes. What what uh what advice do you

[18:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1088s) Yes. What what uh what advice do you have for testing and verification of

[18:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1091s) have for testing and verification of

[18:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1091s) have for testing and verification of code at the factory?

[18:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1092s) code at the factory?

[18:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1092s) code at the factory? Awesome question. So the you know one

[18:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1095s) Awesome question. So the you know one

[18:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1095s) Awesome question. So the you know one thing that could happen if you do this

[18:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1097s) thing that could happen if you do this

[18:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1097s) thing that could happen if you do this approach is you get like a pile up of

[18:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1099s) approach is you get like a pile up of

[18:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1099s) approach is you get like a pile up of PRs right and so you want some way of

[18:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1101s) PRs right and so you want some way of

[18:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1101s) PRs right and so you want some way of like getting them through the big things

[18:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1103s) like getting them through the big things

[18:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1103s) like getting them through the big things that I advise are spec driven

[18:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1105s) that I advise are spec driven

[18:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1105s) that I advise are spec driven development. So you can actually test um

[18:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1108s) development. So you can actually test um

[18:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1108s) development. So you can actually test um make sure that the changes are adherent

[18:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1110s) make sure that the changes are adherent

[18:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1110s) make sure that the changes are adherent to a spec. And the other thing is

[18:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1112s) to a spec. And the other thing is

[18:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1112s) to a spec. And the other thing is computer use verification which I didn't

[18:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1114s) computer use verification which I didn't

[18:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1114s) computer use verification which I didn't show but actually um you know if you can

[18:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1117s) show but actually um you know if you can

[18:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1117s) show but actually um you know if you can have agents that verify or produce

[18:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1120s) have agents that verify or produce

[18:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1120s) have agents that verify or produce evidence from the user standpoint that

[18:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1121s) evidence from the user standpoint that

[18:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1122s) evidence from the user standpoint that the thing works that can go a long way

[18:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1124s) the thing works that can go a long way

[18:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1124s) the thing works that can go a long way in terms of like lessening the code

[18:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1126s) in terms of like lessening the code

[18:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1126s) in terms of like lessening the code review burden. So those are the those

[18:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1127s) review burden. So those are the those

[18:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1127s) review burden. So those are the those are the two things that I recommend.

[18:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1130s) are the two things that I recommend.

[18:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1130s) are the two things that I recommend. Yes.

[18:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1130s) Yes.

[18:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1130s) Yes. So how do you plan um the budgets when

[18:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1133s) So how do you plan um the budgets when

[18:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1133s) So how do you plan um the budgets when we are starting a project? How do you

[18:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1137s) we are starting a project? How do you

[18:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1137s) we are starting a project? How do you plan these verification costs?

[18:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1139s) plan these verification costs?

[18:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1139s) plan these verification costs? Yeah. So the the way that we do this is

[19:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1142s) Yeah. So the the way that we do this is

[19:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1142s) Yeah. So the the way that we do this is like we give people the tools to measure

[19:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1144s) like we give people the tools to measure

[19:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1144s) like we give people the tools to measure like what's your cost per PR and then

[19:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1147s) like what's your cost per PR and then

[19:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1147s) like what's your cost per PR and then what percent of the cost is going

[19:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1148s) what percent of the cost is going

[19:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1148s) what percent of the cost is going towards like scoring or benchmarking or

[19:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1150s) towards like scoring or benchmarking or

[19:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1150s) towards like scoring or benchmarking or stuff like that. Uh and then you can

[19:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1152s) stuff like that. Uh and then you can

[19:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1152s) stuff like that. Uh and then you can kind of extrapolate it out like are you

[19:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1154s) kind of extrapolate it out like are you

[19:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1154s) kind of extrapolate it out like are you willing to spend x amount on a project

[19:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1156s) willing to spend x amount on a project

[19:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1156s) willing to spend x amount on a project or whatever. But the the key the key

[19:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1159s) or whatever. But the the key the key

[19:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1159s) or whatever. But the the key the key thing um that that I want people to

[19:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1161s) thing um that that I want people to

[19:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1161s) thing um that that I want people to leave if there's one idea is like really

[19:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1164s) leave if there's one idea is like really

[19:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1164s) leave if there's one idea is like really set up something that you can measure

[19:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1165s) set up something that you can measure

[19:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1165s) set up something that you can measure and improve. And once you can truly

[19:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1167s) and improve. And once you can truly

[19:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1167s) and improve. And once you can truly measure in a closed loop system, you can

[19:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1170s) measure in a closed loop system, you can

[19:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1170s) measure in a closed loop system, you can you can do all of these things in a way

[19:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1171s) you can do all of these things in a way

[19:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1171s) you can do all of these things in a way that's just never been possible before.

[19:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1176s) Any other questions?

[19:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1178s) Any other questions?

[19:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1178s) Any other questions? Question in the back.

[19:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1178s) Question in the back.

[19:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1178s) Question in the back. Ycess.

[19:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1188s) Yeah. So, so the question was around uh

[19:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1191s) Yeah. So, so the question was around uh

[19:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1191s) Yeah. So, so the question was around uh how do we do cost per PR? So this is

[19:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1193s) how do we do cost per PR? So this is

[19:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1193s) how do we do cost per PR? So this is data curation

[19:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1194s) data curation

[19:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1194s) data curation the the data curation

[19:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1197s) the the data curation

[19:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1197s) the the data curation um are you so I just want to make sure

[19:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1199s) um are you so I just want to make sure

[19:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1199s) um are you so I just want to make sure we're talking about the same thing. So

[20:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1200s) we're talking about the same thing. So

[20:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1200s) we're talking about the same thing. So you're talking about like when I

[20:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1203s) you're talking about like when I

[20:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1203s) you're talking about like when I for the the benchmarking or for like our

[20:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1206s) for the the benchmarking or for like our

[20:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1206s) for the the benchmarking or for like our general

[20:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1210s) Okay. Yeah. So the way that we we do

[20:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1212s) Okay. Yeah. So the way that we we do

[20:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1212s) Okay. Yeah. So the way that we we do this is like you can either build up a

[20:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1214s) this is like you can either build up a

[20:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1214s) this is like you can either build up a benchmark set by hand which a lot of

[20:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1216s) benchmark set by hand which a lot of

[20:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1216s) benchmark set by hand which a lot of people will do. They'll you know and

[20:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1218s) people will do. They'll you know and

[20:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1218s) people will do. They'll you know and that's kind of what we recommend to

[20:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1219s) that's kind of what we recommend to

[20:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1219s) that's kind of what we recommend to start. uh we will pick like for ours we

[20:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1222s) start. uh we will pick like for ours we

[20:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1222s) start. uh we will pick like for ours we pick like a set of sort of small medium

[20:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1224s) pick like a set of sort of small medium

[20:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1224s) pick like a set of sort of small medium hard tasks whatever you want uh to be

[20:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1227s) hard tasks whatever you want uh to be

[20:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1227s) hard tasks whatever you want uh to be representative or you can actually ask

[20:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1230s) representative or you can actually ask

[20:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1230s) representative or you can actually ask the factory to do it because the factory

[20:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1233s) the factory to do it because the factory

[20:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1233s) the factory to do it because the factory is aware of all the past sessions and so

[20:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1235s) is aware of all the past sessions and so

[20:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1235s) is aware of all the past sessions and so and everything is stored in code so it's

[20:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1237s) and everything is stored in code so it's

[20:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1237s) and everything is stored in code so it's possible to have the factory itself look

[20:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1240s) possible to have the factory itself look

[20:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1240s) possible to have the factory itself look at prior sessions and identify

[20:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1242s) at prior sessions and identify

[20:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1242s) at prior sessions and identify interesting ones for a benchmark set

[20:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1244s) interesting ones for a benchmark set

[20:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1244s) interesting ones for a benchmark set which is pretty cool

[20:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1247s) which is pretty cool

[20:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1247s) which is pretty cool y

[20:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1247s) y

[20:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1247s) y how is correctness evaluated on chart.

[20:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1250s) how is correctness evaluated on chart.

[20:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1250s) how is correctness evaluated on chart. Yeah. So everything in this chart is

[20:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1252s) Yeah. So everything in this chart is

[20:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1252s) Yeah. So everything in this chart is driven by LLM as a judge which we we may

[20:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1256s) driven by LLM as a judge which we we may

[20:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1256s) driven by LLM as a judge which we we may I think another way of doing it is human

[20:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1258s) I think another way of doing it is human

[20:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1258s) I think another way of doing it is human grading which makes total sense. We just

[20:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1259s) grading which makes total sense. We just

[20:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1259s) grading which makes total sense. We just haven't added that yet. So I would

[21:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1261s) haven't added that yet. So I would

[21:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1261s) haven't added that yet. So I would suggest that as well. But basically

[21:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1263s) suggest that as well. But basically

[21:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1263s) suggest that as well. But basically anything that is defined uh in the

[21:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1266s) anything that is defined uh in the

[21:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1266s) anything that is defined uh in the system as a scorer. So you define these

[21:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1269s) system as a scorer. So you define these

[21:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1269s) system as a scorer. So you define these scores which basically grade how well

[21:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1271s) scores which basically grade how well

[21:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1271s) scores which basically grade how well something is doing can be used as a

[21:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1273s) something is doing can be used as a

[21:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1273s) something is doing can be used as a benchmark evaluation criteria.

[21:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1277s) benchmark evaluation criteria.

[21:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1277s) benchmark evaluation criteria. Yeah.

[21:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1278s) Yeah.

[21:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1278s) Yeah. this work platform is this open source

[21:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1281s) this work platform is this open source

[21:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1281s) this work platform is this open source or is this only for enterprise?

[21:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1283s) or is this only for enterprise?

[21:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1283s) or is this only for enterprise? So it is is not open source. It's it's

[21:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1287s) So it is is not open source. It's it's

[21:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1287s) So it is is not open source. It's it's like very flexible and like like the

[21:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1289s) like very flexible and like like the

[21:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1290s) like very flexible and like like the idea is to make it open so you can bring

[21:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1291s) idea is to make it open so you can bring

[21:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1291s) idea is to make it open so you can bring any existing tools into it. It's not

[21:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1294s) any existing tools into it. It's not

[21:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1294s) any existing tools into it. It's not only for enterprise like this is for

[21:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1296s) only for enterprise like this is for

[21:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1296s) only for enterprise like this is for small teams and you can try it out on

[21:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1298s) small teams and you can try it out on

[21:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1298s) small teams and you can try it out on your personal project. It is best for

[21:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1301s) your personal project. It is best for

[21:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1301s) your personal project. It is best for teams though I will say because like if

[21:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1303s) teams though I will say because like if

[21:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1303s) teams though I will say because like if you're um if you're an it's just

[21:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1305s) you're um if you're an it's just

[21:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1305s) you're um if you're an it's just overkill for an individual like you're

[21:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1307s) overkill for an individual like you're

[21:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1307s) overkill for an individual like you're probably gonna want to use like your

[21:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1307s) probably gonna want to use like your

[21:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1308s) probably gonna want to use like your cloud code sub or something but once you

[21:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1310s) cloud code sub or something but once you

[21:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1310s) cloud code sub or something but once you have a team of like maybe 10 or more

[21:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1311s) have a team of like maybe 10 or more

[21:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1311s) have a team of like maybe 10 or more engineers having a system like this is

[21:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1313s) engineers having a system like this is

[21:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1313s) engineers having a system like this is very valuable.