# All Code Is Slop (Except Mine) — Lauren Tan, SpaceXSI

Lauren Tan, SpaceXSI

[Talk page](https://ai.engineer/talks/O_zcc6Xl12g-all-code-is-slop-except-mine) · [Video](https://www.youtube.com/watch?v=O_zcc6Xl12g)

## All Code Is Slop (Except Mine)

Lauren Tan shipped 3,000 pull requests to production in a month. On the Grok Bot team at SpaceXSI, getting comfortable with that pace took six months of work on what agents could copy, what the codebase would allow, and how to tell whether a change worked.

### What would you trust an agent to merge in your sleep?

[0:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1s)

It’s easy to look at somebody else’s code and call it slop. The harder case is your own 3,000th pull request. In the month before this talk, Lauren Tan shipped 3,000 PRs to production, many while she was asleep. It had taken her six months to get comfortable doing that. Once agents could write changes faster than she could read them, the engineering problem was how to keep quality up as the changes kept coming.

Where should that engineering time go? Tan’s inverse trust pyramid gives the most room to decisions the team can make once in code. An API or a refactor can remove a troublesome choice altogether. A lint rule, type checker, compiler or test can catch a choice that the code still permits. Runtime and formal verification then let the team ask whether important properties hold and whether the behavior matches the request. The smallest layer is prose, including rules and skills, for decisions that still require judgment.

The codebase itself is one of the largest inputs an agent sees. Tan remembers coming in to review three PRs her colleagues had written in 2024. By the time of the talk, the comparable number was closer to 300 PRs. At that rate her analogy is gardening. Leave a workaround in the repository and the next agent can reasonably take it for the established pattern. Each copy gives the next agent another example. Cleaning up deprecated code changes what agents learn from the repository, and turning a recurring review correction into a deterministic check gives each new change the same scrutiny.

![An inverted pyramid highlights its two widest tiers, codebase and static analysis. Verification, prose and human review appear faintly below. An arrow labeled “cost” points down toward the tip.](https://ai.engineer/api/talk-essay-media/c094802d6073e6ab31aad6a7406c807a88afd34efea9b445be9196cc365761dc)

Tan puts codebase design and static analysis at the wide top of the pyramid. Good APIs constrain what an agent can do, and lint rules catch recurring mistakes. Cost rises toward human review at the tip.

### Dune moves two recurring review decisions into the framework

[5:00](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=300s)

**Dune** is the client framework the Grok Bot team built for its Electron app. Its value to the team lies in the constraints it carries forward from earlier desktop work. Two choices show how narrowly a constraint can target the code that caused trouble.

- **Where a React effect can live.** React’s `useEffect` has legitimate uses, but agents repeatedly struggled with the judgment it takes to use it well. Dune permits it in selected framework APIs and bans it in product code. The framework authors can still implement behavior that needs an effect. A reviewer of a new product feature no longer has to decide whether this particular effect deserves an exception.

- **Which code the renderer can import.** In Cursor, another Electron app, agents had imported main-thread code into the renderer. The resulting long tasks made the interface stutter. How can an agent tell which side a file belongs on? Dune puts Electron renderer code and Electron main code in two explicitly named top-level directories. The names give the agent context while it writes. A dependency graph analyzer and lint rules forbid imports between the two, so the distinction holds even when the agent misses the cue.

### How a timer workaround became a pattern

[6:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=416s)

Agents sometimes used comments and Markdown to preserve their reasoning a little too well. A small correction could reappear as an absolute rule. An expedient fix could acquire a written justification that made it look intentional. Cursor’s timers illustrate the second failure. Agents reached for `setTimeout`, `requestAnimationFrame` and `requestIdleCallback` to reconcile VS Code’s imperative style with React’s declarative style. Deferring work let an agent patch a local timing problem while leaving the architectural clash in place.

Why would the next agent repeat the fix? Alongside the code was a comment explaining why that particular timer was justified. The growing collection of examples made timers look conventional. At application scale, rapidly creating and destroying so many timers drove major garbage collection, and users felt the pauses as lag. The comments explained individual decisions while the app absorbed their combined cost.

On Grok Bot, the team took the unusually strict position of banning what Tan describes as 99% of comments. Comments went stale, humans skipped them, and even a well-written warning had no way to stop the next change. Grok Bot and Dune put guidance in lint diagnostics, where the agent encounters it precisely when its code violates a rule. The message directs it back to the underlying problem. For each recurring correction, the team looks for a check it can enforce or an API change that removes the mistake altogether. As those decisions moved out of manual review, agents could merge most of their own PRs.

### Give the agent a running app and a way to describe it

[9:40](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=580s)

As the code checks took on more of the review, the team spent more time looking at what a change produced. A video can show an interaction unfolding. A screenshot can capture the resulting state. To produce and inspect that evidence, the agent needs to run the code itself. Grok Bot’s runtime verification skills give it access to debugging tools such as the Chrome DevTools Protocol.

Running the app solves only part of the problem. Suppose the request concerns the thing in the left sidebar. Which feature should the agent exercise? A **feature map** lists the application’s features from the user’s point of view, like a site map. It gives the person making the request and the agent carrying it out a shared vocabulary. Custom command-line tools packaged with the skills let agents reuse the same driver across runs. That saves tokens otherwise spent figuring out how to operate the app and gives the team a driver it can test.

Some questions extend beyond the interaction visible in a recording. The team was experimenting with three ways to check them.

- **Model checking with TLA+.** A notification can involve several devices and platforms. Modeling that flow lets the team check whether a required property holds across the possibilities in the model. One successful run through the app exercises only one such possibility.
- **Algorithm proofs with Lean.** For a custom virtualization library, the team used Lean to reason about algorithm behavior that could produce flicker or jitter. The proof concerns properties of the algorithm. Seeing how the integrated UI behaves is still a job for the running app.
- **Property-based UI tests with Bombadil.** The team also exercised the interface against properties it expected to hold as the UI was used. Tan reports that this had already caught a few UI bugs, alongside bugs found through formal verification. These were early results from complementary techniques.

![A branching map connects features, user paths and evidence that each feature works. Beside it, CLI commands click Settings, take a snapshot and save a screenshot so the agent can check its work.](https://ai.engineer/api/talk-essay-media/0a57dd9b23616685acb085340c05e2583095347dc8f0ddb0fea3b4f5943aa210)

A feature map shows an agent how users reach each feature and what to check. In the Settings example, the agent can use a CLI to click Settings in the running app, take a snapshot, and save a screenshot as proof.png.

### Where the verification tokens go

[11:51](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=711s)

Checking the work takes a substantial share of the token budget. Tan recounts another engineer’s experience with subtle bugs slipping through merges. Six months later, that engineer reported spending 70% of her daily tokens on verification. Tan also spends most of her tokens there. What changed the economics for her team was the cost of writing and maintaining the checks. Techniques that once took too much engineering time to understand and sustain became practical when agents could help with that work.

There is also a product question a passing check can leave open. Did the agent understand which behavior the user wanted? Tan built a statechart skill in **pstack** to visualize the state machine from the user’s point of view. A reviewer can look at which states an action can lead to, then compare that account with screenshots and a video of the implementation. A mistaken assumption about what should happen next is easier to discuss when both people can point to the transition. [pstack plugin source](https://github.com/cursor/plugins/tree/main/pstack)

![A November 5 follow-up highlights the author's estimate that 70% of daily token spend goes toward verifying patches and about 20–30% toward creating them. The earlier post appears below.](https://ai.engineer/api/talk-essay-media/c8944857a9d1ebfbb918fbed19ed034b26a01d0d8bce76258b56c891b28b81de)

Six months after subtle bugs slipped into merged AI-generated patches, Cindy Sridharan credits her new verification process with resolving the problem. She spends 70% of her daily tokens verifying patches and about 20–30% creating them.

### What changed across 11,000 production PRs

[13:32](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=812s)

Tan’s 3,000 PRs were part of a team total of 11,000 shipped to production in one month. The team cut a stable client release about once a day. The core team shipped more, and occasional contributors with less project context could land changes too. In October, about 227 people contributed at least one PR. As volume rose, the share of PRs that were reverted fell and settled at about 0.5%. That figure measures reversions. A bug fixed in a subsequent PR would escape that count.

What did that pace feel like to someone using the client? Compared with launch, Tan reports faster startup, more responsive typing, and faster opening and switching of chats. Those comparisons matter in an Electron app where a single bad import can put a long task on the renderer. They track the interactions the team wanted to protect as more people shipped code.

### What happens when Grok Bot cuts a release

[15:11](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=911s)

Tan gives a specific order for teams tempted to automate the whole operation. When an agent repeats a mistake, start with a lint rule that prevents new occurrences. Then clean up the old examples the next agent might copy. Give the agent a way to verify the resulting behavior. If the ordinary edit-and-check loop keeps producing faulty changes, running more copies of it also multiplies the tokens spent on repairs.

Once the inner loop works, connectors and routines can launch Cursor cloud agents, and projects can coordinate hundreds or even thousands of them. Grok Bot uses a release as one such trigger. Cutting a release automatically starts a new Cursor project. It runs the new build and fuzzes the application by interacting with it as a user would. For each bug it finds, it opens a PR with a fix and automatically merges it.

Automerge is the striking part of that sequence. What made Tan willing to use it was the work already done on the path each fix would take. A repair touching product code inherits Dune’s restrictions. The agent has tools for exercising the change, and reviewers can inspect its behavior. By the time the team automated release testing and repair, most product changes already required little deep human review.

## Key takeaways

- A recurring review comment points to two possible edits. Change the API so the mistake is hard to express, or add a check that catches it and tells the author what to fix.
- Grok Bot’s feature map helps the agent identify what the user meant, and its reusable CLI lets the agent exercise that feature.
- Track what users experience alongside what the team ships. Grok Bot paired its PR volume and revert rate with comparisons of startup, typing and chat navigation against launch.

## Slides

### Platinum sponsors

The slide credits six platinum sponsors. AWS, Devin, Databricks and Exa appear in the top row, with Factory and Sonar centered below.

[0:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=8s) [Under “Platinum sponsors,” the top row shows AWS, Devin, Databricks and Exa. Factory and Sonar appear below.](https://assets.ai.engineer/sites/aiecode2025/sha256/10b5de6a78439b5319faee3001be77c8e58898e8972b659a8bafd627af0c132e.webp)

PLATINUM SPONSORS

aws
Devin
databricks
exa

FACTORY
Sonar

### All code is slop

Lauren Tan gives her own code a pass in a small line of flaming text. She asks who has ever dismissed someone else’s code as slop, then admits she has too.

[0:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=20s) [Large black lettering reads “all code is slop.” The speaker credit sits at lower left, with a cartoon of a smiling person in glasses eating a sweet potato at lower right.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a0511175f589ff0035c20480c115b839d84058b3de6ffdba45ebc4011e7f64d.webp)

all code is slop
(except mine)

poteto (aka lauren tan)
Grok Bot @ SpaceXSI

### I shipped 3,000 PRs to prod last month

Lauren Tan shipped more pull requests to production each month, rising from 68 in March to 3,127 in October. The biggest jump came from July to August, when her monthly count more than doubled from 1,113 to 2,401.

[0:47](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=47s) [Eight bars rise each month from 68 pull requests in March to 3,127 in October. The chart labels the total as 10,732.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8696984c98d0237c807928ca0c80699c0eb37e5a6dbc2d8ae752d0973585f58.webp)

i shipped 3,000 PRs to prod last month

10,732 total
Timeline: mar · apr · may · jun · jul · aug · sep · oct
Bar labels: 68 · 254 · 352 · 685 · 1,113 · 2,401 · 2,732 · 3,127

quality & quantity
@poteto

### Quality & quantity

Lauren Tan argues that coding agents put the output of a “coding machine” within anyone’s reach. The engineer’s job is to figure out how to maintain or improve code quality while shipping faster.

[0:53](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=53s) [“Quality & quantity” appears above the pink subtitle “everyone can be a coding machine.” A footer describes shipping fast without compromising on quality.](https://assets.ai.engineer/sites/aiecode2025/sha256/85f9e34601f12d0cc764831e0a34e6bb8926229185e95d7b462748258dee4fb7.webp)

quality & quantity
everyone can be a coding machine

the new meta is shipping fast without compromising on quality
@poteto

Interface text:
er
mmit
ED BY
eepMind
chine

### Trust and the number of agents

Tan sketches a steep rise in trust over the first few agents. The curve flattens as the count reaches hundreds and approaches the frontier at thousands. She argues that engineers build that trust by investing in the agents’ environment.

[1:23](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=83s) [A curve plots trust vertically against agent count horizontally. It rises steeply near one agent and flattens toward a point labeled “frontier” near thousands of agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/20184973670baf106bf51277659cb04aee45368a96ccf54d61cb50528d5f233f.webp)

help agents write good code

Y-axis: trust
X-axis: number of agents
X-axis ticks: 1, 1 to 5, 5 to 10, 10 to 20, hundreds, thousands
🥔 frontier

low trust
agent writes slop code
human verifies it
issues found

@poteto

### Help agents write good code

In the low-trust loop, the agent writes code and waits for a human to find issues. Tan’s goal is to give the agent an environment where it can check its own code and use what it finds to revise it.

[1:31](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=91s) [Under “low trust” and “most people get stuck here,” arrows form a cycle. An agent writes “slop code,” a human checks it, and issues found lead back to the agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/2e9a6cb8d475de1280a81950423e06b5867f63549c906501c274cee87d8c8f4b.webp)

help agents write good code

low trust
most people get stuck here

agent writes slop code
human verifies it
issues found

@poteto

[1:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=110s) [Two feedback loops show who verifies the agent’s code. In the low-trust loop a human checks it. In the high-trust loop the agent checks it. Both feed issues back into writing code.](https://assets.ai.engineer/sites/aiecode2025/sha256/488e697017e0f5c9e5a5cc943567d46e5799b34ec593c5b1dfb80e461f23250f.webp)

help agents write good code

low trust
most people get stuck here

agent writes slop code
human verifies it
issues found

high trust
you want to be here

agent writes good code
agent verifies it
issues found

@poteto

### The Poteto pyramid — codebase and static analysis

Tan puts codebase design and static analysis at the wide top of the pyramid. Good APIs constrain what an agent can do, and lint rules catch recurring mistakes. Cost rises toward human review at the tip.

[2:07](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=127s) [An inverted pyramid highlights its two widest tiers, codebase and static analysis. Verification, prose and human review appear faintly below. An arrow labeled “cost” points down toward the tip.](https://assets.ai.engineer/sites/aiecode2025/sha256/c094802d6073e6ab31aad6a7406c807a88afd34efea9b445be9196cc365761dc.webp)

how do i trust my agents more?
the poteto pyramid

codebase
(good api design, architecture, refactoring)

static analysis
(lints, types, compilers, tests, ci)

verification
(formal, runtime)

prose
(rules, skills, AGENTS.md)

human review

cost

@poteto

### The Poteto pyramid — verification

In the middle of Tan’s pyramid, formal and runtime verification build on static analysis. They help catch bugs and check whether agent-written code preserves the properties you care about and behaves as you intended.

[2:43](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=163s) [The middle tier of the inverted pyramid reads “verification” with “formal, runtime” beneath it. The other four tier labels are faded. A narrow conference panel with the presenter appears at the left.](https://assets.ai.engineer/sites/aiecode2025/sha256/2bdca151116cb4258d72a5d36fe6e84b34735e7de057bb953a114dff1c2ba1e5.webp)

how do i trust my agents more?
the poteto pyramid

codebase
(good api design, architecture, refactoring)

static analysis
(lints, types, compilers, tests, ci)

verification
(formal, runtime)

prose
(rules, skills, AGENTS.md)

human review

cost

@poteto

Interface text:
[illegible]er
[illegible]mmit
[illegible]ED BY
[illegible]epMind

### Code is memory

The codebase fills the widest tier of Tan’s pyramid. It’s one of an agent’s largest inputs, so when you correct an agent, Tan suggests turning the correction into a deterministic check you can run again.

[2:57](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=177s) [Large black lettering reading “code is memory” spans the center of a faded inverted pyramid.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b2512a75f55f0e8fa5823cbfa9499fd9688b9876ccd335b86f47d1bb15d522f.webp)

code is memory

how do i trust my agents more?
the poteto pyramid

cost ↓

codebase
(good api design, architecture, refactoring)

static analysis
(lints, types, compilers, tests, ci)

verification
(formal, empirical)

prose
(rules, skills, AGENTS.md)

human review

@poteto

### Each copy makes the next copy more likely

Underground stems connect one weed to a row of lookalikes. A workaround spreads through a codebase the same way. Each copy makes it look more like an established pattern, so the next agent is more likely to copy it.

[4:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=264s) [A plant labeled “one workaround” connects underground to a row of similar plants labeled “the pattern.” Three arrows labeled “copy” point along the row.](https://assets.ai.engineer/sites/aiecode2025/sha256/3be2b5f403a08633b50bd2b541a99687c14b80bb3a8ecd973d2c7fee6026a34a.webp)

each copy makes the next copy more likely

copy →    copy →    copy →

one workaround
the pattern

@poteto

### Yes, you should read the code sometimes

A developer points to a line and asks the robot, “Walk me through this code.” Tan’s reason to read along is that the code you keep becomes an example the next agent may copy.

[4:49](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=289s) [A cartoon person points to a line in an open book of code beside a robot holding a pencil. The person asks, “Walk me through this code.”](https://assets.ai.engineer/sites/aiecode2025/sha256/4d92eb763efde2e84bae7ec473f51eebab0ef4d7fcc9fed6838346d2c22ef3cb.webp)

yes, you should read the code sometimes

Walk me through this code.

</>
{   }

@poteto

### Nuance tends to lead to slop

Tan uses React’s useEffect as an example of an API that agents struggle to use well. Her team bans the hook in product code and allows it only in specific parts of Dune’s APIs.

[5:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=324s) [“Nuance tends to lead to slop” appears above the pink subtitle “just design better APIs.” A cartoon in the lower right shows a person and a robot beside a control panel with tangled cables.](https://assets.ai.engineer/sites/aiecode2025/sha256/b819b4edcf5dcac8f62a4e11f55c7cae6b976edd37ce81b0c18897f40ab18d04.webp)

nuance tends to lead to slop
just design better APIs

??
{}
</>

@poteto

### Directory structure is a form of context

Dune’s lint rules block imports between Electron’s renderer and main directories, even for types. Both get common types from shared/. In tibo.dog, the name tick.hot.ts signals that the code runs every tick and must avoid allocations.

[6:28](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=388s) [A crossed-out arrow labeled “not even types” marks the import restriction between electron-renderer/ and electron-main/. Both point to shared/ for shared types. Below, src/sim/tick.hot.ts signals code that runs every tick with no allocations.](https://assets.ai.engineer/sites/aiecode2025/sha256/ceab9ab439a292dbcf979f75d1343f1584e0d101f36bb7acace5462be78ca01b.webp)

directory structure is a form of context

Dune

electron-renderer/
  features/
  client/
  primitives/

not even types

electron-main/
  startup/
  notifications/
  update/

shared/
shared types

tibo.dog (side project)

src/
  sim/
    tick.hot.ts

runs every tick
no allocations

@poteto

### Stop agents from writing unsolicited Markdown

A reader holds up a Markdown file and asks “Source?” The robot replies “I made it up”. Tan has seen agents use documentation to justify a decision or turn a small correction into a rule they treat as permanent.

[6:55](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=415s) [Beside the headline “stop agents from writing unsolicited markdown,” a person holding a .md page asks “Source?” A robot replies “I made it up.”](https://assets.ai.engineer/sites/aiecode2025/sha256/d0822c4adef85294562b8410b6ecb8d27a3a69debf1a8ab4a910019c4fa6246c.webp)

stop agents from writing unsolicited markdown

Source?
I made it up

.md
#

.md
#

.md
#

@poteto

### Quick fixes considered harmful

At Cursor, a zero-delay timer fixed a flicker on Sep 9. The workaround spread to a shared module on Oct 3. By Oct 17, there were 183 timers. Tan says repeatedly creating and destroying timers triggered garbage collection and slowed the app.

[7:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=447s) [On Sep 9, a zero-delay setTimeout fixes a flicker. On Oct 3, the workaround is copied into a shared module. The Oct 17 box reads “183 timers.”](https://assets.ai.engineer/sites/aiecode2025/sha256/bd4de2c97eceb48fa03a6448b52f83a1303014095cd7b9f5001fe86ce67f3329.webp)

quick fixes considered harmful

Sep 9
// release later, in case it comes back
setTimeout(() => release(id), 0)
fixes a flicker
agent: "it would be a larger change"

Oct 3
setTimeout(…, 0)
copied into a shared module

Oct 17
183 timers

@poteto

### Comments are liabilities

The codebase went from 65,824 comments on July 31 to 1,227 by the time of the talk. Tan’s team decided to ban 99% of comments because they go stale and can’t enforce the instructions they leave for the next agent.

[8:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=498s) [A tall red bar shows 65,824 comments on July 31. A short green bar shows 1,227 today.](https://assets.ai.engineer/sites/aiecode2025/sha256/b041b808695666744a7729bb6d68139b05a02eda0fcbaedb706265b5d3515577.webp)

comments are liabilities

jul 31: 65,824
today: 1,227

also see /no-comments in pstack: github.com/cursor/plugins
@poteto

### Whenever you correct your agent

The chart climbs in steps from 7 in July to 167 by November 1. Tan’s team turns each agent correction into a lint rule or redesigns the code to prevent the mistake, giving reviewers fewer things to check.

[9:28](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=568s) [From July to November, a green staircase climbs through 7, 17, 38, 90, 139, 149, 157 and 167. The footer advises adding a lint rule or eliminating the underlying problem.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fa619d07fc6021f37cf128ed482c3a69d2a94b4d819c9f2fd1730d951aef045.webp)

whenever you correct your agent

Step labels: 7, 17, 38, 90, 139, 149, 157, 167
Timeline: jul 1, aug 1, sep 1, oct 1, nov 1

lint against it or categorically eliminate the problem
@poteto

### Runtime verification

A feature map shows an agent how users reach each feature and what to check. In the Settings example, the agent can use a CLI to click Settings in the running app, take a snapshot, and save a screenshot as proof.png.

[10:36](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=636s) [A branching map connects features, user paths and evidence that each feature works. Beside it, CLI commands click Settings, take a snapshot and save a screenshot so the agent can check its work.](https://assets.ai.engineer/sites/aiecode2025/sha256/0a57dd9b23616685acb085340c05e2583095347dc8f0ddb0fea3b4f5943aa210.webp)

runtime verification

feature map
what features exist
how users reach it
what proves it works

+

CLI
$ app aria-click --accessible-name "Settings"
$ app snapshot
$ app screenshot proof.png

drive the real app

agent checks its own work

also see /create-verification-skill in pstack: github.com/cursor/plugins
@poteto

### Formal verification

Tan’s team uses TLA+ to check that a model of cross-device notifications follows its rules in every state. Lean checks each step of a proof about code, which the team uses to check for flicker or jitter in its virtualization library.

[10:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=656s) [Two speech bubbles above a cartoon figure explain how TLA+ checks rules in a state-machine model and Lean checks each logical step in a proof about code.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7ad204bcf9a484ecae68f277aa0f6ab44d6a9f9da38ac15661a7d418c4e1901.webp)

formal verification

TLA+
Model a state machine. Check that its rules hold.

Lean
Write proofs about code. Check every logical step.

TLA+ model
model state machines and check that all states are correctly handled

Lean proof
check implementation for bugs

also see /create-formal-verification-skill in pstack: github.com/cursor/plugins
@poteto

### Subtle bugs in merged AI-generated pull requests

Cindy Sridharan questions how much time AI saved that week. Several AI-generated pull requests merged with subtle bugs, and fixing them took several additional days and extensive manual verification.

[11:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=710s) [A May 15 post highlights and circles the author's report that subtle bugs in several merged AI-generated pull requests took days of work and manual verification to fix.](https://assets.ai.engineer/sites/aiecode2025/sha256/86c7fc6465095d8ee8aaa8c4ff34dd7cf6be64d6b0903e8b3182c8ae13528a1f.webp)

Cindy Sridharan
@copyconstruct

Might be anecdotal, but just this week multiple AI generated PRs with subtle bugs got merged that required several additional days and a lot of manual verification to fix.

“Velocity” isn’t going up when you consider how much effort was spent fixing...

May 15, 2026

@poteto

Interface text:
er
mmit
ED BY
eepMind
ent

### Daily token use for patch verification

Six months after subtle bugs slipped into merged AI-generated patches, Cindy Sridharan credits her new verification process with resolving the problem. She spends 70% of her daily tokens verifying patches and about 20–30% creating them.

[12:05](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=725s) [A November 5 follow-up highlights the author's estimate that 70% of daily token spend goes toward verifying patches and about 20–30% toward creating them. The earlier post appears below.](https://assets.ai.engineer/sites/aiecode2025/sha256/c8944857a9d1ebfbb918fbed19ed034b26a01d0d8bce76258b56c891b28b81de.webp)

Cindy Sridharan
@copyconstruct

Revisiting my tweets from 6m ago is funny, because I don’t run into any of these problems now at all.

What was missing in my setup 6 months ago was the verification process I now use.

70% of my daily token spend goes into verifying patches, and only ~20-30% into creating them.

Cindy Sridharan @copyconstruct · May 15

Might be anecdotal, but just this week multiple AI generated PRs with subtle bugs got merged that required several additional days and a lot of manual verification to fix.

“Velocity” isn’t going up when you consider how much effort was spent fixing...

Nov 5, 2026

@poteto

### Testing practices for trusting agent-generated code

Cindy Sridharan says she needs a mix of familiar tests to trust agent-generated code. Her chart follows a change from unit tests before production through load tests at deployment, canarying at release and real user monitoring afterward.

[12:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=740s) [A quoted post explains why its author needs a mix of testing techniques to trust agent-generated code. Beside it, a chart groups those techniques by pre-production, deployment, release and post-release.](https://assets.ai.engineer/sites/aiecode2025/sha256/b09d695cb84b0963bff029f22f2d1543e128a6f4b4fe77a8ac7cdfb3b6d1ec0c.webp)

Cindy Sridharan
@copyconstruct

A mix of the techniques shown here, in short.

This has been my dream for nearly a decade now, and with agents I’m finally able to see this (partially) come true.

But turns out this is what I necessarily require to trust agent generated code. It’s not optional anymore, imo.

Nov 5, 2026

PRE-PRODUCTION
- UNIT TESTS
- FUNCTIONAL TESTS
- COMPONENT TESTS
- FUZZ TESTS
- PROPERTY BASED TESTS
- FORMAL VERIFICATION
- STATIC ANALYSIS
- MUTATION TESTS
- COVERAGE TESTS
- BENCHMARK TESTS
- CONTRACT TESTS
- LINT TESTS
- ACCEPTANCE TESTS
- REGRESSION TESTS
- SMOKE TESTS
- USABILITY TESTS
- UI/UX TESTS
- PENETRATION TESTS
- THREAT MODELING
- PERFORMANCE TESTS

TESTING IN PRODUCTION

DEPLOY
- INTEGRATION TESTS
- TAP COMPARE
- LOAD TESTS
- SHADOWING
- CONFIG TESTS
- SOAK TESTS

RELEASE
- CANARYING
- MONITORING
- TRAFFIC SHAPING
- FEATURE FLAGGING
- EXCEPTION TRACKING

POST RELEASE
- TEEING
- PROFILING
- LOGS/EVENTS
- CHAOS TESTING
- MONITORING
- A/B TESTS
- TRACING
- DYNAMIC EXPLORATION
- REAL USER MONITORING
- AUDITING
- ONCALL EXPERIENCE

chart from this tweet by Cindy Sridharan (@copyconstruct), redrawn

@poteto

Interface text:
[illegible]er
[illegible]mmit
[illegible]ED BY
[illegible]epMind
MONITORIN[illegible]
- AUDITING
- ONCALL EXPERIENCE

### Testing techniques before and during production

Verification runs from formal checks before deployment to canarying at release and tracing afterward. Tan argues that agents cut the cost of writing and maintaining tests, making more thorough testing practical.

[12:29](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=749s) [Four columns organize testing practices by stage. Examples include unit tests and formal verification before production, load tests at deployment, canarying at release, and tracing and real user monitoring after release.](https://assets.ai.engineer/sites/aiecode2025/sha256/96e21201de585964844a6d11ec6d95e450c1c21f8319e71dfe8a6a7a50473cec.webp)

PRE-PRODUCTION
- UNIT TESTS
- FUNCTIONAL TESTS
- COMPONENT TESTS
- FUZZ TESTS
- PROPERTY BASED TESTS
- FORMAL VERIFICATION
- STATIC ANALYSIS
- MUTATION TESTS
- COVERAGE TESTS
- BENCHMARK TESTS
- CONTRACT TESTS
- LINT TESTS
- ACCEPTANCE TESTS
- REGRESSION TESTS
- SMOKE TESTS
- USABILITY TESTS
- UI/UX TESTS
- PENETRATION TESTS
- THREAT MODELING
- PERFORMANCE TESTS

TESTING IN PRODUCTION

DEPLOY
- INTEGRATION TESTS
- TAP COMPARE
- LOAD TESTS
- SHADOWING
- CONFIG TESTS
- SOAK TESTS

RELEASE
- CANARYING
- MONITORING
- TRAFFIC SHAPING
- FEATURE FLAGGING
- EXCEPTION TRACKING

POST RELEASE
- TEEING
- PROFILING
- LOGS/EVENTS
- CHAOS TESTING
- MONITORING
- A/B TESTS
- TRACING
- DYNAMIC EXPLORATION
- REAL USER MONITORING
- AUDITING
- ONCALL EXPERIENCE

Cindy Sridharan
@copyconstruct

A mix [illegible]
in sho[illegible]

This h[illegible]
decad[illegible]
finally[illegible]
come t[illegible]

But tu[illegible]
necess[illegible]
gener[illegible]
anymo[illegible]

Nov 5, [illegible]

@poteto

### ok so what

A cartoon rider raises a fist on a rocket as Tan turns from how her team reviews changes to what the team gets back for that investment.

[13:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=798s) [The words “ok so what” appear in large black lettering on white. A cartoon rider raises a fist on a rocket in the lower right.](https://assets.ai.engineer/sites/aiecode2025/sha256/822a31b18d5b1e915f277230c03e55bf44b5ae3c8a2382cf84524529863a1ae3.webp)

ok so what

@poteto

### Grok bot team merged 10,926 pull requests in October

The Grok bot team merged more pull requests each month, rising from 809 in June to 10,926 in October. October alone accounted for nearly half of the 23,370 merges over those five months.

[13:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=810s) [Monthly merged pull requests rise from 809 in June to 1,784 in July, 4,203 in August, 5,648 in September and 10,926 in October. The chart gives a five-month total of 23,370.](https://assets.ai.engineer/sites/aiecode2025/sha256/afda811b8e19a7af2a3e301bbd551fa378a1eec832632e281c56d4100e25990e.webp)

grok bot team merged 11k PRs in oct

23,370 total

jun: 809
jul: 1,784
aug: 4,203
sep: 5,648
oct: 10,926

@poteto

## Full transcript

[0:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=12s) All right. Um, so again, I'm my name is

[0:16](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=16s) Potato. Uh, I work on Graphbot at

[0:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=19s) SpaceXsi.

[0:21](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=21s) And can I ask you a quick show of hands?

[0:23](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=23s) How many of you have ever looked at code

[0:26](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=26s) that somebody else wrote and you thought

[0:28](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=28s) that's all slop?

[0:31](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=31s) Well, I have too.

[0:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=35s) But today, I'm going to convince you

[0:36](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=36s) that you can turn that slop into gold.

[0:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=42s) Uh, and so I did something crazy last

[0:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=44s) month and I shipped 3,000 ploy requests

[0:47](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=47s) to production. Now, before agents, we

[0:49](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=49s) never really cared about volume. uh

[0:52](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=52s) except for the outliers.

[0:54](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=54s) And at my previous company, we call

[0:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=56s) these people coding machines. People

[0:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=58s) like Brett Taylor, who famously rewrote

[1:00](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=60s) all of Google Maps uh in a weekend, just

[1:03](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=63s) an insanely productive engineer. Uh and

[1:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=66s) well, with agents, anyone can now be a

[1:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=68s) coding machine. And my thesis is that

[1:11](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=71s) the job of a software engineer, the meta

[1:13](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=73s) of being a software engineer is figuring

[1:15](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=75s) out how to maintain or even improve the

[1:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=78s) quality of your code while massively

[1:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=80s) increasing your productivity.

[1:23](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=83s) To get there, some people like to call

[1:25](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=85s) this a software factory. Uh, but I like

[1:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=87s) to keep it simple and I think of it as

[1:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=90s) all about trust.

[1:32](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=92s) not blind trust, but investing in your

[1:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=95s) agents environment so you can trust that

[1:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=97s) your agents are going to do the right

[1:39](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=99s) thing. Now, it takes quite a lot of

[1:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=102s) effort to get there. Um, and it took me

[1:45](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=105s) 6 months to get to a point where I could

[1:47](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=107s) ship thousands of pull requests, a lot

[1:49](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=109s) of them in my sleep, but it's certainly

[1:52](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=112s) possible and it will take you much less

[1:54](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=114s) time than I did. So, let's start by

[1:57](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=117s) talking about how we move from a low

[1:59](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=119s) trust environment to a high trust one.

[2:03](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=123s) So this inverse pyramid is how I think

[2:05](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=125s) about where to spend time on your agents

[2:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=128s) environment.

[2:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=130s) The first two layers are deterministic.

[2:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=132s) So using your code itself. So things

[2:14](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=134s) like good API design architecture

[2:17](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=137s) refactoring or using lit rules, type

[2:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=140s) systems, compilers or you know different

[2:22](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=142s) kinds of testing. And you can use both

[2:25](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=145s) of these layers together to constrain

[2:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=147s) what agents can do and force them to do

[2:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=150s) the right thing by default.

[2:32](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=152s) Verification, both runtime and formal,

[2:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=155s) are great tools to layer on to find bugs

[2:38](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=158s) and check properties and the invariance

[2:40](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=160s) in your system that you want to hold

[2:41](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=161s) true and also help you check whether or

[2:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=164s) not the agent understood your goals and

[2:46](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=166s) your intent.

[2:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=168s) And then finally, you want to layer with

[2:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=170s) a little bit of pros. So things like

[2:52](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=172s) rules, skills, agents.mmd, things that

[2:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=176s) require judgment.

[2:59](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=179s) But let's start with your codebase.

[3:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=181s) Agents generate their outputs based on

[3:03](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=183s) their inputs. And so the the quality of

[3:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=186s) the input really matters. And it just so

[3:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=188s) happens that your codebase is one of the

[3:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=190s) largest inputs. So whenever you're

[3:13](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=193s) correcting your agent, think about how

[3:15](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=195s) you can make that a deterministic check

[3:17](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=197s) instead of adding another line to your

[3:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=199s) rules or your agents.mmd because it's

[3:22](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=202s) far more reproducible when it's in code

[3:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=204s) and it lets you trust it more.

[3:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=207s) And the analogy that I like to use is

[3:29](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=209s) that your codebase is kind of like a

[3:31](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=211s) garden. You know, code used to arrive a

[3:34](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=214s) few pull requests at a time. And uh I

[3:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=217s) remember going into work in 2024 and

[3:40](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=220s) reviewing the three pull requests that

[3:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=222s) my colleagues wrote colleagues wrote and

[3:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=224s) today that number is more like 300. So

[3:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=228s) uh you know every engineer is now a team

[3:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=230s) of hundreds or even thousands of agents

[3:53](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=233s) and code is arriving so rapidly that it

[3:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=236s) almost feels like a continuous stream

[3:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=238s) and it almost feels organic like a

[4:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=241s) living thing.

[4:03](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=243s) And in this analogy, your codebase, the

[4:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=246s) constraints that you impose, the

[4:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=248s) guardrails that you put up are the

[4:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=250s) concrete foundations that the new code

[4:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=252s) that your agents write grow around just

[4:15](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=255s) like uh the building in this picture.

[4:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=258s) And so in many ways, the job of being a

[4:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=259s) software engineer is kind of becoming

[4:21](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=261s) like gardening

[4:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=264s) because code is the biggest input for

[4:26](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=266s) agents. It means that any workarounds,

[4:29](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=269s) um, anti-atterns, deprecated code, tech,

[4:33](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=273s) it's now likely to be just copied by the

[4:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=275s) next agent since it looks like the

[4:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=277s) established pattern in your codebase.

[4:40](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=280s) And uh, it's like you have some weeds or

[4:43](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=283s) pests growing in your garden and then

[4:45](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=285s) they spread everywhere before you know

[4:47](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=287s) it.

[4:51](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=291s) So you want to make your codebase as

[4:53](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=293s) pristine as possible so that agents are

[4:55](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=295s) always copying the good stuff. All

[4:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=298s) right, let's look at some real examples.

[5:00](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=300s) Uh so Dune is a client framework that we

[5:03](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=303s) created for Grabbot to implement this

[5:05](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=305s) idea for organic architecture.

[5:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=308s) It's not really that interesting

[5:09](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=309s) technically. Uh you can kind of think of

[5:11](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=311s) Dune as um kind of like Nex.js for

[5:14](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=314s) Electron apps. But the interesting part

[5:16](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=316s) of Dune is that it has a lot of

[5:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=318s) constraints and lit rules that we've

[5:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=320s) encoded uh a lot of the lessons that

[5:22](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=322s) we've learned building desktop

[5:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=324s) applications.

[5:26](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=326s) And one such lesson is that anything

[5:28](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=328s) that requires nuance to use well often

[5:31](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=331s) leads to slop. Uh you know your classic

[5:34](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=334s) example is uh you know use effect in

[5:36](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=336s) React, right? It has its use cases uh

[5:39](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=339s) but it's often misused. You know some

[5:41](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=341s) people joke that you need a PhD to use

[5:43](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=343s) use effect. Um and in in other words, it

[5:46](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=346s) really requires a lot of nuance to use B

[5:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=350s) and agents are not good at using it. Um

[5:53](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=353s) so our solution has been to just ban it

[5:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=356s) in product code and it's really only

[5:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=358s) allowed in some places inside of Dune

[6:00](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=360s) APIs

[6:02](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=362s) and effectively this problem has now

[6:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=364s) been categorically eliminated and we

[6:07](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=367s) never need to catch that during code

[6:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=368s) review again.

[6:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=370s) Another interesting pattern that we use

[6:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=372s) is context through directory structure.

[6:15](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=375s) Uh so one anti-attern that we saw with

[6:17](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=377s) cursor which is an electron app is that

[6:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=380s) agents would often import main thread

[6:22](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=382s) code into the renderer thread which

[6:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=384s) would cause long tasks and jank.

[6:28](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=388s) So in uh grapot and in dunes we've made

[6:31](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=391s) this very very conventional. Uh we have

[6:33](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=393s) these two top level directories

[6:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=395s) literally called electron renderer and

[6:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=397s) electron main and imports between them

[6:40](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=400s) are forbidden or banned by our

[6:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=402s) dependency graph analyzer and some lint

[6:45](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=405s) rules. And again you know these are

[6:47](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=407s) simple lint rules to set up but it means

[6:49](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=409s) one less thing that we need to review

[6:51](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=411s) and think about uh you know during code

[6:54](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=414s) review.

[6:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=416s) And then one more surprising thing that

[6:59](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=419s) I've seen is agents using markdown. So

[7:02](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=422s) things like documentation, uh, code

[7:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=424s) comments, rules or skills, and they use

[7:07](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=427s) this as a place to write down their

[7:09](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=429s) justifications for not doing the right

[7:11](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=431s) thing, or I don't know if you've ever

[7:13](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=433s) seen this, but they'll over rotate on

[7:15](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=435s) some small correction that you made, and

[7:17](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=437s) they treat it like a hard rule that can

[7:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=439s) never be broken ever again.

[7:22](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=442s) And here's another example from cursor.

[7:25](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=445s) So agents really loved using imperative

[7:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=447s) timers. So things like set timeout,

[7:29](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=449s) request animation frame, request idle

[7:31](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=451s) call back. And they were using these as

[7:33](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=453s) quick fixes because they were really

[7:36](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=456s) patching over an architecture clash

[7:38](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=458s) between uh VS Code's imperative style

[7:41](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=461s) and React's declarative style.

[7:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=464s) So our choice of architecture actually

[7:46](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=466s) made both the client and the agent

[7:49](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=469s) performance much worse. And if you've

[7:52](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=472s) ever done any performance work in

[7:53](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=473s) JavaScript, you'll know that when you

[7:54](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=474s) have tons of timers being rapidly

[7:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=476s) created and destroyed causes a lot of uh

[7:59](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=479s) major garbage collection, which causes a

[8:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=481s) lot of lag and jen. Uh but the worst

[8:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=484s) part is that agents wrote a lot of code

[8:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=486s) comments like you can see in red here

[8:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=488s) explaining why this quick fix was

[8:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=490s) justified and before you know it, we had

[8:14](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=494s) timers spreading everywhere and the you

[8:16](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=496s) know the performance of the app really

[8:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=498s) suffered.

[8:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=500s) Now in graphbot uh we learned our lesson

[8:23](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=503s) there and we decided to just ban 99% of

[8:25](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=505s) comments uh because they go stale easily

[8:29](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=509s) uh you know humans just really don't

[8:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=510s) bother reading the comment slop anyway

[8:33](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=513s) and again it gives an agent the an

[8:36](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=516s) escape hatch to justify not doing the

[8:38](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=518s) right thing because it's hard. Um and

[8:41](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=521s) the worst part about comments though is

[8:43](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=523s) that they're not enforcable. So what

[8:45](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=525s) seems well-intentioned, you know, like

[8:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=528s) I'm going to tell the the next agent not

[8:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=530s) to do something, never ends up actually

[8:52](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=532s) being anything that you can trust. So in

[8:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=536s) Grogbot and Dune and in Dune, uh we now

[8:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=538s) have lint rules that give uh good and

[9:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=541s) very helpful error messages telling the

[9:03](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=543s) agent to fix the root cause, the real

[9:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=546s) problem, uh rather than just patch

[9:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=548s) around it.

[9:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=550s) And so our general principle is that

[9:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=552s) every time we correct our agents, it

[9:13](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=553s) turns into one of these deterministic

[9:15](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=555s) checks or we think about how to

[9:17](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=557s) categorically eliminate that problem in

[9:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=559s) the code through better API design,

[9:21](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=561s) through better architecture. And if you

[9:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=564s) do this enough times, you'll find that

[9:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=567s) you have just less things to look out

[9:29](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=569s) for in code review. And at some point

[9:31](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=571s) you realize there's there's nothing left

[9:33](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=573s) to review and you can just let agents

[9:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=575s) merge their most of their own pull

[9:38](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=578s) requests.

[9:40](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=580s) All right. Uh moving on. Verification is

[9:43](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=583s) also a really big part of how we make

[9:45](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=585s) our agents more autonomous and also

[9:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=588s) increase our confidence and trust in

[9:49](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=589s) their work. And our code review process

[9:52](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=592s) has now really shrunk into looking at

[9:54](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=594s) verification artifacts instead. So

[9:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=596s) things like videos, screenshots,

[9:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=598s) diagrams, formal verification.

[10:02](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=602s) Um, and the most basic example of this

[10:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=604s) is runtime verification. Uh, which is

[10:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=606s) just a fancy way of saying giving your

[10:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=608s) agent a way to run the code for itself

[10:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=610s) and use debugging tools that you know,

[10:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=612s) human developers use as well, like the

[10:14](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=614s) Chrome DevTools protocol. Um, in our

[10:17](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=617s) runtime verification skills, we have

[10:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=619s) this thing that we call a feature map.

[10:21](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=621s) And that's basically kind of like a site

[10:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=624s) map of all of the features that are

[10:25](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=625s) available in your application from the

[10:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=627s) user's point of view. And so with this,

[10:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=630s) agents don't need to guess when you say,

[10:32](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=632s) you know, the thingy in the left

[10:33](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=633s) sidebar. It gives you some shared

[10:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=635s) vocabulary.

[10:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=637s) And we also make use of custom CLIs

[10:39](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=639s) inside these skills so that uh agents

[10:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=642s) can reuse the the CLI and the driver

[10:45](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=645s) every time which cuts down on token

[10:46](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=646s) usage and is also testable.

[10:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=650s) Um and then recently we've also started

[10:52](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=652s) looking at formal verification. Uh so we

[10:55](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=655s) now use TA plus to do some model

[10:57](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=657s) checking of major user flows such as uh

[10:59](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=659s) notifications across multiple devices

[11:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=661s) and platforms. Uh we use a lean as well

[11:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=664s) for proving certain algorithms. Uh for

[11:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=666s) example, we have our own we wrote our

[11:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=668s) own custom virtualization library and we

[11:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=670s) use lean there to check that we don't

[11:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=672s) have you know flickering or jittering

[11:14](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=674s) problems. Um and we're still

[11:16](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=676s) experimenting with this approach.

[11:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=678s) Overall it seems very promising. Uh

[11:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=680s) we've already found quite a few bugs uh

[11:22](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=682s) with formal verification.

[11:25](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=685s) Um but it's certainly not a silver

[11:26](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=686s) bullet and we're treating it as another

[11:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=690s) tool, another layer in our agents uh

[11:33](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=693s) environment.

[11:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=695s) Uh and then this is a very fast video

[11:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=697s) but uh another thing that we just

[11:39](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=699s) started looking at is property based

[11:41](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=701s) testing as well for UIs. Uh so we use

[11:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=704s) this library called Bombadil. Um and

[11:46](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=706s) this has also caught a few UI bugs for

[11:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=708s) us already.

[11:51](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=711s) Okay. Okay, so zooming out, I saw this

[11:53](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=713s) tweet from Cindy and six months ago she

[11:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=716s) she said that she was or she was talking

[11:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=718s) about not being able to trust her agents

[12:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=721s) because of subtle bugs that were getting

[12:03](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=723s) merged.

[12:05](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=725s) And really what was missing for her in

[12:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=726s) being able to trust her agents was the

[12:08](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=728s) verification process. And now 6 months

[12:11](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=731s) later, she says that 70% of her daily

[12:13](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=733s) tokens go into verification. And I'd say

[12:16](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=736s) that that tracks with what I see as

[12:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=738s) well, that most of my token usage goes

[12:20](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=740s) to verification.

[12:22](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=742s) And if you really look closely at our

[12:23](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=743s) process, none of these things here are

[12:26](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=746s) new or novel techniques that we've never

[12:28](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=748s) had before.

[12:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=750s) These are just good software engineering

[12:32](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=752s) tools, but ones that might have been

[12:34](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=754s) prohibitively expensive to use before as

[12:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=757s) they were hard to understand or write or

[12:39](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=759s) maintain. But agents drastically cut

[12:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=762s) this cost down. And really, we're

[12:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=764s) entering a golden age of software

[12:46](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=766s) testing and verification because agents

[12:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=768s) let us test everything to a much greater

[12:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=770s) degree than used to be economical.

[12:54](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=774s) And then formal verification can also

[12:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=776s) help you better reason about how your

[12:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=778s) product should work and whether your

[13:00](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=780s) agents really understand your intent and

[13:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=781s) goals. Uh I made this new skill called

[13:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=784s) state chart uh in my plug-in PSAC potato

[13:07](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=787s) stack uh which helps you visualize state

[13:09](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=789s) machines from uh from the user's point

[13:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=792s) of view. Uh, and it's I find this much

[13:14](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=794s) easier to review alongside the videos

[13:16](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=796s) and screenshots that we provide. Um, and

[13:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=799s) so that's been great. Sorry for the

[13:21](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=801s) flashbang. Uh, okay. Now, let's look at

[13:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=804s) what the promised land looks like. What

[13:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=807s) do you get in return for all of this,

[13:29](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=809s) uh, you know, investment?

[13:32](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=812s) So, on the Graphbot team, our velocity

[13:34](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=814s) is uh, taking off and rapidly

[13:36](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=816s) increasing. And last month, our team

[13:39](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=819s) shipped a ridiculous 11,000 pull

[13:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=822s) requests to production, which is a wild

[13:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=824s) number. And our pace of releasing the

[13:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=828s) client has also increased. And we cut a

[13:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=830s) new stable version about once a day.

[13:54](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=834s) And then the core team is also shipping

[13:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=836s) much more. And we also have uh a lot

[13:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=838s) more casual or drive by contributors who

[14:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=841s) don't always have a lot of context uh

[14:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=844s) but they're also contributing. Um, and

[14:06](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=846s) in October we had about 227 people who

[14:10](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=850s) contributed at least one pull request.

[14:14](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=854s) So despite this velocity accelerating,

[14:16](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=856s) you might be thinking to yourself like,

[14:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=858s) isn't this going to be chaos? You know,

[14:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=859s) isn't this going to be a bunch of flop?

[14:21](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=861s) But actually, the number of uh PRs that

[14:23](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=863s) get reverted has actually gotten lower

[14:25](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=865s) on a relative basis and it's steady at

[14:27](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=867s) about half a percent.

[14:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=870s) But I think the really interesting thing

[14:32](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=872s) about this shipping pace and this volume

[14:34](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=874s) of pull requests is that the performance

[14:38](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=878s) of Grabbot has stayed not just stable

[14:40](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=880s) but has improved on multiple dimensions

[14:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=882s) compared to when we launched. So startup

[14:45](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=885s) typing, opening new chats, switching

[14:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=888s) chats, all of this has gotten faster and

[14:51](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=891s) keep in mind Grothbot is still an

[14:53](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=893s) Electron app for now. Uh so there is a

[14:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=896s) ceiling, right? But it's already much

[14:58](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=898s) faster than many other agent UIs out

[15:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=901s) there. And for an Electron app with this

[15:05](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=905s) high velocity and this many people

[15:07](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=907s) contributing, I'd say that's pretty

[15:09](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=909s) good.

[15:11](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=911s) So before you jump into building out a

[15:13](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=913s) software factory and you know automating

[15:15](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=915s) everything that you can, whenever you

[15:17](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=917s) correct your agent, try and do it by

[15:19](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=919s) improving your codebase first. you know,

[15:21](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=921s) start with a lint rule to prevent new

[15:23](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=923s) occurrences of anti-atterns and then

[15:25](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=925s) clean up those uh old patterns as well

[15:28](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=928s) and then make sure that your

[15:29](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=929s) verification story is really strong.

[15:32](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=932s) A lot of this work is a prerequisite to

[15:35](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=935s) automation anyway. And if your inner

[15:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=937s) coding loop uh produces slop, then

[15:40](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=940s) automating that loop just produces more

[15:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=942s) slop and you need to throw more and more

[15:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=944s) and more tokens at the problem, which we

[15:46](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=946s) all know is very expensive.

[15:49](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=949s) And once you've made these investments

[15:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=950s) into you know code into your codebase

[15:53](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=953s) static analysis runtime informal

[15:55](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=955s) verification you'll start to find that

[15:57](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=957s) many things become possible with these

[16:00](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=960s) foundations and that you can really

[16:02](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=962s) easily build your own factory on top of

[16:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=964s) these foundations

[16:07](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=967s) and when you're ready is a great way to

[16:09](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=969s) build that uh outer loop. We have tons

[16:11](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=971s) of connectors, routines that can kick

[16:13](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=973s) off cursor cloud agents and projects uh

[16:16](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=976s) which let you orchestrate thousands of u

[16:18](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=978s) hundreds or even thousands of agents.

[16:21](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=981s) And as a quick example, you know,

[16:22](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=982s) whenever we cut a new release now,

[16:24](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=984s) Grockbot will automatically spin up a

[16:26](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=986s) new cursor project for us that will fuzz

[16:28](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=988s) the entire application, the entire

[16:30](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=990s) build, and it will find bugs uh by

[16:33](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=993s) running the app and using it like a real

[16:34](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=994s) user. And then it's going to open fixed

[16:37](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=997s) PRs for each of those bugs that it finds

[16:39](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=999s) and it's going to automerge it as uh

[16:42](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1002s) crazy as that sounds. And that's because

[16:44](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1004s) our code base is good enough that pull

[16:46](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1006s) requests tend to just be good by

[16:48](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1008s) default. And most product code that we

[16:50](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1010s) land no longer really requires deep

[16:52](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1012s) human review.

[16:56](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1016s) So when you have solid foundations and

[16:59](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1019s) good organic architecture, you can turn

[17:01](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1021s) slop into gold and you can ship with

[17:04](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1024s) high quality uh high velocity without

[17:07](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1027s) compromising on quality. My name is

[17:09](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1029s) Potato, also known as Lauren. Thanks you

[17:12](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1032s) again for listening to my talk.

[17:14](https://www.youtube.com/watch?v=O_zcc6Xl12g&t=1034s) [applause]
