All Code Is Slop (Except Mine) — Lauren Tan, SpaceXSI

Loading slides…

Read the talk

All Code Is Slop (Except Mine)

Lauren Tan shipped 3,000 pull requests to production in a month. On the Grok Bot team at SpaceXSI, getting comfortable with that pace took six months of work on what agents could copy, what the codebase would allow, and how to tell whether a change worked.

From a talk by Lauren Tan

At a glance

Ideas worth remembering

  • A recurring review comment points to two possible edits. Change the API so the mistake is hard to express, or add a check that catches it and tells the author what to fix.

  • Grok Bot’s feature map helps the agent identify what the user meant, and its reusable CLI lets the agent exercise that feature.

  • Track what users experience alongside what the team ships. Grok Bot paired its PR volume and revert rate with comparisons of startup, typing and chat navigation against launch.

What would you trust an agent to merge in your sleep?

An inverted pyramid highlights its two widest tiers, codebase and static analysis. Verification, prose and human review appear faintly below. An arrow labeled “cost” points down toward the tip.Open full source frame
Tan puts codebase design and static analysis at the wide top of the pyramid. Good APIs constrain what an agent can do, and lint rules catch recurring mistakes. Cost rises toward human review at the tip.

It’s easy to look at somebody else’s code and call it slop. The harder case is your own 3,000th pull request. In the month before this talk, Lauren Tan shipped 3,000 PRs to production, many while she was asleep. It had taken her six months to get comfortable doing that. Once agents could write changes faster than she could read them, the engineering problem was how to keep quality up as the changes kept coming.

Where should that engineering time go? Tan’s inverse trust pyramid gives the most room to decisions the team can make once in code. An API or a refactor can remove a troublesome choice altogether. A lint rule, type checker, compiler or test can catch a choice that the code still permits. Runtime and formal verification then let the team ask whether important properties hold and whether the behavior matches the request. The smallest layer is prose, including rules and skills, for decisions that still require judgment.

The codebase itself is one of the largest inputs an agent sees. Tan remembers coming in to review three PRs her colleagues had written in 2024. By the time of the talk, the comparable number was closer to 300 PRs. At that rate her analogy is gardening. Leave a workaround in the repository and the next agent can reasonably take it for the established pattern. Each copy gives the next agent another example. Cleaning up deprecated code changes what agents learn from the repository, and turning a recurring review correction into a deterministic check gives each new change the same scrutiny.

0:230:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Dune moves two recurring review decisions into the framework

Dune is the client framework the Grok Bot team built for its Electron app. Its value to the team lies in the constraints it carries forward from earlier desktop work. Two choices show how narrowly a constraint can target the code that caused trouble.

  • Where a React effect can live. React’s useEffect has legitimate uses, but agents repeatedly struggled with the judgment it takes to use it well. Dune permits it in selected framework APIs and bans it in product code. The framework authors can still implement behavior that needs an effect. A reviewer of a new product feature no longer has to decide whether this particular effect deserves an exception.
  • Which code the renderer can import. In Cursor, another Electron app, agents had imported main-thread code into the renderer. The resulting long tasks made the interface stutter. How can an agent tell which side a file belongs on? Dune puts Electron renderer code and Electron main code in two explicitly named top-level directories. The names give the agent context while it writes. A dependency graph analyzer and lint rules forbid imports between the two, so the distinction holds even when the agent misses the cue.
5:005:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:00 · section reference included

How a timer workaround became a pattern

Agents sometimes used comments and Markdown to preserve their reasoning a little too well. A small correction could reappear as an absolute rule. An expedient fix could acquire a written justification that made it look intentional. Cursor’s timers illustrate the second failure. Agents reached for setTimeout, requestAnimationFrame and requestIdleCallback to reconcile VS Code’s imperative style with React’s declarative style. Deferring work let an agent patch a local timing problem while leaving the architectural clash in place.

Why would the next agent repeat the fix? Alongside the code was a comment explaining why that particular timer was justified. The growing collection of examples made timers look conventional. At application scale, rapidly creating and destroying so many timers drove major garbage collection, and users felt the pauses as lag. The comments explained individual decisions while the app absorbed their combined cost.

On Grok Bot, the team took the unusually strict position of banning what Tan describes as 99% of comments. Comments went stale, humans skipped them, and even a well-written warning had no way to stop the next change. Grok Bot and Dune put guidance in lint diagnostics, where the agent encounters it precisely when its code violates a rule. The message directs it back to the underlying problem. For each recurring correction, the team looks for a check it can enforce or an API change that removes the mistake altogether. As those decisions moved out of manual review, agents could merge most of their own PRs.

6:597:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:56 · section reference included

Give the agent a running app and a way to describe it

A branching map connects features, user paths and evidence that each feature works. Beside it, CLI commands click Settings, take a snapshot and save a screenshot so the agent can check its work.Open full source frame
A feature map shows an agent how users reach each feature and what to check. In the Settings example, the agent can use a CLI to click Settings in the running app, take a snapshot, and save a screenshot as proof.png.

As the code checks took on more of the review, the team spent more time looking at what a change produced. A video can show an interaction unfolding. A screenshot can capture the resulting state. To produce and inspect that evidence, the agent needs to run the code itself. Grok Bot’s runtime verification skills give it access to debugging tools such as the Chrome DevTools Protocol.

Running the app solves only part of the problem. Suppose the request concerns the thing in the left sidebar. Which feature should the agent exercise? A feature map lists the application’s features from the user’s point of view, like a site map. It gives the person making the request and the agent carrying it out a shared vocabulary. Custom command-line tools packaged with the skills let agents reuse the same driver across runs. That saves tokens otherwise spent figuring out how to operate the app and gives the team a driver it can test.

Some questions extend beyond the interaction visible in a recording. The team was experimenting with three ways to check them.

  • Model checking with TLA+. A notification can involve several devices and platforms. Modeling that flow lets the team check whether a required property holds across the possibilities in the model. One successful run through the app exercises only one such possibility.
  • Algorithm proofs with Lean. For a custom virtualization library, the team used Lean to reason about algorithm behavior that could produce flicker or jitter. The proof concerns properties of the algorithm. Seeing how the integrated UI behaves is still a job for the running app.
  • Property-based UI tests with Bombadil. The team also exercised the interface against properties it expected to hold as the UI was used. Tan reports that this had already caught a few UI bugs, alongside bugs found through formal verification. These were early results from complementary techniques.
9:499:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:40 · section reference included

Where the verification tokens go

A November 5 follow-up highlights the author's estimate that 70% of daily token spend goes toward verifying patches and about 20–30% toward creating them. The earlier post appears below.Open full source frame
Six months after subtle bugs slipped into merged AI-generated patches, Cindy Sridharan credits her new verification process with resolving the problem. She spends 70% of her daily tokens verifying patches and about 20–30% creating them.

Checking the work takes a substantial share of the token budget. Tan recounts another engineer’s experience with subtle bugs slipping through merges. Six months later, that engineer reported spending 70% of her daily tokens on verification. Tan also spends most of her tokens there. What changed the economics for her team was the cost of writing and maintaining the checks. Techniques that once took too much engineering time to understand and sustain became practical when agents could help with that work.

There is also a product question a passing check can leave open. Did the agent understand which behavior the user wanted? Tan built a statechart skill in pstack to visualize the state machine from the user’s point of view. A reviewer can look at which states an action can lead to, then compare that account with screenshots and a video of the implementation. A mistaken assumption about what should happen next is easier to discuss when both people can point to the transition. pstack plugin source

11:5812:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:51 · section reference included

What changed across 11,000 production PRs

Tan’s 3,000 PRs were part of a team total of 11,000 shipped to production in one month. The team cut a stable client release about once a day. The core team shipped more, and occasional contributors with less project context could land changes too. In October, about 227 people contributed at least one PR. As volume rose, the share of PRs that were reverted fell and settled at about 0.5%. That figure measures reversions. A bug fixed in a subsequent PR would escape that count.

What did that pace feel like to someone using the client? Compared with launch, Tan reports faster startup, more responsive typing, and faster opening and switching of chats. Those comparisons matter in an Electron app where a single bad import can put a long task on the renderer. They track the interactions the team wanted to protect as more people shipped code.

0:440:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:32 · section reference included

What happens when Grok Bot cuts a release

Tan gives a specific order for teams tempted to automate the whole operation. When an agent repeats a mistake, start with a lint rule that prevents new occurrences. Then clean up the old examples the next agent might copy. Give the agent a way to verify the resulting behavior. If the ordinary edit-and-check loop keeps producing faulty changes, running more copies of it also multiplies the tokens spent on repairs.

Once the inner loop works, connectors and routines can launch Cursor cloud agents, and projects can coordinate hundreds or even thousands of them. Grok Bot uses a release as one such trigger. Cutting a release automatically starts a new Cursor project. It runs the new build and fuzzes the application by interacting with it as a user would. For each bug it finds, it opens a PR with a fix and automatically merges it.

Automerge is the striking part of that sequence. What made Tan willing to use it was the work already done on the path each fix would take. A repair touching product code inherits Dune’s restrictions. The agent has tools for exercising the change, and reviewers can inspect its behavior. By the time the team automated release testing and repair, most product changes already required little deep human review.

15:1115:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:11 · section reference included

Transcript download
Full transcript

Loading transcript…