AI Engineer Code 2025
Why AI Didn't Actually Make You Ship Faster — Gabriel Spencer-Harper, Meticulous
Loading slides…
Read the talk
Why AI Didn't Actually Make You Ship Faster
Gabriel Spencer-Harper explains how Meticulous moves frontend verification from handwritten assertions to recorded workflows and visual comparisons—and why network mocking, browser determinism and coverage-guided selection have to work together.
From a talk by Gabriel Spencer-Harper
At a glance
Ideas worth remembering
Faster code generation can leave verification as the limiting step: the team still needs to understand a change across roles, permissions, flags and configurations.
Meticulous compares recorded workflows before and after a change. It exposes visible differences; a person or agent decides whether they are expected.
Recorded network responses make replay repeatable and isolate tests for parallel execution. Browser-level timing control addresses variation that network mocking cannot remove.
Coverage-guided selection depends on frequent screenshots to turn executed paths into visually checked paths. At large screenshot volumes, controlling flakiness is essential to keeping the differences useful.
Writing code faster leaves the merge decision behind
An agent finishes a pull request. Would you feel comfortable merging and shipping it? That is the practical question behind Gabriel Spencer-Harper’s talk. The co-founder and CEO of Meticulous starts with a mismatch: AI can produce code faster than people can establish that the change is safe. Someone still has to verify it, so faster code generation can move the bottleneck rather than remove it.
A frontend change can behave differently under different feature flags, roles, permissions, settings and configurations. Reviewing the patch does not automatically reveal its effect across those combinations. The work before merge includes reaching the relevant application states, observing what changed and deciding whether those changes are acceptable.
Assertion-based tests encode expected behavior in advance. A human or agent chooses what should happen and writes checks for it. Spencer-Harper’s objection is the size of the remaining space: even diligent test authors cannot anticipate every possible regression. A passing assertion answers the question someone thought to ask; the merge decision often depends on effects they did not anticipate.
The costs show up in several places:
- Regressions: shipping faster can expose users to bugs with business consequences.
- Verification work: manual validation, review, flaky-test debugging and suite maintenance consume engineering time. Spencer-Harper describes this as double-digit percentages of an organization’s time.
- Changes deferred: dependency upgrades and sweeping refactors become easier to attempt when engineers can inspect their consequences with confidence.
Meticulous is presented as a way to obtain exhaustive or near-exhaustive frontend verification without developers maintaining the tests. That ambition needs a precise reading: the mechanisms described here compare recorded frontend behavior and select flows by executed lines of code. They do not establish that every possible state has been recorded or that every kind of correctness is visible in a screenshot.
Replay the same workflow before and after the change
The demo begins with one line of JavaScript injected into non-production environments: localhost, QA, development or staging. It instruments the browser and records workflows as people use the application. Clicking a login button, opening settings and visiting analytics becomes a sequence of events that the system can replay. The recorded collection can contain thousands or tens of thousands of flows.
When a pull request opens, the CI runner starts the application on localhost port 3000. Meticulous selects recorded workflows and dispatches their events one by one. Throughout each replay, it captures screenshots at what Spencer-Harper calls each atomic moment. Running the workflow before and after the code change produces two screenshot sequences; comparing corresponding states reveals the application’s visible differences before merge.
The concrete example contains two changes: a name has changed, and an error has been introduced into a dropdown. The comparison exposes the name change in both light and dark modes, along with a visible manifestation of the dropdown’s logical error. The causal path is straightforward: replay reaches the relevant state, the changed code renders it differently, and the before/after comparison puts that difference in front of the reviewer.
A pull-request comment leads to the differences, typically within a few minutes in the workflow described. Meticulous does not label each difference a bug. A reviewer decides whether the name change is intended and whether the dropdown’s changed behavior is acceptable. Business judgment moves from writing an assertion before the change to interpreting an observed difference afterward; Spencer-Harper also allows for an agent to perform that review.
Where does the reviewer enter the process? The diagram separates event replay and comparison from the decision about correctness. Both versions receive the recorded workflow, but the resulting diff still needs someone to interpret it. In the demo, that distinction lets an expected name change and an unexpected dropdown change appear through the same mechanism.
One workflow, two versions, one review decision
Read the diagram as text
- Recorded workflow. Browser events captured in a non-production environment.
- Before the change. replay — Replay events and capture a screenshot sequence.
- After the change. replay — Replay events and capture the corresponding sequence.
- Visual differences. Name changes in light and dark modes; a visible dropdown error.
- Review. A person or agent decides which changes are expected.
- Recorded workflow → Before the change: Dispatch recorded events.
- Recorded workflow → After the change: Dispatch recorded events.
- Before the change → Visual differences: Baseline screenshots.
- After the change → Visual differences: Changed screenshots.
- Visual differences → Review: Interpret before and after.
Control the network and the browser’s timing
Comparing screenshots only helps if repeated runs of unchanged code produce comparable images. Meticulous addresses two sources of variation at different layers:
- Recorded network traffic: at record time, it captures network requests and responses. At replay time, it stubs them in, giving the workflow recorded responses rather than relying on changing live network results.
- Browser scheduling: it augments the browser from the scheduling-engine layer upward to control randomness that network mocking alone cannot remove.
Network mocking has two jobs in the design. First, it makes replay repeatable: Spencer-Harper describes running a test a thousand times and obtaining the same result. Second, each test is isolated from the others. Workflows need not compete through shared network state, so the system can run them horizontally in parallel and return results within a few minutes. The same decision that supports repeatability also supports speed.
The browser itself can still introduce differences. An animation spinner may render at different positions depending on the machine’s speed. Different interleavings of setTimeout and setInterval can alter what happens before a screenshot. If those differences survive replay, unchanged code can look changed, producing flaky results and wasting review attention.
Meticulous therefore aims for a deterministic or near-deterministic browser, controlling this variation below the test script. Spencer-Harper claims radically fewer flakes than Cypress or Playwright; the recording supplies the scheduling rationale rather than a quantified comparison. The useful architectural point is that visual comparison depends on controlling when the browser does work, as well as what data it receives.
Choose workflows by the code they exercise
A large recording library creates another problem: which workflows should run on each pull request? Meticulous replays each recorded session once against the main or master branch and monitors the lines of code it executes. That produces an index from workflows to executed lines. It then selects a subset of workflows that maximizes code coverage across the application.
This gives selection a concrete target. Instead of treating every recording as equally valuable, the system uses its execution history to choose flows that collectively reach code throughout the application. Feature flags, roles, permissions and configurations matter because they can lead execution through different branches. The index connects observed journeys to the implementation they actually exercised.
Why use code coverage when executing a line does not prove it behaved correctly? Spencer-Harper gives a sharp counterexample: a Cypress or Playwright test could execute 100% of a codebase while making only one assertion. Most of that execution would go unchecked. His phrase is the distinction to remember: “code covered is not the same as code tested.”
Meticulous tries to narrow that gap by comparing screenshots throughout the flow. The stated example is a 412-step workflow in which a single-pixel difference would be flagged. Executed code is therefore accompanied by much denser observation of its visible consequences than a single end-of-test assertion. That is the basis for Spencer-Harper’s claim that coverage and testing become approximately equivalent in this system: the approximation depends on the behavior producing an observable visual difference.
Dense observation makes determinism essential
The technical ending ties the design together. Coverage-guided selection supplies breadth across the recorded workflows. Screenshots throughout those workflows supply dense observation. But that density brings an enormous volume of comparisons: Spencer-Harper describes screenshot counts on the order of tens or hundreds of millions. Without controlling flakiness, the additional observation could drown the reviewer in noise.
What makes more observation useful rather than overwhelming? The stack below shows the dependency. Controlled replay makes differences more meaningful; frequent screenshots let selected flows expose changes along the way; coverage-guided selection determines which recorded paths contribute breadth. Removing browser variability is consequently part of the verification mechanism, not merely a convenience for running tests.
The merge decision still ends with judgment. In the demonstrated pull request, the name change and dropdown error both arrive as visual differences. The system’s contribution is to reach those states repeatedly and make the consequences inspectable before shipping. That is how the proposed verification layer supports the ambition from the opening: making larger changes, including dependency upgrades and refactors, without expanding manual exploration at the same pace as code generation.
The dependencies behind broad visual verification
Read the diagram as text
- Controlled replay. Recorded network responses isolate tests; browser scheduling controls timing variation.
- Screenshots throughout each flow. Observe visible changes along the workflow without drowning in incidental variation.
- Coverage-guided workflow selection. Choose recorded flows using their executed-line index.
- Controlled replay → Screenshots throughout each flow: Reduce flaky visual differences.
- Screenshots throughout each flow → Coverage-guided workflow selection: Make executed paths visually checked.
Full transcript
Loading transcript…