I Made a Fruit Fly Review My Code - Nick Nisi, WorkOS

A pull request can pass continuous integration and linting, earn five out of five from Greptile, and still sit waiting for Nick Nisi. At WorkOS, where he leads developer and agent experiences, coding agents have made it easier to put up another PR while the first one waits. The Greptile score already gives him confidence that he won’t break production. What does the team gain by having him read the diff again?

There are two things Nisi still wants to know. Does the software do what he expects? And why does the proposed change matter to his team? Clicking Approve records a decision without answering either question. To see how little it takes to produce a plausible vote, he gave the job to a fruit fly.

In 1947, a moth got into a relay and disrupted a computation. Nisi turns the old debugging story around. Other people had put a simulated fly to work playing Doom and Beat Saber. His assignment for it was reviewing JavaScript.

What part of this reviewer comes from a real fly? Its connectome is a map of the wiring in a male fruit fly’s brain. Researchers cut the brain into thousands of slices, photographed them under an electron microscope and traced the connections across slices. Nisi describes roughly 176,000 neurons with 6.3 million connections between them, accounting for more than 125 million synapses.

The map gave Nisi a circuit he could run on a laptop. For learning, he let the strength of a small subset of connections change with experience. That still left an interface problem. A fly can respond to an odor. How do you give a JavaScript diff a smell?

A fly senses odors through tiny hairlike structures called sensilla. In the demo, Jev stands in for the nose. It answers 30 yes-or-no questions about the diff. The encoder drops the two answers about whether behavior changed because that question belongs to the tests. It combines the remaining 28 answers with signals from changed code tokens, including identifiers and operators. That pattern of inputs is the smell the simulated brain receives. Explore the demo’s odor encoder

Where does a test result enter the vote? Before the fly reviews a PR, Nisi trains it on changes to the codebase, using mutation testing to generate examples by altering code and running tests. During a lesson, the smell determines which cells are active. A taste delivered at the same time determines which of their connections weaken.

• A reproduced regression tastes bitter. A simulated punishment signal weakens active connections that would otherwise draw the fly toward that smell.
• A known harmless edit tastes sweet. When a change chosen to preserve behavior passes the checks, simulated sugar triggers a dopamine reward signal that weakens connections driving avoidance. A test run that simply fails to find a bug in some other edit leaves the outcome unknown. How the demo teaches the fly

Nisi compares the memory to a hash table whose keys collide on purpose. Different diffs can activate overlapping patterns, so a taste learned from one edit can influence the response to another. At review time, the smell alone may make the fly extend its proboscis, its tongue, as if reaching for sugar water. That’s approval. If the tongue stays retracted, the demo reads the balance between approach and avoidance signals in a brain region called the mushroom body. That balance can yield no objection, a request for changes, or an abstention when neither signal is decisive. How the demo reads the vote

How much of that vote comes from the fly? Nisi tried erasing the learned memories while keeping the nose. The tongue started coming out for changes it should have avoided. Jev could still describe the diff, but the fly’s response depended on what earlier diffs had tasted like.

Would you approve the first diff? The game draws three changes at random from the fly project’s own codebase. For each one, the room votes, the fly gets its turn, and then the tests settle it. On the first change, nobody raises a hand for approval. Nisi allows that nobody may want to participate, then records the room’s vote as request changes.

The first diff smells good to the fly. It’s a bug. On the second change, in PKCE code, several hands go up for approval. The fly approves too, and this time the edit is harmless. On the last diff, only one hand goes up. Nisi calls it a rejection, the fly agrees, and the tests reveal a bug. The room goes three for three. The fly goes two for three.

In Nisi’s larger exam, the fly got 88.3% of 301 unseen changes right. There was a shortcut in that exam. Every bug changed behavior, and every harmless edit preserved it. A reviewer that distrusted changes in behavior could earn an impressive score. What would happen on a PR where new behavior was the point? Nisi’s account of the exam

Nisi pastes in a pull request he already approved for one of his Pi extensions. The system is supposed to fetch and read the change. He switches to his backup, which gives him no clear verdict either. This time there’s no vote to compare with his own.

How often did the harder examples draw a request for changes? Nisi reports 96% for the first example and 98% for the second, which he describes as a plausible mistake. Those figures tell us how readily the fly flagged the examples. A reviewer can reliably flag mistakes and still make it hard to ship a correct fix.

Nisi’s companion account puts a number on that cost. He gave the fly 24 tickets that required a behavior change, each with a correct solution and a plausible mistake. It approved 0 of the 24 correct solutions. Spotting an altered conditional or a piece of removed logic can alert a reviewer to a change. Deciding whether to accept it requires another piece of information. Did the team want to allow the behavior that conditional used to prevent? The fly can neither answer that question nor tell the author how to repair the JavaScript. Results from the 24-ticket test

Nisi’s proposal starts with a routing decision WorkOS already makes. GitHub’s CODEOWNERS identifies changes that belong to the developer experience team. When a PR would ordinarily call for a DX engineer, it could first invoke a review skill or script that the team maintains. Writing that skill means spelling out the minimum checks a DX engineer actually performs and keeping them up to date.

A PR that passed the team check would go to a lightweight classifier. It could weigh the complexity of the change, whether the implementation does what the description promises, and the risk if it goes wrong. A pass there would send the diff to an automated reviewer such as Greptile, CodeRabbit or Qodo. Where would a person enter? A failure at any of the three checks would call in a human. Passing all three would make the PR eligible to merge on its own.

If a merged change broke something, an agent could help repair the code while the team traced how the defect got through. A missed team convention might call for an edit to the skill. A defect the automated reviewer overlooked might call for a change there. The summary back to the human would also explain the decision behind the diff. What business logic changed? Why? Which alternatives did the team test and discard?

A summary can explain what the agent meant to accomplish. To judge whether the software actually does it, Nisi wants the PR to carry evidence from the run. The form depends on what the reviewer needs to establish.

• See the behavior. A screenshot can show the resulting state. A video can show how the software got there.
• Inspect the run. Test results and command output let the reviewer see what the agent checked and what came back.
• Repeat the check. A runnable smoke test gives the reviewer a quick way to exercise the change and compare outputs. Reproduction steps let someone try it manually while the team learns how much to trust the automated checks.

What would make a DX engineer stop and request changes? That’s the question the team has to answer when writing its skill. A platform team would bring different expectations to the same PR. Nisi jokes that he wouldn’t know what to do with the YAML files. The people who do know can specify which changes deserve their attention.

The fly offers a smaller experiment you can try yourself. Give it a pull request you’ve already judged and see whether it agrees, objects or sits this one out. Try the fly · Explore the presentation