60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood

A schema change passes its tests and the linter. Is it safe to merge? A pipeline can produce correct data too late to meet its freshness deadline. A change can break a consumer three teams away or give someone access to sensitive data. For Chetas Joshi’s data platform team at Robinhood, answering the merge question means understanding who depends on the changed code and what they expect from it.

Coding assistants could open a PR for a new query, schema change or access grant in minutes. Reviews of pipeline code and configuration also landed with the data platform team. CI and human review already caught broken code, inefficient queries and data leaks. What slowed the queue was waiting for one of the few engineers who knew who read a table, how large it was and what sensitive data it held. The number of people carrying that context stayed flat as the PRs piled up.

Robinhood split the work by the kind of answer each reviewer could give. A script can look up a fact in a schema registry or catalog and block a PR if the check fails. An agent can read the code, assess risk, and either approve or ask a person to decide. Its rules tell it which questions the scripts cover, saving the cost of asking a model to reconstruct a fact the team can look up. A merge needs both passing checks and an approving review.

An ingestion pipeline reads an upstream source table and writes to a table in the data lake. One PR makes two changes. The source has gained a column, and the pipeline will bring it into the lake. A different column, already in the lake table, will widen from INTEGER to BIGINT so it can hold larger values. The first edit raises a question about the source. The second raises a question about every job that reads the widened column.

• Data contract. A script compares the proposed schema and keys with the schema registry. The registry confirms that the new column exists upstream.

• Required tags. Every table needs an owner, and personally identifiable information (PII) needs a label in the catalog. A direct catalog lookup confirms both for this table.

• Downstream dependencies. The lineage tool records which jobs read a column. A script walks every job registered for the widened column and checks whether the type change breaks it.

• Performance. CI runs the change in staging with mock data at production volume. The run checks whether the cluster is sized for the job and whether the job can finish in time to meet its freshness service-level agreement.

A failed check blocks the PR until someone fixes it. Here all four passed. How far downstream did that green result reach? Every registered consumer could handle the wider type. The word registered matters. The script starts with the lineage tool’s list, so a job missing from that list contributes nothing to the result.

The agent starts with the diff. The PR description and commit messages explain what the author intended, but a finding in the code takes precedence over a claim that the change is safe. The agent can list, search and read files across the repository. It has no direct access to the live catalog, lineage tool or production state.

The type of change tells the agent what to look for. In a query, an unpartitioned cross join can return the right answer while hammering a cluster. In an access grant, an ordinary-looking edit can open sensitive data. For a schema change, the rules direct the agent into pipeline definitions and access grants to find consumers beyond those recorded in lineage. For this PR, the question was where a value would go after a job read it from the BIGINT column.

The search found a job defined in the repository that had never been registered in lineage. It read the widened column and copied the value to another table, where the receiving column was still INTEGER. Values within the original range would continue to fit. The first value large enough to need the wider range would reach that last write and fail. The agent’s review comment traced the path from the changed column through the job to the destination.

Each query engine has to publish its own lineage. If an engine omits a relationship, a repository search can still recover it from a job definition. The search has limits of its own. An ad hoc query someone runs by hand, or a dashboard defined outside the codebase, leaves the agent nothing to read. When a search comes up empty, the agent reports where it looked and that it found no consumers there.

The agent attaches its reasoning to specific lines and scores the severity of what it found. For an ordinary change, a confident review with no serious finding can supply the approval needed alongside passing CI. For the widening PR, the agent scored the downstream write as high risk and escalated. The engineer who picked it up already knew which job to investigate. The open question was whether that job actually ran in production.

The human reviewer confirmed the consumer was live. The PR waited while the team widened the receiving column in the downstream table. Once that destination could hold the larger values, the original PR merged and deployed without incident.

Some decisions always go to a person. In each case, the agent passes along its findings and reasoning.

• Sensitive access grants. A person decides who may reach sensitive data, however confident the agent sounds.

• Migrations that appear to change meaning. If a migration changes what the data represents, a person decides whether that change of intent is acceptable.

• Low-confidence verdicts. Even when the search turns up no concrete defect, uncertainty sends the review to a person.

A person can reply to the agent’s comment, agree or disagree, and explain why. The explanation gives the on-call engineer something specific to change. The team collects replies daily. Once a week the on-call engineer reads them and edits the review rules by hand. Those edits shape subsequent reviews without retraining the model. Using an incident-investigation agent to help with the weekly pass was still an idea.

In a separate PR, a source database marked a column as PII. The ingestion pipeline encrypted the column before writing it to the lake, and the author proposed a nonsensitive label for the lake copy. The agent saw a disagreement between the upstream and downstream labels and flagged it. The author pointed to comparable columns in other tables and asked the agent to consider how those columns were classified.

A human reviewer agreed with the author about this column. What happened between the source and the lake mattered to the decision. A future review could check where the pipeline applies encryption and how the team classifies comparable output before escalating a label mismatch. The decision here depended on encryption before this particular write. Encryption alone cannot settle the classification of another dataset.

How can the team tell whether a review rule helps? Cross the agent’s decision with the decision the PR needed. Approving a safe change and escalating a risky one are both correct. Approving a risky change is a miss that lets the change ship. Escalating a safe change unnecessarily consumes an engineer’s time. A reviewer could avoid every unsafe approval by forwarding every PR, leaving the original queue intact.

As of August 31, 2026, Joshi reported a miss rate of about 2%, without specifying the denominator. The team traced most misses to gaps in the review rules. It had no single reliable rate for false escalations, but the most common cause was missing context of the kind the encrypted-column example exposed.

One miss started as a privacy correction. A downstream job read several columns from an open copy of a dataset. When the privacy team retagged the columns as personal information, they moved to a restricted copy and disappeared from the open one. The retag itself looked reasonable. The reviewer approved it, the PR merged, and the job broke when it tried to read the missing columns.

The team added a rule with a specific trigger and search target. When a column is retagged as sensitive, search the codebase for readers that still reference that column in the copy about to lose it. That gives the reviewer a way to flag a broken read even when the privacy classification is correct.

As of August 31, 2026, about 60% of Robinhood’s data pipeline PRs merged without human review. The team reported about 75% less manual review effort, saving roughly 132 engineering hours a month. The widening PR illustrates how an escalation could also save work. The engineer could go straight to checking whether a particular consumer was live because the agent had already traced the dependency.

The team also observed about 30% fewer incidents tied to these PRs. That figure describes the team’s experience with the hybrid reviewer in place, including the engineers who decided protected cases and investigated escalations.

The two layers in the deployed design run in parallel. Separate triggers start the scripts and the agent on the same PR, and each produces its own result. The agent knows which checks are configured, but it cannot see whether a particular check passed, failed or hit an outage. Nor can it call a check when its investigation turns up a question about production. Merge still requires passing CI and approval. If either system cannot finish, the team waits for a human.

What would a single entry point change? In the proposed design, a classifier would examine the whole PR before deciding which changes to review together and which to handle separately. That would let the grouping decision account for relationships across files. The agent would also get callable deterministic checks alongside its repository tools, so it could request a live fact when the code raised a question. The team was exploring this design. The reported savings came from the parallel systems already in use.

Joshi would carry three decisions into that redesign. Keep yes-or-no lookups in scripts, where each lookup saves model work. Give people the final call on sensitive grants, migrations that change meaning and uncertain verdicts. And count unnecessary escalations alongside misses. Each needless referral spends the time of the same few engineers the reviewer was built to free up.