Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
0:00:00 Picture a pull request. It changes a schema in one of our data pipelines. It goes green in CI. The tests pass. The linter is happy. And everything about it looks clean. So is it safe to merge? Because a change like this can still break something downstream. Maybe it blows a freshness SLA. Or maybe it breaks a consumer three teams away who never saw it coming. Or maybe it exposes data that never supposed to be reachable.
0:00:29 That gap between CI passed and this change was actually safe is the gap my team has to close every day.
0:00:37 Hello, everyone. I'm Chetas. I'm an engineer on the data platform team at Robinhood.
0:00:42 I design and build the distributed data ingestion and processing frameworks behind our data lake,
0:00:48 the ones teams across the company rely on to make decisions with data.
0:00:53 Notice the title of the stock.
0:00:55 The bots in it are the ones opening the pull requests.
0:00:59 And as you will see, the ones reviewing them as well.
0:01:04 Today I want to show you how we answer that question when the PRs are coming in faster
0:01:09 than any of us can read them.
0:01:13 So here's the problem in one sentence.
0:01:16 Pull requests to our production data pipelines now arrive at machine speed.
0:01:21 but reviewing them safely still runs at human speed.
0:01:26 First, a quick word on the scope of this talk
0:01:29 because it shapes everything else in this talk.
0:01:32 AI coding assistants have made pull requests cheap to open.
0:01:36 A new query, a schema change, or an access grant
0:01:40 now takes just a few minutes.
0:01:42 And an assistant can open pull requests far faster
0:01:46 than any team of humans ever put.
0:01:50 The PRs I'm talking about today are the PRs toward data pipelines, like schema changes,
0:01:55 pipeline code, configuration changes, and grants that give someone access to data.
0:02:02 Data Platform is the team that owns reviewing that kind of PR. So when those PRs pile up,
0:02:08 they pile up on us. And the reviewer I'm about to describe is the system we built to handle them.
0:02:15 So the people who feel this are a small group, the engineers who carry the context to say
0:02:20 what is safe to merge into production and everyone who is waiting on them.
0:02:27 So why is this a problem?
0:02:30 AI open PRs just keep arriving.
0:02:34 And to be fair, the safety net we already had works.
0:02:37 CI checks together with the human review have always caught a lot of real bugs like broken
0:02:44 code, inefficient queries, and risks like downstream breakage or sensitive data leaks.
0:02:50 But the human half only works at the human speed and only with the context a person happens
0:02:56 to carry in their head.
0:02:59 Who reads this table?
0:03:00 How big is it?
0:03:01 And how sensitive is the data in it?
0:03:04 The PRs are coming in at the machine speed, but the number of people who understand what
0:03:09 is safe to merge into a production data pipeline did not grow at all.
0:03:13 That's the mismatch.
0:03:15 The queue builds up in front of the few people with the context and that's where things start
0:03:20 to slip.
0:03:22 So the problem we set out to solve was this.
0:03:25 Build a PR reviewer using AI that can keep up with the speed at which PRs are being opened.
0:03:33 So, here's the whole answer before we get into the details.
0:03:37 Three duties, three owners.
0:03:41 Scripts prove what can be proven.
0:03:43 An agent judges what needs context and people own the calls only they should make.
0:03:50 We took a hybrid approach.
0:03:52 Scripts for the facts and AI for judgment.
0:03:55 We did not want AI reviewing every change when some questions already have an authoritative
0:04:01 answer.
0:04:02 And that's layer one. Deterministic scripts check a source of truth like the schema registry
0:04:09 or a catalog. A failed check turns CI red and blocks the PR until it is fixed.
0:04:16 Layer two is the agent. The agent's rules tell it what layer one already checks. So it focuses
0:04:23 on the risks those checks cannot catch. It reviews the PR, scores the risk, and approves
0:04:30 or escalates to a person. Both these layers review the same PR. Merge requires passing
0:04:37 checks and an approving review. And this also keeps AI costs down. We use AI for judgment,
0:04:44 not to rediscover facts a script can look up. So here is where we are going today.
0:04:52 We have covered the problem and the approach. Now I'll walk you through the hybrid reviewer we build
0:04:58 and the decisions we deliberately keep with the people.
0:05:01 After that, how the reviewer learns from the human feedback
0:05:05 and what results we have seen so far.
0:05:07 And we'll finish with the lessons we will pass on to you.
0:05:12 Three things to listen for as we go.
0:05:15 Scripts for facts, an agent for the judgment,
0:05:18 and people for the protected calls.
0:05:20 I'll come back to all three at the end.
0:05:24 So to make this concrete,
0:05:26 Let me show you the one PR will follow through the rest of this talk.
0:05:32 This PR makes two changes to a table in our data lake.
0:05:37 It adds new column and widens an existing column from integer to big integer.
0:05:42 It's the kind of PR we see all the time.
0:05:44 On the left is an upstream source table.
0:05:47 Our ingestion pipeline reads from it and writes into a table in the data lake.
0:05:52 Call it our table.
0:05:55 The first change, the source has gained a new column and this PR makes ingestion bring
0:06:01 that new column into our lake table.
0:06:05 The second change is to a column our table already has.
0:06:09 Its type gets widened from integer to big integer, so it can hold bigger values.
0:06:15 On its own, that sounds completely harmless.
0:06:19 widening a type is about the safest schema change you can make, right?
0:06:24 Well, whenever a column changes, the real question is, who reads it downstream?
0:06:29 Some of those readers are registered in our lineage tool, so we know about them.
0:06:35 But keep a second question in the back of your mind, are those all of them?
0:06:40 Hold on to this picture, we will come back to this exact diagram each time the PR moves forward through the reviewer.
0:06:49 So let's start with layer one.
0:06:51 Layer one is the scripts and their whole job is checking facts in the live systems.
0:06:57 Things the code alone cannot tell you.
0:07:00 Here are the two of those checks and what they said about our running PR.
0:07:06 First, data contracts.
0:07:09 When a PR changes a schema, a script compares the schema and keys against a source of truth,
0:07:15 the schema registry.
0:07:18 For our PR the question is simple.
0:07:19 Does that new column actually exist upstream
0:07:22 as registered in the registry?
0:07:25 It does.
0:07:26 So that check passes.
0:07:29 Second, the required tags.
0:07:32 Every table needs an owner
0:07:33 and any PII has to be labeled in the catalog.
0:07:37 The script checks the catalog directly.
0:07:40 Does our table have an owner?
0:07:41 Yes.
0:07:43 Is its PII labeled?
0:07:44 Yes.
0:07:45 Pass again.
0:07:48 And if either of those had failed,
0:07:50 there is no debate and no model involved.
0:07:53 CI goes red like any other failing checks.
0:07:55 And the PR is blocked until it is fixed.
0:08:00 Contracts and tax are two of the layer one checks.
0:08:04 Next, the two that look at downstream impact and performance.
0:08:07 So these are the other two layer one checks.
0:08:10 They look past the table itself.
0:08:12 What happens downstream?
0:08:14 And whether the job will still run in time?
0:08:18 First, the lineage blast radius.
0:08:21 When a column changes like the one our PR widens,
0:08:25 a script walks every downstream job
0:08:27 the lineage tool has on record for that column
0:08:30 and asks a simple question.
0:08:31 Does any of them break?
0:08:33 It's a real check against a real system.
0:08:36 But notice the boundary I've drawn.
0:08:39 It can only see what is recorded in lineage.
0:08:42 Its view of downstream is exactly as complete
0:08:46 as that record, no more.
0:08:49 Second, the performance.
0:08:51 A correct change can still be too slow or too expensive.
0:08:55 So the CI performance test runs the change in staging
0:08:59 at the same data volume as production using mock data.
0:09:03 So that tells us two things before the PR merges,
0:09:07 whether the cluster is sized right for the job
0:09:09 and whether the job will still make its freshness SLA.
0:09:13 Those are the kind of problems
0:09:15 that otherwise show up weeks later as an incident.
0:09:18 So that's the full set of layer one checks.
0:09:22 Let's see what they said about our PR.
0:09:27 So back to our running PR, same picture as before.
0:09:31 Here's what the layer one said about it.
0:09:34 The data contract check went to the schema registry
0:09:38 and asked whether the new column really exists upstream
0:09:41 as register.
0:09:42 It does and the check passes.
0:09:46 The lineage blast radius check walked every registered consumer of the widened column.
0:09:51 The jobs the lineage tool has on record.
0:09:54 None of them break when the column goes from integer to big integer.
0:09:57 They all look fine.
0:09:59 Along with that, tags pass, performance passes, every check is green.
0:10:05 As far as the scripts can see, this PR is safe.
0:10:08 But I chose those words carefully.
0:10:10 as far as the scripts can see. Everything on the slide is a statement about the registered picture,
0:10:17 the jobs and the columns our systems have a record of. And keep that in mind, it is not the whole
0:10:23 story. Now let's look at the agent. With the diff, what actually changed? The PR description and the
0:10:33 comment messages explain the author's intent, but intent never explains away a real finding.
0:10:38 If the description says it is safe and the code says otherwise, the code wins.
0:10:46 File tools let the agent read, search and list the files across the repository.
0:10:50 It follows changes beyond the diff into pipeline definitions and access grants, looking for
0:10:56 the relationships the author may have missed.
0:11:01 These are the repository tools only.
0:11:03 The agent cannot query the life catalog,
0:11:06 lineage tool or production state.
0:11:09 Life facts stay layer one's job.
0:11:12 The agent reasons over the code.
0:11:14 So why walk that code for a schema change?
0:11:17 Because registered lineage is not the whole picture.
0:11:23 The rules define where the judgment belongs.
0:11:27 They are configured by the change type in the PR,
0:11:29 knowing what layer one already covers.
0:11:31 They target query anti-patterns, sensitive data asks, and consumers outside lineage.
0:11:38 For example, an unpartitioned cross-join can run correctly and still hammer a cluster.
0:11:44 A grant can look ordinary and still open sensitive data.
0:11:48 And for the schema changes, the agent walks the pipeline definitions and access grants
0:11:53 in the repository to find consumers that were not registered.
0:11:58 It's a tedious search a person would otherwise do by hand.
0:12:03 Let's return to our PR and see what that search found.
0:12:08 So back to our running PR.
0:12:09 Same picture.
0:12:10 Layer 1 said every registered consumer of the Widen column looks fine.
0:12:15 Now the agent walks the court.
0:12:18 And it finds a job that lineage never knew about.
0:12:21 It's right there in the repository.
0:12:24 It reads our widened column from our late table, but it was never registered in the
0:12:29 lineage, which is exactly why the layer one check could not see it.
0:12:34 And that job takes the value and writes it into another table.
0:12:38 And in that other table, the column is still an integer.
0:12:43 Now follow it through.
0:12:44 Our column is a big integer now.
0:12:46 The first time a value comes through that's bigger than an integer can hold, that write
0:12:52 phase. A valid schema chain, green in CI, passes every registered check and a downstream
0:12:59 job still breaks. The agent flagged exactly this chain in its review comment with the
0:13:07 reasoning so a person could see the whole path from our column to the failing right.
0:13:11 We'll get to what happened next in a moment. So why did the lineage miss that job? Because
0:13:19 enterprise data world is messy. Workloads run across different query engines and each engine
0:13:25 has to publish its own lineage. If one does not publish, that relationship is missing from the
0:13:30 picture. There is no single catalog that automatically sees everything. So the registered
0:13:36 picture is useful, but it is not the whole picture. Now, I don't want to oversell the code walk.
0:13:43 is genuinely useful, but it cannot see everything.
0:13:48 The agent can search what is in the repo.
0:13:51 Pipeline code, access grants, and the jobs defined there
0:13:54 that read our tables.
0:13:56 That's where it found our unregistered job.
0:14:00 But plenty of data use lives outside the codebase.
0:14:03 Ad hoc queries, where someone runs by hand, dashboards,
0:14:07 those are completely invisible to a repository search.
0:14:10 A consumer out there never shows up.
0:14:12 no matter how carefully the agent looks.
0:14:15 So we make the agent honest about that limit.
0:14:18 If it does not find anything,
0:14:20 it does not get to say no downstream consumers exist.
0:14:23 It says it found none in the places it searched.
0:14:27 None found is not the same as none exist.
0:14:30 And the agent never gets to claim an all clear.
0:14:35 So the agent has walked the code.
0:14:37 What does it actually produce?
0:14:39 A review.
0:14:40 the same shape a good human reviewer would give you. First review comments on
0:14:47 the specific lines that matter each with its reasoning. Not awake looks risky but
0:14:53 this line changes this column and here is the job that reads it. Second a risk
0:14:59 score how severe is what it found that's what drives the next step. Then it makes
0:15:06 a call. If there is no serious risk, it can approve. If there is a risk that needs a person,
0:15:11 it escalates to a human and the comments and reasoning go with it. So the person has the
0:15:17 context to investigate instead of starting from scratch. And to be clear about what it takes to
0:15:24 merge, layer 1's checks have to pass and the review has to approve either the agent or after
0:15:31 an escalation a person both have to say yes now let's see what that looked like for our pr
0:15:40 here is how our running pr actually ended same picture the registered consumers look fine
0:15:46 and the core walk found an unregistered job that writes our widened column into a table
0:15:52 when it is still an integer because the agent found a real downstream break it scored the risk
0:15:58 high and escalated to a human with the whole chain spelled out in its review comment.
0:16:04 Now the human does the part the agent cannot. The agent only reasons over the repo. It cannot see
0:16:10 whether the job is actually running in production. Human reviewer checked and confirmed that the
0:16:16 consumer the agent found was live and active. So this was not a theoretical risk. The PR waited.
0:16:24 The fix was about order, not about rejecting the change.
0:16:29 The downstream table was updated first,
0:16:32 so its column could handle the wider type.
0:16:36 And only then did our PR merge and deploy.
0:16:39 No incident.
0:16:40 The scripts in layer one said safe,
0:16:43 the agent found the gap, and a human made the call.
0:16:49 And that was not a one-time thing.
0:16:51 Most changes can be cleared by the agent.
0:16:53 When the checks pass and the agent is confident in its verdict, it approves on its own.
0:16:58 But there's a whole category of changes where the agent never gets to say approve and no matter how confident it sounds.
0:17:06 Grants touching sensitive data are one.
0:17:09 Who can reach sensitive data is a decision we want a person to own every time.
0:17:15 A migration that looks like it changes intent, not just mechanics is another.
0:17:21 If a change alters what the data means rather than just how it is moved or stored, a person
0:17:27 decides.
0:17:29 And if the agent itself is not confident in its own verdict, that's the third.
0:17:35 It does not get to round up to approve just because nothing concrete came back wrong.
0:17:39 In all the three cases, its job is only ever to surface the risk and its reasoning.
0:17:46 The decision stays with the person.
0:17:49 That is one PR.
0:17:52 Now how does this hold up across every PR we see and how does the reviewer get better
0:17:56 over time?
0:17:59 How do we know the flags are useful or whether a rule change helps?
0:18:03 We start with the agent's verdict and its reasoning in the review comments.
0:18:09 A person replies, agreeing or disagreeing and explaining why.
0:18:14 That explanation is what makes the feedback useful rather than just a vote.
0:18:19 Those replies are collected every day.
0:18:23 Once a week, the on-call reads the feedback and updates the review rules by hand.
0:18:28 Those rules shape the next review.
0:18:31 I know, this is manual today.
0:18:32 nothing retrains itself and no rule changes automatically.
0:18:36 But we have an idea to let an incident investigation agent
0:18:40 help with that weekly pass, but it is not built.
0:18:44 Without the feedback, every rule change would be a guess.
0:18:47 So let me show you an example where a human corrected
0:18:50 the agent and we use that feedback to improve
0:18:54 its next review.
0:18:57 So here's a real one.
0:18:59 And note the label in the corner.
0:19:00 This is a separate example, not our schema change PR.
0:19:05 A column was marked as personal information in the source database.
0:19:10 But by the time it lands in the data lake, the ingestion pipeline has already encrypted it.
0:19:15 So the PR proposed labeling it non-sensitive in the data lake, even though the upstream data is still PII.
0:19:23 The agent flagged this.
0:19:25 From where it stood, the upstream says PII, so a non-sensitive label looked wrong.
0:19:31 The author pushed back and asked the agent to look at how similar columns on the other
0:19:35 tables were already classified, instead of going purely off the upstream tagging.
0:19:42 A human agreed with the author.
0:19:45 The encryption before write made the non-sensitive label look okay for this column.
0:19:49 And that kind of correction is exactly the signal that teaches the rules to get smarter.
0:19:54 because the next time a similar column comes through,
0:19:58 that context can stop a needless escalation.
0:20:02 One caution though, this depends on the specific pipeline
0:20:05 encrypting before the write.
0:20:07 It's not a blanket rule that encryption
0:20:09 always makes data non-sensitive.
0:20:14 So how do we know whether the reviewer
0:20:16 is actually doing a good job?
0:20:18 We look at two kinds of mistakes.
0:20:20 Bad PRs it approved and fine PRs
0:20:23 it unnecessarily send to a person.
0:20:26 That matrix separates four possibilities.
0:20:31 A fine PR approved and a bad PR escalated
0:20:33 are the two correct outcomes.
0:20:36 A bad PR approved is a miss.
0:20:39 That's dangerous because the change ships.
0:20:42 A fine PR escalated is a false escalation
0:20:46 that costs a person's time.
0:20:48 A gate that escalates everything may avoid misses,
0:20:51 but it has not reduced the review load.
0:20:55 As of August 31st, 2026, our miss rate was about 2%.
0:21:01 Mostly from the gaps in the rules
0:21:03 rather than the failures of the judgment.
0:21:05 We don't have a clean single false escalation number,
0:21:09 but the dominant cause is missing context
0:21:11 like the encrypted column example that I just gave you.
0:21:16 Misses give us new rules
0:21:17 and false escalation show us context the reviewer needs.
0:21:21 So here is one of those misses.
0:21:23 Again, know the label.
0:21:25 This is a separate example, not our schema change PR.
0:21:29 We have an open copy of some data.
0:21:32 Some of its columns are about to be re-tagged.
0:21:35 And there is a downstream reader
0:21:37 that reads those columns from the open copy.
0:21:41 Privacy re-tagged those columns as personal information.
0:21:44 When that happens, they move into a more restricted copy
0:21:48 of the data and they are gone from the open copy.
0:21:51 The re-tagged PR looked fine and it merged.
0:21:54 But what nobody checked was whether anything downstream was still reading those columns
0:21:59 from the old open copy.
0:22:01 Something was.
0:22:02 And once the columns were gone, that reader broke.
0:22:05 That's a miss.
0:22:06 A PR that should have been escalated got approved.
0:22:11 It wasn't a failure of judgment.
0:22:13 It was a gap in the rules.
0:22:14 A corner case nobody had written down.
0:22:17 So it became a new rule.
0:22:18 When a column gets re-tagged as sensitive, search the code base for anyone still reading
0:22:24 it from the copy that's about to lose it.
0:22:26 And that's how the rules get better.
0:22:29 The misses get found and get turned into new rules.
0:22:34 So let's pull it all together.
0:22:37 Remember the pull request from the start?
0:22:39 The one that was green in CI?
0:22:41 Three things happened to it.
0:22:44 First, scripts proved what a source of truth could prove.
0:22:49 The contracts, the tags, the registered lineage and performance all passed.
0:22:54 Second, the agent read the code and found what the scripts could not see.
0:22:58 A job that lineage never knew about, quietly waiting to fail on a wider column.
0:23:04 It flagged the whole chain and escalated.
0:23:07 Third, a person confirmed the job was live.
0:23:09 the downstream table was fixed first and only then did the PR merge. No incident.
0:23:15 And behind all of that, the reviewer keeps getting better because every correction a person makes
0:23:22 can become a rule. These results are as of August 31st, 2026. So you might ask, Chetis,
0:23:29 does it all work? Yes. About 60% of the data pipeline PRs merge without a human review.
0:23:37 We have seen about 75% less manual review effort.
0:23:42 The agent's specific comments and reasoning help focus the review rather than making a person reconstruct every risk from scratch.
0:23:51 The time saved comes to about 132 engineering hours a month.
0:23:56 Incidents tied to these PRs are down about 30%.
0:24:01 And that does not mean human-free review.
0:24:04 The protected decisions still reach a person by design.
0:24:08 The goal is less review work, not removing accountability.
0:24:12 And those are the results.
0:24:14 Now let's look at the architecture choices we got wrong.
0:24:20 Here is what we got wrong.
0:24:22 Today, layer one and layer two are separate systems,
0:24:25 triggered independently and running in parallel.
0:24:29 Each produces its own result.
0:24:31 that seemed like an obvious design when we built it.
0:24:35 But neither sees the others pass, fail or outage state.
0:24:40 Layer two knows what the scripts are configured to cover,
0:24:43 not what happened when they ran.
0:24:46 Merge still needs passing checks and an approving review,
0:24:49 but no single system has a full runtime picture.
0:24:53 The agent also cannot call live layer one checks.
0:24:57 Its tools only read, search and list the repository files.
0:25:01 If a judgment needs a live fact from a live production system, it cannot fetch that fact
0:25:07 itself.
0:25:09 When one site cannot finish, we wait for a human.
0:25:12 There is no automatic fallback.
0:25:15 These are current limitations and they motivate the proposal on the next slide.
0:25:21 Platform team is exploring one entry point.
0:25:23 This is a proposal and not what's running today.
0:25:25 The left panel is the current design and the right is where we want to go.
0:25:31 A classifier looks at the whole PR then decides which changes to review separately and which to review together.
0:25:39 This is a whole PR decision, not independent routing of each file.
0:25:44 The agent would also get the layer one checks as callable tools alongside its repository tools.
0:25:50 when judgment needs a live fact it could request the appropriate check instead of being limited to
0:25:55 code alone the aim here is one system with the whole PR in view we haven't built this yet so
0:26:02 it's a direction not reserved let me close with what we would carry forward so here is the whole
0:26:10 talk in three lines deterministic evaluations first if something can be proven against a source
0:26:17 of truth, let a script prove it. Don't spend a judgment call on a yes or no fact. It's
0:26:23 not just cleaner, it's cheaper too. Every fact a script proves is review work that never
0:26:29 needs AI. So the cost of running this stays down. Flag never clear. Not for every PR,
0:26:38 but for the changes that actually scare people like sensitive grants, migration intent, and
0:26:43 low confidence verdicts. There, the agent's job stops at surfacing the risk and the reasoning
0:26:50 behind it. The decision stays with the person every time. Count the false escalations. A gate
0:26:58 that escalates everything is not actually saving anyone any time. It's just moving the work around.
0:27:05 Measure that cost the same way you measure a miss. So I started with a pull request. There
0:27:10 there was green NCI and a problem.
0:27:13 PRs arriving at machine speed and too few people
0:27:16 with the context to review them safely.
0:27:18 We built a hybrid reviewer, scripts for the facts,
0:27:21 an agent for judgment and people for the protected calls.
0:27:25 We walked one PR through all three,
0:27:28 saw it get lighter without going human free
0:27:31 and saw how it learns from the corrections.
0:27:33 We told you what we got wrong
0:27:35 and we left you with three lessons.
0:27:37 prove what you can cheaply,
0:27:40 flag and never clear the scary changes,
0:27:43 and count the false escalations.
0:27:46 So who reviews the bots?
0:27:48 Scripts for everything a source of truth can prove,
0:27:51 an agent for everything that needs context and judgment,
0:27:54 and humans for the calls only they get to make.
0:27:58 Thank you for listening.
