1 00:00:00,000 --> 00:00:28,700 Picture a pull request. It changes a schema in one of our data pipelines. It goes green in CI. The tests pass. The linter is happy. And everything about it looks clean. So is it safe to merge? Because a change like this can still break something downstream. Maybe it blows a freshness SLA. Or maybe it breaks a consumer three teams away who never saw it coming. Or maybe it exposes data that never supposed to be reachable. 2 00:00:29,320 --> 00:00:36,760 That gap between CI passed and this change was actually safe is the gap my team has to close every day. 3 00:00:37,600 --> 00:00:42,140 Hello, everyone. I'm Chetas. I'm an engineer on the data platform team at Robinhood. 4 00:00:42,500 --> 00:00:47,800 I design and build the distributed data ingestion and processing frameworks behind our data lake, 5 00:00:48,100 --> 00:00:52,600 the ones teams across the company rely on to make decisions with data. 6 00:00:53,720 --> 00:00:55,300 Notice the title of the stock. 7 00:00:55,660 --> 00:00:58,860 The bots in it are the ones opening the pull requests. 8 00:00:59,260 --> 00:01:03,140 And as you will see, the ones reviewing them as well. 9 00:01:04,200 --> 00:01:09,340 Today I want to show you how we answer that question when the PRs are coming in faster 10 00:01:09,340 --> 00:01:11,060 than any of us can read them. 11 00:01:13,640 --> 00:01:15,620 So here's the problem in one sentence. 12 00:01:16,980 --> 00:01:21,300 Pull requests to our production data pipelines now arrive at machine speed. 13 00:01:21,300 --> 00:01:24,720 but reviewing them safely still runs at human speed. 14 00:01:26,020 --> 00:01:28,660 First, a quick word on the scope of this talk 15 00:01:29,200 --> 00:01:31,240 because it shapes everything else in this talk. 16 00:01:32,440 --> 00:01:36,120 AI coding assistants have made pull requests cheap to open. 17 00:01:36,420 --> 00:01:40,180 A new query, a schema change, or an access grant 18 00:01:40,180 --> 00:01:42,240 now takes just a few minutes. 19 00:01:42,680 --> 00:01:46,300 And an assistant can open pull requests far faster 20 00:01:46,300 --> 00:01:48,460 than any team of humans ever put. 21 00:01:50,080 --> 00:01:55,420 The PRs I'm talking about today are the PRs toward data pipelines, like schema changes, 22 00:01:55,920 --> 00:02:01,280 pipeline code, configuration changes, and grants that give someone access to data. 23 00:02:02,960 --> 00:02:08,400 Data Platform is the team that owns reviewing that kind of PR. So when those PRs pile up, 24 00:02:08,660 --> 00:02:14,320 they pile up on us. And the reviewer I'm about to describe is the system we built to handle them. 25 00:02:15,040 --> 00:02:20,620 So the people who feel this are a small group, the engineers who carry the context to say 26 00:02:20,620 --> 00:02:24,760 what is safe to merge into production and everyone who is waiting on them. 27 00:02:27,380 --> 00:02:28,800 So why is this a problem? 28 00:02:30,520 --> 00:02:33,320 AI open PRs just keep arriving. 29 00:02:34,180 --> 00:02:37,900 And to be fair, the safety net we already had works. 30 00:02:37,900 --> 00:02:44,180 CI checks together with the human review have always caught a lot of real bugs like broken 31 00:02:44,180 --> 00:02:50,460 code, inefficient queries, and risks like downstream breakage or sensitive data leaks. 32 00:02:50,760 --> 00:02:56,720 But the human half only works at the human speed and only with the context a person happens 33 00:02:56,720 --> 00:02:58,040 to carry in their head. 34 00:02:59,140 --> 00:02:59,940 Who reads this table? 35 00:03:00,300 --> 00:03:01,320 How big is it? 36 00:03:01,580 --> 00:03:03,660 And how sensitive is the data in it? 37 00:03:04,300 --> 00:03:09,240 The PRs are coming in at the machine speed, but the number of people who understand what 38 00:03:09,240 --> 00:03:12,800 is safe to merge into a production data pipeline did not grow at all. 39 00:03:13,660 --> 00:03:14,880 That's the mismatch. 40 00:03:15,360 --> 00:03:20,580 The queue builds up in front of the few people with the context and that's where things start 41 00:03:20,580 --> 00:03:20,980 to slip. 42 00:03:22,640 --> 00:03:25,020 So the problem we set out to solve was this. 43 00:03:25,720 --> 00:03:31,680 Build a PR reviewer using AI that can keep up with the speed at which PRs are being opened. 44 00:03:33,760 --> 00:03:37,020 So, here's the whole answer before we get into the details. 45 00:03:37,640 --> 00:03:39,400 Three duties, three owners. 46 00:03:41,160 --> 00:03:42,640 Scripts prove what can be proven. 47 00:03:43,700 --> 00:03:49,340 An agent judges what needs context and people own the calls only they should make. 48 00:03:50,700 --> 00:03:51,660 We took a hybrid approach. 49 00:03:52,500 --> 00:03:54,280 Scripts for the facts and AI for judgment. 50 00:03:55,040 --> 00:04:01,140 We did not want AI reviewing every change when some questions already have an authoritative 51 00:04:01,140 --> 00:04:01,640 answer. 52 00:04:02,360 --> 00:04:09,340 And that's layer one. Deterministic scripts check a source of truth like the schema registry 53 00:04:09,340 --> 00:04:15,060 or a catalog. A failed check turns CI red and blocks the PR until it is fixed. 54 00:04:16,620 --> 00:04:23,940 Layer two is the agent. The agent's rules tell it what layer one already checks. So it focuses 55 00:04:23,940 --> 00:04:30,340 on the risks those checks cannot catch. It reviews the PR, scores the risk, and approves 56 00:04:30,340 --> 00:04:37,200 or escalates to a person. Both these layers review the same PR. Merge requires passing 57 00:04:37,200 --> 00:04:44,660 checks and an approving review. And this also keeps AI costs down. We use AI for judgment, 58 00:04:44,980 --> 00:04:51,560 not to rediscover facts a script can look up. So here is where we are going today. 59 00:04:52,700 --> 00:04:58,020 We have covered the problem and the approach. Now I'll walk you through the hybrid reviewer we build 60 00:04:58,020 --> 00:05:00,880 and the decisions we deliberately keep with the people. 61 00:05:01,940 --> 00:05:05,240 After that, how the reviewer learns from the human feedback 62 00:05:05,240 --> 00:05:07,440 and what results we have seen so far. 63 00:05:07,900 --> 00:05:10,600 And we'll finish with the lessons we will pass on to you. 64 00:05:12,040 --> 00:05:14,460 Three things to listen for as we go. 65 00:05:15,060 --> 00:05:18,000 Scripts for facts, an agent for the judgment, 66 00:05:18,400 --> 00:05:20,480 and people for the protected calls. 67 00:05:20,920 --> 00:05:22,920 I'll come back to all three at the end. 68 00:05:24,220 --> 00:05:25,480 So to make this concrete, 69 00:05:26,240 --> 00:05:29,680 Let me show you the one PR will follow through the rest of this talk. 70 00:05:32,780 --> 00:05:36,740 This PR makes two changes to a table in our data lake. 71 00:05:37,040 --> 00:05:41,800 It adds new column and widens an existing column from integer to big integer. 72 00:05:42,120 --> 00:05:43,880 It's the kind of PR we see all the time. 73 00:05:44,920 --> 00:05:47,420 On the left is an upstream source table. 74 00:05:47,960 --> 00:05:52,380 Our ingestion pipeline reads from it and writes into a table in the data lake. 75 00:05:52,600 --> 00:05:53,720 Call it our table. 76 00:05:55,020 --> 00:06:01,140 The first change, the source has gained a new column and this PR makes ingestion bring 77 00:06:01,140 --> 00:06:03,360 that new column into our lake table. 78 00:06:05,380 --> 00:06:08,460 The second change is to a column our table already has. 79 00:06:09,700 --> 00:06:15,400 Its type gets widened from integer to big integer, so it can hold bigger values. 80 00:06:15,700 --> 00:06:18,140 On its own, that sounds completely harmless. 81 00:06:19,020 --> 00:06:22,660 widening a type is about the safest schema change you can make, right? 82 00:06:24,040 --> 00:06:28,780 Well, whenever a column changes, the real question is, who reads it downstream? 83 00:06:29,360 --> 00:06:33,820 Some of those readers are registered in our lineage tool, so we know about them. 84 00:06:35,160 --> 00:06:39,000 But keep a second question in the back of your mind, are those all of them? 85 00:06:40,360 --> 00:06:47,040 Hold on to this picture, we will come back to this exact diagram each time the PR moves forward through the reviewer. 86 00:06:49,420 --> 00:06:50,960 So let's start with layer one. 87 00:06:51,400 --> 00:06:56,600 Layer one is the scripts and their whole job is checking facts in the live systems. 88 00:06:57,200 --> 00:06:59,540 Things the code alone cannot tell you. 89 00:07:00,800 --> 00:07:04,920 Here are the two of those checks and what they said about our running PR. 90 00:07:06,940 --> 00:07:08,720 First, data contracts. 91 00:07:09,220 --> 00:07:15,760 When a PR changes a schema, a script compares the schema and keys against a source of truth, 92 00:07:15,940 --> 00:07:16,820 the schema registry. 93 00:07:18,140 --> 00:07:19,640 For our PR the question is simple. 94 00:07:19,940 --> 00:07:22,720 Does that new column actually exist upstream 95 00:07:22,720 --> 00:07:24,620 as registered in the registry? 96 00:07:25,040 --> 00:07:25,840 It does. 97 00:07:26,300 --> 00:07:27,960 So that check passes. 98 00:07:29,560 --> 00:07:31,640 Second, the required tags. 99 00:07:32,060 --> 00:07:33,720 Every table needs an owner 100 00:07:33,720 --> 00:07:36,380 and any PII has to be labeled in the catalog. 101 00:07:37,120 --> 00:07:39,760 The script checks the catalog directly. 102 00:07:40,080 --> 00:07:41,540 Does our table have an owner? 103 00:07:41,840 --> 00:07:42,460 Yes. 104 00:07:43,040 --> 00:07:44,680 Is its PII labeled? 105 00:07:44,940 --> 00:07:45,400 Yes. 106 00:07:45,840 --> 00:07:46,520 Pass again. 107 00:07:48,140 --> 00:07:50,040 And if either of those had failed, 108 00:07:50,280 --> 00:07:52,600 there is no debate and no model involved. 109 00:07:53,180 --> 00:07:55,500 CI goes red like any other failing checks. 110 00:07:55,920 --> 00:07:58,480 And the PR is blocked until it is fixed. 111 00:08:00,200 --> 00:08:02,680 Contracts and tax are two of the layer one checks. 112 00:08:04,060 --> 00:08:07,180 Next, the two that look at downstream impact and performance. 113 00:08:07,860 --> 00:08:10,000 So these are the other two layer one checks. 114 00:08:10,600 --> 00:08:12,400 They look past the table itself. 115 00:08:12,680 --> 00:08:13,940 What happens downstream? 116 00:08:14,260 --> 00:08:16,740 And whether the job will still run in time? 117 00:08:18,640 --> 00:08:21,240 First, the lineage blast radius. 118 00:08:21,720 --> 00:08:24,820 When a column changes like the one our PR widens, 119 00:08:25,200 --> 00:08:27,500 a script walks every downstream job 120 00:08:27,500 --> 00:08:29,620 the lineage tool has on record for that column 121 00:08:30,140 --> 00:08:31,580 and asks a simple question. 122 00:08:31,880 --> 00:08:33,180 Does any of them break? 123 00:08:33,780 --> 00:08:36,440 It's a real check against a real system. 124 00:08:36,920 --> 00:08:39,060 But notice the boundary I've drawn. 125 00:08:39,500 --> 00:08:42,440 It can only see what is recorded in lineage. 126 00:08:42,780 --> 00:08:46,360 Its view of downstream is exactly as complete 127 00:08:46,360 --> 00:08:48,160 as that record, no more. 128 00:08:49,700 --> 00:08:51,220 Second, the performance. 129 00:08:51,680 --> 00:08:54,840 A correct change can still be too slow or too expensive. 130 00:08:55,220 --> 00:08:59,000 So the CI performance test runs the change in staging 131 00:08:59,000 --> 00:09:02,300 at the same data volume as production using mock data. 132 00:09:03,820 --> 00:09:06,800 So that tells us two things before the PR merges, 133 00:09:07,120 --> 00:09:09,540 whether the cluster is sized right for the job 134 00:09:09,540 --> 00:09:12,740 and whether the job will still make its freshness SLA. 135 00:09:13,520 --> 00:09:15,000 Those are the kind of problems 136 00:09:15,000 --> 00:09:18,420 that otherwise show up weeks later as an incident. 137 00:09:18,980 --> 00:09:21,560 So that's the full set of layer one checks. 138 00:09:22,180 --> 00:09:24,600 Let's see what they said about our PR. 139 00:09:27,960 --> 00:09:30,880 So back to our running PR, same picture as before. 140 00:09:31,340 --> 00:09:33,460 Here's what the layer one said about it. 141 00:09:34,900 --> 00:09:38,080 The data contract check went to the schema registry 142 00:09:38,080 --> 00:09:41,280 and asked whether the new column really exists upstream 143 00:09:41,280 --> 00:09:42,200 as register. 144 00:09:42,200 --> 00:09:44,860 It does and the check passes. 145 00:09:46,800 --> 00:09:50,700 The lineage blast radius check walked every registered consumer of the widened column. 146 00:09:51,020 --> 00:09:53,320 The jobs the lineage tool has on record. 147 00:09:54,380 --> 00:09:57,020 None of them break when the column goes from integer to big integer. 148 00:09:57,280 --> 00:09:58,540 They all look fine. 149 00:09:59,500 --> 00:10:04,140 Along with that, tags pass, performance passes, every check is green. 150 00:10:05,760 --> 00:10:07,920 As far as the scripts can see, this PR is safe. 151 00:10:08,200 --> 00:10:10,320 But I chose those words carefully. 152 00:10:10,320 --> 00:10:16,940 as far as the scripts can see. Everything on the slide is a statement about the registered picture, 153 00:10:17,320 --> 00:10:23,800 the jobs and the columns our systems have a record of. And keep that in mind, it is not the whole 154 00:10:23,800 --> 00:10:33,160 story. Now let's look at the agent. With the diff, what actually changed? The PR description and the 155 00:10:33,160 --> 00:10:38,740 comment messages explain the author's intent, but intent never explains away a real finding. 156 00:10:38,740 --> 00:10:44,100 If the description says it is safe and the code says otherwise, the code wins. 157 00:10:46,520 --> 00:10:50,280 File tools let the agent read, search and list the files across the repository. 158 00:10:50,740 --> 00:10:56,620 It follows changes beyond the diff into pipeline definitions and access grants, looking for 159 00:10:56,620 --> 00:10:58,580 the relationships the author may have missed. 160 00:11:01,300 --> 00:11:03,620 These are the repository tools only. 161 00:11:03,620 --> 00:11:06,520 The agent cannot query the life catalog, 162 00:11:06,860 --> 00:11:08,640 lineage tool or production state. 163 00:11:09,080 --> 00:11:11,820 Life facts stay layer one's job. 164 00:11:12,260 --> 00:11:14,400 The agent reasons over the code. 165 00:11:14,820 --> 00:11:17,120 So why walk that code for a schema change? 166 00:11:17,620 --> 00:11:21,020 Because registered lineage is not the whole picture. 167 00:11:23,760 --> 00:11:26,640 The rules define where the judgment belongs. 168 00:11:27,020 --> 00:11:29,340 They are configured by the change type in the PR, 169 00:11:29,680 --> 00:11:31,400 knowing what layer one already covers. 170 00:11:31,400 --> 00:11:37,880 They target query anti-patterns, sensitive data asks, and consumers outside lineage. 171 00:11:38,700 --> 00:11:43,580 For example, an unpartitioned cross-join can run correctly and still hammer a cluster. 172 00:11:44,620 --> 00:11:47,440 A grant can look ordinary and still open sensitive data. 173 00:11:48,380 --> 00:11:53,660 And for the schema changes, the agent walks the pipeline definitions and access grants 174 00:11:53,660 --> 00:11:57,460 in the repository to find consumers that were not registered. 175 00:11:58,100 --> 00:12:01,440 It's a tedious search a person would otherwise do by hand. 176 00:12:03,160 --> 00:12:05,660 Let's return to our PR and see what that search found. 177 00:12:08,060 --> 00:12:09,080 So back to our running PR. 178 00:12:09,380 --> 00:12:10,140 Same picture. 179 00:12:10,540 --> 00:12:14,880 Layer 1 said every registered consumer of the Widen column looks fine. 180 00:12:15,920 --> 00:12:17,100 Now the agent walks the court. 181 00:12:18,160 --> 00:12:21,520 And it finds a job that lineage never knew about. 182 00:12:21,860 --> 00:12:23,680 It's right there in the repository. 183 00:12:24,240 --> 00:12:29,020 It reads our widened column from our late table, but it was never registered in the 184 00:12:29,020 --> 00:12:32,640 lineage, which is exactly why the layer one check could not see it. 185 00:12:34,220 --> 00:12:37,720 And that job takes the value and writes it into another table. 186 00:12:38,200 --> 00:12:41,300 And in that other table, the column is still an integer. 187 00:12:43,240 --> 00:12:43,900 Now follow it through. 188 00:12:44,140 --> 00:12:46,320 Our column is a big integer now. 189 00:12:46,740 --> 00:12:52,740 The first time a value comes through that's bigger than an integer can hold, that write 190 00:12:52,740 --> 00:12:59,740 phase. A valid schema chain, green in CI, passes every registered check and a downstream 191 00:12:59,740 --> 00:13:07,080 job still breaks. The agent flagged exactly this chain in its review comment with the 192 00:13:07,080 --> 00:13:11,500 reasoning so a person could see the whole path from our column to the failing right. 193 00:13:11,920 --> 00:13:19,480 We'll get to what happened next in a moment. So why did the lineage miss that job? Because 194 00:13:19,480 --> 00:13:25,240 enterprise data world is messy. Workloads run across different query engines and each engine 195 00:13:25,240 --> 00:13:30,900 has to publish its own lineage. If one does not publish, that relationship is missing from the 196 00:13:30,900 --> 00:13:36,160 picture. There is no single catalog that automatically sees everything. So the registered 197 00:13:36,160 --> 00:13:43,700 picture is useful, but it is not the whole picture. Now, I don't want to oversell the code walk. 198 00:13:43,700 --> 00:13:47,080 is genuinely useful, but it cannot see everything. 199 00:13:48,240 --> 00:13:51,040 The agent can search what is in the repo. 200 00:13:51,400 --> 00:13:54,740 Pipeline code, access grants, and the jobs defined there 201 00:13:54,740 --> 00:13:56,160 that read our tables. 202 00:13:56,680 --> 00:13:59,440 That's where it found our unregistered job. 203 00:14:00,080 --> 00:14:03,420 But plenty of data use lives outside the codebase. 204 00:14:03,960 --> 00:14:07,060 Ad hoc queries, where someone runs by hand, dashboards, 205 00:14:07,320 --> 00:14:10,100 those are completely invisible to a repository search. 206 00:14:10,360 --> 00:14:12,680 A consumer out there never shows up. 207 00:14:12,680 --> 00:14:14,780 no matter how carefully the agent looks. 208 00:14:15,500 --> 00:14:18,000 So we make the agent honest about that limit. 209 00:14:18,180 --> 00:14:19,860 If it does not find anything, 210 00:14:20,180 --> 00:14:23,120 it does not get to say no downstream consumers exist. 211 00:14:23,400 --> 00:14:26,700 It says it found none in the places it searched. 212 00:14:27,480 --> 00:14:30,280 None found is not the same as none exist. 213 00:14:30,560 --> 00:14:33,280 And the agent never gets to claim an all clear. 214 00:14:35,880 --> 00:14:37,720 So the agent has walked the code. 215 00:14:37,920 --> 00:14:39,380 What does it actually produce? 216 00:14:39,640 --> 00:14:40,320 A review. 217 00:14:40,320 --> 00:14:46,680 the same shape a good human reviewer would give you. First review comments on 218 00:14:47,240 --> 00:14:53,700 the specific lines that matter each with its reasoning. Not awake looks risky but 219 00:14:53,700 --> 00:14:59,700 this line changes this column and here is the job that reads it. Second a risk 220 00:14:59,700 --> 00:15:06,140 score how severe is what it found that's what drives the next step. Then it makes 221 00:15:06,140 --> 00:15:11,420 a call. If there is no serious risk, it can approve. If there is a risk that needs a person, 222 00:15:11,740 --> 00:15:17,960 it escalates to a human and the comments and reasoning go with it. So the person has the 223 00:15:17,960 --> 00:15:24,640 context to investigate instead of starting from scratch. And to be clear about what it takes to 224 00:15:24,640 --> 00:15:31,640 merge, layer 1's checks have to pass and the review has to approve either the agent or after 225 00:15:31,640 --> 00:15:38,620 an escalation a person both have to say yes now let's see what that looked like for our pr 226 00:15:40,160 --> 00:15:46,300 here is how our running pr actually ended same picture the registered consumers look fine 227 00:15:46,300 --> 00:15:52,060 and the core walk found an unregistered job that writes our widened column into a table 228 00:15:52,060 --> 00:15:58,760 when it is still an integer because the agent found a real downstream break it scored the risk 229 00:15:58,760 --> 00:16:03,540 high and escalated to a human with the whole chain spelled out in its review comment. 230 00:16:04,320 --> 00:16:10,820 Now the human does the part the agent cannot. The agent only reasons over the repo. It cannot see 231 00:16:10,820 --> 00:16:16,440 whether the job is actually running in production. Human reviewer checked and confirmed that the 232 00:16:16,440 --> 00:16:23,500 consumer the agent found was live and active. So this was not a theoretical risk. The PR waited. 233 00:16:24,640 --> 00:16:29,480 The fix was about order, not about rejecting the change. 234 00:16:29,940 --> 00:16:32,240 The downstream table was updated first, 235 00:16:32,480 --> 00:16:34,940 so its column could handle the wider type. 236 00:16:36,500 --> 00:16:38,980 And only then did our PR merge and deploy. 237 00:16:39,300 --> 00:16:40,080 No incident. 238 00:16:40,440 --> 00:16:42,840 The scripts in layer one said safe, 239 00:16:43,160 --> 00:16:46,500 the agent found the gap, and a human made the call. 240 00:16:49,320 --> 00:16:51,020 And that was not a one-time thing. 241 00:16:51,020 --> 00:16:53,440 Most changes can be cleared by the agent. 242 00:16:53,760 --> 00:16:58,480 When the checks pass and the agent is confident in its verdict, it approves on its own. 243 00:16:58,920 --> 00:17:05,100 But there's a whole category of changes where the agent never gets to say approve and no matter how confident it sounds. 244 00:17:06,740 --> 00:17:08,880 Grants touching sensitive data are one. 245 00:17:09,760 --> 00:17:14,400 Who can reach sensitive data is a decision we want a person to own every time. 246 00:17:15,940 --> 00:17:20,900 A migration that looks like it changes intent, not just mechanics is another. 247 00:17:21,840 --> 00:17:27,520 If a change alters what the data means rather than just how it is moved or stored, a person 248 00:17:27,520 --> 00:17:28,080 decides. 249 00:17:29,800 --> 00:17:34,120 And if the agent itself is not confident in its own verdict, that's the third. 250 00:17:35,360 --> 00:17:39,340 It does not get to round up to approve just because nothing concrete came back wrong. 251 00:17:39,340 --> 00:17:46,000 In all the three cases, its job is only ever to surface the risk and its reasoning. 252 00:17:46,400 --> 00:17:48,600 The decision stays with the person. 253 00:17:49,960 --> 00:17:51,240 That is one PR. 254 00:17:52,360 --> 00:17:56,740 Now how does this hold up across every PR we see and how does the reviewer get better 255 00:17:56,740 --> 00:17:57,340 over time? 256 00:17:59,800 --> 00:18:03,760 How do we know the flags are useful or whether a rule change helps? 257 00:18:03,760 --> 00:18:08,180 We start with the agent's verdict and its reasoning in the review comments. 258 00:18:09,600 --> 00:18:13,220 A person replies, agreeing or disagreeing and explaining why. 259 00:18:14,160 --> 00:18:18,120 That explanation is what makes the feedback useful rather than just a vote. 260 00:18:19,960 --> 00:18:21,940 Those replies are collected every day. 261 00:18:23,540 --> 00:18:27,620 Once a week, the on-call reads the feedback and updates the review rules by hand. 262 00:18:28,840 --> 00:18:30,180 Those rules shape the next review. 263 00:18:31,180 --> 00:18:32,700 I know, this is manual today. 264 00:18:32,700 --> 00:18:36,240 nothing retrains itself and no rule changes automatically. 265 00:18:36,800 --> 00:18:40,260 But we have an idea to let an incident investigation agent 266 00:18:40,820 --> 00:18:43,400 help with that weekly pass, but it is not built. 267 00:18:44,400 --> 00:18:47,260 Without the feedback, every rule change would be a guess. 268 00:18:47,740 --> 00:18:50,600 So let me show you an example where a human corrected 269 00:18:50,600 --> 00:18:54,080 the agent and we use that feedback to improve 270 00:18:54,080 --> 00:18:55,040 its next review. 271 00:18:57,320 --> 00:18:58,700 So here's a real one. 272 00:18:59,240 --> 00:19:00,840 And note the label in the corner. 273 00:19:00,840 --> 00:19:04,740 This is a separate example, not our schema change PR. 274 00:19:05,960 --> 00:19:09,660 A column was marked as personal information in the source database. 275 00:19:10,540 --> 00:19:15,420 But by the time it lands in the data lake, the ingestion pipeline has already encrypted it. 276 00:19:15,680 --> 00:19:22,200 So the PR proposed labeling it non-sensitive in the data lake, even though the upstream data is still PII. 277 00:19:23,420 --> 00:19:25,160 The agent flagged this. 278 00:19:25,580 --> 00:19:30,380 From where it stood, the upstream says PII, so a non-sensitive label looked wrong. 279 00:19:31,120 --> 00:19:35,880 The author pushed back and asked the agent to look at how similar columns on the other 280 00:19:35,880 --> 00:19:40,700 tables were already classified, instead of going purely off the upstream tagging. 281 00:19:42,700 --> 00:19:44,060 A human agreed with the author. 282 00:19:45,140 --> 00:19:49,300 The encryption before write made the non-sensitive label look okay for this column. 283 00:19:49,620 --> 00:19:54,860 And that kind of correction is exactly the signal that teaches the rules to get smarter. 284 00:19:54,860 --> 00:19:57,760 because the next time a similar column comes through, 285 00:19:58,040 --> 00:20:01,340 that context can stop a needless escalation. 286 00:20:02,240 --> 00:20:05,680 One caution though, this depends on the specific pipeline 287 00:20:05,680 --> 00:20:07,080 encrypting before the write. 288 00:20:07,320 --> 00:20:09,280 It's not a blanket rule that encryption 289 00:20:09,280 --> 00:20:11,700 always makes data non-sensitive. 290 00:20:14,260 --> 00:20:16,100 So how do we know whether the reviewer 291 00:20:16,100 --> 00:20:17,420 is actually doing a good job? 292 00:20:18,340 --> 00:20:20,200 We look at two kinds of mistakes. 293 00:20:20,780 --> 00:20:23,900 Bad PRs it approved and fine PRs 294 00:20:23,900 --> 00:20:25,920 it unnecessarily send to a person. 295 00:20:26,500 --> 00:20:29,620 That matrix separates four possibilities. 296 00:20:31,500 --> 00:20:33,900 A fine PR approved and a bad PR escalated 297 00:20:33,900 --> 00:20:35,460 are the two correct outcomes. 298 00:20:36,900 --> 00:20:38,600 A bad PR approved is a miss. 299 00:20:39,080 --> 00:20:41,360 That's dangerous because the change ships. 300 00:20:42,860 --> 00:20:45,420 A fine PR escalated is a false escalation 301 00:20:46,280 --> 00:20:47,900 that costs a person's time. 302 00:20:48,400 --> 00:20:51,440 A gate that escalates everything may avoid misses, 303 00:20:51,440 --> 00:20:53,800 but it has not reduced the review load. 304 00:20:55,580 --> 00:20:59,400 As of August 31st, 2026, our miss rate was about 2%. 305 00:21:01,420 --> 00:21:03,100 Mostly from the gaps in the rules 306 00:21:03,100 --> 00:21:04,840 rather than the failures of the judgment. 307 00:21:05,520 --> 00:21:08,440 We don't have a clean single false escalation number, 308 00:21:09,360 --> 00:21:11,300 but the dominant cause is missing context 309 00:21:11,300 --> 00:21:14,480 like the encrypted column example that I just gave you. 310 00:21:16,160 --> 00:21:17,720 Misses give us new rules 311 00:21:17,720 --> 00:21:21,100 and false escalation show us context the reviewer needs. 312 00:21:21,980 --> 00:21:23,500 So here is one of those misses. 313 00:21:23,920 --> 00:21:25,140 Again, know the label. 314 00:21:25,280 --> 00:21:27,960 This is a separate example, not our schema change PR. 315 00:21:29,540 --> 00:21:32,180 We have an open copy of some data. 316 00:21:32,980 --> 00:21:35,440 Some of its columns are about to be re-tagged. 317 00:21:35,720 --> 00:21:37,240 And there is a downstream reader 318 00:21:37,240 --> 00:21:39,520 that reads those columns from the open copy. 319 00:21:41,380 --> 00:21:44,200 Privacy re-tagged those columns as personal information. 320 00:21:44,840 --> 00:21:48,360 When that happens, they move into a more restricted copy 321 00:21:48,360 --> 00:21:50,800 of the data and they are gone from the open copy. 322 00:21:51,940 --> 00:21:54,580 The re-tagged PR looked fine and it merged. 323 00:21:54,940 --> 00:21:59,160 But what nobody checked was whether anything downstream was still reading those columns 324 00:21:59,160 --> 00:22:00,580 from the old open copy. 325 00:22:01,620 --> 00:22:01,940 Something was. 326 00:22:02,340 --> 00:22:05,360 And once the columns were gone, that reader broke. 327 00:22:05,680 --> 00:22:06,500 That's a miss. 328 00:22:06,940 --> 00:22:09,720 A PR that should have been escalated got approved. 329 00:22:11,640 --> 00:22:12,960 It wasn't a failure of judgment. 330 00:22:13,220 --> 00:22:14,480 It was a gap in the rules. 331 00:22:14,960 --> 00:22:17,200 A corner case nobody had written down. 332 00:22:17,540 --> 00:22:18,980 So it became a new rule. 333 00:22:18,980 --> 00:22:24,140 When a column gets re-tagged as sensitive, search the code base for anyone still reading 334 00:22:24,140 --> 00:22:26,200 it from the copy that's about to lose it. 335 00:22:26,580 --> 00:22:28,600 And that's how the rules get better. 336 00:22:29,580 --> 00:22:32,540 The misses get found and get turned into new rules. 337 00:22:34,760 --> 00:22:36,380 So let's pull it all together. 338 00:22:37,020 --> 00:22:38,860 Remember the pull request from the start? 339 00:22:39,100 --> 00:22:40,740 The one that was green in CI? 340 00:22:41,600 --> 00:22:43,140 Three things happened to it. 341 00:22:44,440 --> 00:22:48,200 First, scripts proved what a source of truth could prove. 342 00:22:49,060 --> 00:22:53,580 The contracts, the tags, the registered lineage and performance all passed. 343 00:22:54,320 --> 00:22:58,540 Second, the agent read the code and found what the scripts could not see. 344 00:22:58,860 --> 00:23:03,660 A job that lineage never knew about, quietly waiting to fail on a wider column. 345 00:23:04,020 --> 00:23:06,360 It flagged the whole chain and escalated. 346 00:23:07,060 --> 00:23:09,900 Third, a person confirmed the job was live. 347 00:23:09,900 --> 00:23:15,260 the downstream table was fixed first and only then did the PR merge. No incident. 348 00:23:15,880 --> 00:23:22,580 And behind all of that, the reviewer keeps getting better because every correction a person makes 349 00:23:22,580 --> 00:23:29,460 can become a rule. These results are as of August 31st, 2026. So you might ask, Chetis, 350 00:23:29,680 --> 00:23:36,620 does it all work? Yes. About 60% of the data pipeline PRs merge without a human review. 351 00:23:37,540 --> 00:23:41,600 We have seen about 75% less manual review effort. 352 00:23:42,000 --> 00:23:49,960 The agent's specific comments and reasoning help focus the review rather than making a person reconstruct every risk from scratch. 353 00:23:51,240 --> 00:23:56,280 The time saved comes to about 132 engineering hours a month. 354 00:23:56,780 --> 00:24:00,480 Incidents tied to these PRs are down about 30%. 355 00:24:01,400 --> 00:24:04,020 And that does not mean human-free review. 356 00:24:04,020 --> 00:24:07,740 The protected decisions still reach a person by design. 357 00:24:08,160 --> 00:24:12,320 The goal is less review work, not removing accountability. 358 00:24:12,900 --> 00:24:14,360 And those are the results. 359 00:24:14,860 --> 00:24:17,540 Now let's look at the architecture choices we got wrong. 360 00:24:20,220 --> 00:24:21,660 Here is what we got wrong. 361 00:24:22,260 --> 00:24:25,480 Today, layer one and layer two are separate systems, 362 00:24:25,880 --> 00:24:28,440 triggered independently and running in parallel. 363 00:24:29,160 --> 00:24:31,220 Each produces its own result. 364 00:24:31,220 --> 00:24:34,140 that seemed like an obvious design when we built it. 365 00:24:35,420 --> 00:24:39,040 But neither sees the others pass, fail or outage state. 366 00:24:40,060 --> 00:24:43,000 Layer two knows what the scripts are configured to cover, 367 00:24:43,220 --> 00:24:45,040 not what happened when they ran. 368 00:24:46,040 --> 00:24:48,920 Merge still needs passing checks and an approving review, 369 00:24:49,240 --> 00:24:52,120 but no single system has a full runtime picture. 370 00:24:53,340 --> 00:24:57,060 The agent also cannot call live layer one checks. 371 00:24:57,060 --> 00:25:01,160 Its tools only read, search and list the repository files. 372 00:25:01,440 --> 00:25:07,200 If a judgment needs a live fact from a live production system, it cannot fetch that fact 373 00:25:07,200 --> 00:25:07,680 itself. 374 00:25:09,680 --> 00:25:12,520 When one site cannot finish, we wait for a human. 375 00:25:12,800 --> 00:25:14,600 There is no automatic fallback. 376 00:25:15,620 --> 00:25:19,620 These are current limitations and they motivate the proposal on the next slide. 377 00:25:21,200 --> 00:25:22,900 Platform team is exploring one entry point. 378 00:25:23,740 --> 00:25:25,920 This is a proposal and not what's running today. 379 00:25:25,920 --> 00:25:30,880 The left panel is the current design and the right is where we want to go. 380 00:25:31,580 --> 00:25:38,060 A classifier looks at the whole PR then decides which changes to review separately and which to review together. 381 00:25:39,040 --> 00:25:43,080 This is a whole PR decision, not independent routing of each file. 382 00:25:44,180 --> 00:25:50,120 The agent would also get the layer one checks as callable tools alongside its repository tools. 383 00:25:50,120 --> 00:25:55,700 when judgment needs a live fact it could request the appropriate check instead of being limited to 384 00:25:55,700 --> 00:26:02,940 code alone the aim here is one system with the whole PR in view we haven't built this yet so 385 00:26:02,940 --> 00:26:10,920 it's a direction not reserved let me close with what we would carry forward so here is the whole 386 00:26:10,920 --> 00:26:17,800 talk in three lines deterministic evaluations first if something can be proven against a source 387 00:26:17,800 --> 00:26:23,800 of truth, let a script prove it. Don't spend a judgment call on a yes or no fact. It's 388 00:26:23,800 --> 00:26:29,800 not just cleaner, it's cheaper too. Every fact a script proves is review work that never 389 00:26:29,800 --> 00:26:38,160 needs AI. So the cost of running this stays down. Flag never clear. Not for every PR, 390 00:26:38,360 --> 00:26:43,600 but for the changes that actually scare people like sensitive grants, migration intent, and 391 00:26:43,600 --> 00:26:50,120 low confidence verdicts. There, the agent's job stops at surfacing the risk and the reasoning 392 00:26:50,120 --> 00:26:58,740 behind it. The decision stays with the person every time. Count the false escalations. A gate 393 00:26:58,740 --> 00:27:04,600 that escalates everything is not actually saving anyone any time. It's just moving the work around. 394 00:27:05,200 --> 00:27:10,920 Measure that cost the same way you measure a miss. So I started with a pull request. There 395 00:27:10,920 --> 00:27:12,880 there was green NCI and a problem. 396 00:27:13,280 --> 00:27:16,320 PRs arriving at machine speed and too few people 397 00:27:16,320 --> 00:27:18,020 with the context to review them safely. 398 00:27:18,540 --> 00:27:21,660 We built a hybrid reviewer, scripts for the facts, 399 00:27:21,940 --> 00:27:25,040 an agent for judgment and people for the protected calls. 400 00:27:25,400 --> 00:27:27,720 We walked one PR through all three, 401 00:27:28,240 --> 00:27:30,760 saw it get lighter without going human free 402 00:27:31,460 --> 00:27:33,280 and saw how it learns from the corrections. 403 00:27:33,700 --> 00:27:35,340 We told you what we got wrong 404 00:27:35,860 --> 00:27:37,260 and we left you with three lessons. 405 00:27:37,960 --> 00:27:39,780 prove what you can cheaply, 406 00:27:40,500 --> 00:27:42,860 flag and never clear the scary changes, 407 00:27:43,200 --> 00:27:45,280 and count the false escalations. 408 00:27:46,120 --> 00:27:47,780 So who reviews the bots? 409 00:27:48,300 --> 00:27:50,780 Scripts for everything a source of truth can prove, 410 00:27:51,120 --> 00:27:54,020 an agent for everything that needs context and judgment, 411 00:27:54,460 --> 00:27:58,400 and humans for the calls only they get to make. 412 00:27:58,640 --> 00:27:59,620 Thank you for listening.