# 60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood Chetas Joshi, Robinhood [Talk page](https://ai.engineer/talks/MQiojLnK-E0-60-our-data-prs-now-merge-no) · [Video](https://www.youtube.com/watch?v=MQiojLnK-E0) ## 60% of Our Data PRs Now Merge With No Human Review At Robinhood, a PR to widen an integer column passed every CI check. Following the value one table farther explains what a review agent can catch and why this particular merge still needed an engineer. ### The queue forms around the people who know the table [0:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=0s) A schema change passes its tests and the linter. Is it safe to merge? A pipeline can produce correct data too late to meet its freshness deadline. A change can break a consumer three teams away or give someone access to sensitive data. For Chetas Joshi’s data platform team at Robinhood, answering the merge question means understanding who depends on the changed code and what they expect from it. Coding assistants could open a PR for a new query, schema change or access grant in minutes. Reviews of pipeline code and configuration also landed with the data platform team. CI and human review already caught broken code, inefficient queries and data leaks. What slowed the queue was waiting for one of the few engineers who knew who read a table, how large it was and what sensitive data it held. The number of people carrying that context stayed flat as the PRs piled up. Robinhood split the work by the kind of answer each reviewer could give. A script can look up a fact in a schema registry or catalog and block a PR if the check fails. An agent can read the code, assess risk, and either approve or ask a person to decide. Its rules tell it which questions the scripts cover, saving the cost of asking a model to reconstruct a fact the team can look up. A merge needs both passing checks and an approving review. ### What four green checks establish about one schema change [5:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=324s) An ingestion pipeline reads an upstream source table and writes to a table in the data lake. One PR makes two changes. The source has gained a column, and the pipeline will bring it into the lake. A different column, already in the lake table, will widen from `INTEGER` to `BIGINT` so it can hold larger values. The first edit raises a question about the source. The second raises a question about every job that reads the widened column. - **Data contract.** A script compares the proposed schema and keys with the schema registry. The registry confirms that the new column exists upstream. - **Required tags.** Every table needs an owner, and personally identifiable information (PII) needs a label in the catalog. A direct catalog lookup confirms both for this table. - **Downstream dependencies.** The lineage tool records which jobs read a column. A script walks every job registered for the widened column and checks whether the type change breaks it. - **Performance.** CI runs the change in staging with mock data at production volume. The run checks whether the cluster is sized for the job and whether the job can finish in time to meet its freshness service-level agreement. A failed check blocks the PR until someone fixes it. Here all four passed. How far downstream did that green result reach? Every registered consumer could handle the wider type. The word *registered* matters. The script starts with the lineage tool’s list, so a job missing from that list contributes nothing to the result. ### A repository search finds the narrower destination [10:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=626s) The agent starts with the diff. The PR description and commit messages explain what the author intended, but a finding in the code takes precedence over a claim that the change is safe. The agent can list, search and read files across the repository. It has no direct access to the live catalog, lineage tool or production state. The type of change tells the agent what to look for. In a query, an unpartitioned cross join can return the right answer while hammering a cluster. In an access grant, an ordinary-looking edit can open sensitive data. For a schema change, the rules direct the agent into pipeline definitions and access grants to find consumers beyond those recorded in lineage. For this PR, the question was where a value would go after a job read it from the `BIGINT` column. The search found a job defined in the repository that had never been registered in lineage. It read the widened column and copied the value to another table, where the receiving column was still `INTEGER`. Values within the original range would continue to fit. The first value large enough to need the wider range would reach that last write and fail. The agent’s review comment traced the path from the changed column through the job to the destination. Each query engine has to publish its own lineage. If an engine omits a relationship, a repository search can still recover it from a job definition. The search has limits of its own. An ad hoc query someone runs by hand, or a dashboard defined outside the codebase, leaves the agent nothing to read. When a search comes up empty, the agent reports where it looked and that it found no consumers there. Where would the first out-of-range value fail? Follow the data out of the lake table. The lineage check covered the registered branch. Following the other branch through repository code revealed a write to a narrower column. - Lake table — The PR widens an existing column from INTEGER to BIGINT. - Registered consumers — Every reader listed for the column passes the lineage check. - Unregistered job — The repository contains the job that copies this column onward. - Downstream table — Its INTEGER column cannot store a value outside the original range. - Lake table → Registered consumers (Read the widened column) - Lake table → Unregistered job (Reads the same column) - Unregistered job → Downstream table (Writes the value) ### The reviewer changes the rollout order [14:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=876s) The agent attaches its reasoning to specific lines and scores the severity of what it found. For an ordinary change, a confident review with no serious finding can supply the approval needed alongside passing CI. For the widening PR, the agent scored the downstream write as high risk and escalated. The engineer who picked it up already knew which job to investigate. The open question was whether that job actually ran in production. The human reviewer confirmed the consumer was live. The PR waited while the team widened the receiving column in the downstream table. Once that destination could hold the larger values, the original PR merged and deployed without incident. Some decisions always go to a person. In each case, the agent passes along its findings and reasoning. - **Sensitive access grants.** A person decides who may reach sensitive data, however confident the agent sounds. - **Migrations that appear to change meaning.** If a migration changes what the data represents, a person decides whether that change of intent is acceptable. - **Low-confidence verdicts.** Even when the search turns up no concrete defect, uncertainty sends the review to a person. ### An encrypted column changes the review question [17:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1070s) A person can reply to the agent’s comment, agree or disagree, and explain why. The explanation gives the on-call engineer something specific to change. The team collects replies daily. Once a week the on-call engineer reads them and edits the review rules by hand. Those edits shape subsequent reviews without retraining the model. Using an incident-investigation agent to help with the weekly pass was still an idea. In a separate PR, a source database marked a column as PII. The ingestion pipeline encrypted the column before writing it to the lake, and the author proposed a nonsensitive label for the lake copy. The agent saw a disagreement between the upstream and downstream labels and flagged it. The author pointed to comparable columns in other tables and asked the agent to consider how those columns were classified. A human reviewer agreed with the author about this column. What happened between the source and the lake mattered to the decision. A future review could check where the pipeline applies encryption and how the team classifies comparable output before escalating a label mismatch. The decision here depended on encryption before this particular write. Encryption alone cannot settle the classification of another dataset. ### A false alarm costs time. A missed reader breaks. [20:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1214s) How can the team tell whether a review rule helps? Cross the agent’s decision with the decision the PR needed. Approving a safe change and escalating a risky one are both correct. Approving a risky change is a miss that lets the change ship. Escalating a safe change unnecessarily consumes an engineer’s time. A reviewer could avoid every unsafe approval by forwarding every PR, leaving the original queue intact. As of August 31, 2026, Joshi reported a miss rate of about 2%, without specifying the denominator. The team traced most misses to gaps in the review rules. It had no single reliable rate for false escalations, but the most common cause was missing context of the kind the encrypted-column example exposed. One miss started as a privacy correction. A downstream job read several columns from an open copy of a dataset. When the privacy team retagged the columns as personal information, they moved to a restricted copy and disappeared from the open one. The retag itself looked reasonable. The reviewer approved it, the PR merged, and the job broke when it tried to read the missing columns. The team added a rule with a specific trigger and search target. When a column is retagged as sensitive, search the codebase for readers that still reference that column in the copy about to lose it. That gives the reviewer a way to flag a broken read even when the privacy classification is correct. ### Even escalated PRs took less work to review [22:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1354s) As of August 31, 2026, about 60% of Robinhood’s data pipeline PRs merged without human review. The team reported about 75% less manual review effort, saving roughly 132 engineering hours a month. The widening PR illustrates how an escalation could also save work. The engineer could go straight to checking whether a particular consumer was live because the agent had already traced the dependency. The team also observed about 30% fewer incidents tied to these PRs. That figure describes the team’s experience with the hybrid reviewer in place, including the engineers who decided protected cases and investigated escalations. ### Two independent reviewers and a proposal for one entry point [24:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1460s) The two layers in the deployed design run in parallel. Separate triggers start the scripts and the agent on the same PR, and each produces its own result. The agent knows which checks are configured, but it cannot see whether a particular check passed, failed or hit an outage. Nor can it call a check when its investigation turns up a question about production. Merge still requires passing CI and approval. If either system cannot finish, the team waits for a human. What would a single entry point change? In the proposed design, a classifier would examine the whole PR before deciding which changes to review together and which to handle separately. That would let the grouping decision account for relationships across files. The agent would also get callable deterministic checks alongside its repository tools, so it could request a live fact when the code raised a question. The team was exploring this design. The reported savings came from the parallel systems already in use. Joshi would carry three decisions into that redesign. Keep yes-or-no lookups in scripts, where each lookup saves model work. Give people the final call on sensitive grants, migrations that change meaning and uncertain verdicts. And count unnecessary escalations alongside misses. Each needless referral spends the time of the same few engineers the reviewer was built to free up. ## Key takeaways - A passing lineage check establishes compatibility for registered readers. A repository search can reveal another reader, and an engineer can check whether it is active in production. - An escalation can save engineering time when it names the changed line, traces the dependency and leaves the reviewer a specific question to resolve. - Unsafe approvals can reveal a missing search rule. Unnecessary escalations can reveal missing context. A human explanation helps the on-call engineer decide which change to make. ## Slides ### Who Reviews the Bots? A schema change can pass CI and still break a consumer three teams away. Robinhood’s Chetas Joshi introduces a hybrid PR reviewer to help decide whether a bot’s change is safe to merge. [0:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=0s) [Beside the subtitle “Building a Hybrid PR Reviewer Agent,” a schema-change pull request marked “CI passed” leads to the question “Safe to merge?”](https://assets.ai.engineer/sites/aiecode2025/sha256/fc70edc9398480928c0fcc9a5f3351dbbe4f46f31713f4787c006baa683812c2.webp) Who Reviews the Bots? Building a Hybrid PR Reviewer Agent Chetas Joshi Engineer, Data Platform · Robinhood AIE CODE Summit 2026 PULL REQUEST Schema change CI passed ? Safe to merge? ### AI now opens PRs at machine speed An AI coding assistant can open a data-pipeline pull request in minutes. At Robinhood, schema changes and data access grants land in the same review queue. When requests pile up, the Data Platform team handles the backlog. [2:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=121s) [Arrows connect an AI coding assistant to a stack of data-pipeline pull requests and then to the Data Platform review team. The requests cover schema changes, pipeline code, configuration changes and data access grants.](https://assets.ai.engineer/sites/aiecode2025/sha256/7d96bc721380216c6b876bc882362ef7216d30a9a7a43036d0d5e6b1b9dffceb.webp) THE PROBLEM AI now opens PRs at machine speed WHAT AI coding assistant opens a PR in minutes Data-pipeline PR Schema changes Pipeline code Config changes Data access grants WHO REVIEW OWNED BY Data Platform My team at Robinhood reviews these changes Who Reviews the Bots? 2/26 ### Platform context is the bottleneck CI checks catch bugs as AI keeps opening pull requests. Deciding whether a change is safe to merge still depends on engineers who know who reads the affected table and how sensitive its data is. [2:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=145s) [A grid of incoming AI-opened pull requests points to a box labeled “CI checks,” with the caption “catch real bugs.”](https://assets.ai.engineer/sites/aiecode2025/sha256/d7aa885ec2fc0d30f2276cdd72fc73b1a3debd684e3a2f7c465350ae81e48deb.webp) THE PROBLEM · WHY Platform context is the bottleneck PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR AI-opened PRs keep arriving CI checks catch real bugs Who Reviews the Bots? 3/26 ### Platform context is the bottleneck CI catches bugs, but AI-opened pull requests still queue up for the few reviewers who know who reads a table and how sensitive its data is. The team wants an AI reviewer that can keep pace with incoming requests. [2:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=170s) [AI-opened pull requests pass through CI checks and enter a funnel labeled “the queue” before reaching reviewers with platform context. A goal box calls for an AI reviewer that keeps pace with incoming requests.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f7b38d94070958f7d3aa672b39bb867695b35f3fea4ddf87cb8e366b6e3ff9c.webp) THE PROBLEM · WHY Platform context is the bottleneck PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR PR AI-opened PRs keep arriving CI checks catch real bugs the queue Reviewers with platform context catch real bugs too, at human speed PRs now arrive at machine speed. Review that needs platform context still moves at human speed. THE GOAL Build a PR reviewer, with AI, that keeps pace with how fast PRs are opened. Who Reviews the Bots? 3/26 ### Use AI only where proof runs out Scripts check a pull request against sources like the schema registry. The agent judges the remaining risks and either approves the request or sends it to a person. Merging requires both passing checks and approval. [4:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=279s) [Scripts check a pull request against the schema registry or data catalog and block it on failure. An agent judges the remaining questions and approves or escalates to a person. A callout links this division of work to lower AI costs.](https://assets.ai.engineer/sites/aiecode2025/sha256/49e9310df3733349460e05a68e61acde6d3e7df43497629338fbb2bf1574f7cc.webp) THE HYBRID DESIGN Use AI only where proof runs out Scripts for facts. An agent for judgment. People for the protected calls. PR LAYER 1 · DETERMINISTIC SCRIPTS Prove facts against a source of truth like the schema registry or the data catalog Fail: CI goes red, the PR is blocked LAYER 2 · AGENT REVIEW Judge what scripts can't prove rules scoped to what Layer 1 doesn't cover Approve, or escalate to a human AI does only the judgment work, not checks a script can prove, which keeps AI cost lower. Who Reviews the Bots? 4/26 ### Roadmap Scripts check facts, an agent exercises judgment, and people make the protected decisions. Joshi then traces how the reviewer learns from human feedback, what results the team has seen, and what it learned. [4:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=287s) [Five connected markers list the reviewer, human boundaries, how it learns, results, and lessons. Below them, the slide assigns fact checks to scripts, judgment to an agent, and protected decisions to people.](https://assets.ai.engineer/sites/aiecode2025/sha256/e504f239b50e0ba5b688f5dc45f2a005cc13345d34c4dffa6d92a3eceecba118.webp) WHERE WE'RE GOING Roadmap We've covered the problem and the approach. The reviewer Human boundaries How it learns Results Lessons Three things to listen for: scripts for facts, an agent for judgment, and people for the protected calls. Who Reviews the Bots? 5/26 ### One PR, two schema changes The pull request updates the ingestion pipeline to bring a new upstream column into the data lake table. It also widens an existing column in that table from int to bigint so it can hold larger values. [5:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=330s) [An ingestion pipeline carries a new column from an upstream table into a data lake table. The same pull request also widens an existing lake table column from int to bigint.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d5d4fb846e9763dffaba3d8024394a9730540433841bc1a839fa68dc93d58fd.webp) One PR, two schema changes THE RUNNING EXAMPLE RUNNING PR · PR OPENED SOURCE TABLE Upstream system + new column Ingestion pipeline DATA LAKE TABLE Our table + new column widened: int → bigint CHANGE 1 · ADD A COLUMN The source gains a new column, and ingestion now brings it into our lake table. CHANGE 2 · WIDEN A COLUMN A column the lake table already has goes from integer to big integer. Who Reviews the Bots? 6/26 ### One PR, two schema changes The PR brings a new upstream column into the lake table and widens an existing column from int to bigint. The lineage tool lists registered readers of the widened column. Who else might read it? [6:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=383s) [The pipeline diagram shows a new column arriving in the lake table and an existing column widening from int to bigint. Arrows lead from the lake table to registered consumers recorded in lineage and to a dashed box asking whether anyone else reads the widened column.](https://assets.ai.engineer/sites/aiecode2025/sha256/752c727b0f3cca0b6c98a500f213092d5e5eac9fb81a16de8771196efa1c74d5.webp) One PR, two schema changes THE RUNNING EXAMPLE RUNNING PR · PR OPENED SOURCE TABLE Upstream system + new column Ingestion pipeline DATA LAKE TABLE Our table + new column widened: int → bigint WHO READS THE WIDENED COLUMN? Registered consumers recorded in lineage Any others? CHANGE 1 · ADD A COLUMN The source gains a new column, and ingestion now brings it into our lake table. CHANGE 2 · WIDEN A COLUMN A column the lake table already has goes from integer to big integer. Who Reviews the Bots? 6/26 ### Layer 1 checks facts in live systems Scripts use the schema registry to confirm the new column exists upstream and the data catalog to verify the table’s owner and PII labels. Both checks pass for this PR. If either fails, CI blocks the merge until the issue is fixed. [6:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=407s) [A pipeline diagram traces the new column from the upstream system to the lake table and marks the widened integer there. Below it are three headings for the check, the source of truth, and the running pull request. The space beneath the headings is empty.](https://assets.ai.engineer/sites/aiecode2025/sha256/f47907dff876fb2f2f496cbaff3a31f2293d2972b1079d0b3e71f4159d34c788.webp) Layer 1 checks facts in live systems LAYER 1 · SCRIPTS RUNNING PR · CONTRACT + TAGS Upstream system + new column → Ingestion pipeline → Our table + new column widened: int → bigint THE CHECK SOURCE OF TRUTH OUR RUNNING PR Who Reviews the Bots? 7/26 [7:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=466s) [The schema registry confirms the new column exists upstream. The data catalog confirms ownership and PII labeling. Both checks pass for this pull request. A banner says CI blocks the pull request if either check fails.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f9725c5e15241d8b0f6c2c6eacf9ce3d2703e0bac8c8105d7c6186a84e7aee3.webp) Layer 1 checks facts in live systems LAYER 1 · SCRIPTS RUNNING PR · CONTRACT + TAGS Upstream system + new column → Ingestion pipeline → Our table + new column widened: int → bigint THE CHECK SOURCE OF TRUTH OUR RUNNING PR DATA CONTRACTS Schema and keys compared with the registry Schema registry PASS The new column already exists upstream, as registered REQUIRED TAGS Owner and PII labels must be present on the table Data catalog PASS Our table has an owner, and its PII is labeled If either check fails, CI goes red and the PR is blocked until it's fixed. No judgment, no AI. Who Reviews the Bots? 7/26 ### Checking downstream impact and performance When a column widens, the script checks only the downstream jobs recorded in lineage. Before merge, a staging run uses mock data at production volume to check cluster sizing and whether the job meets its freshness deadline. [8:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=529s) [A widened column points to two jobs recorded in lineage. Alongside it, a staging test uses mocked data at production volume to check cluster sizing and whether the job meets its freshness target.](https://assets.ai.engineer/sites/aiecode2025/sha256/25b00c91c6044a338adca85f5f4f30dfe5e3f3e858a7234596d67b58a68146d7.webp) Checking downstream impact and performance LAYER 1 · SCRIPTS RUNNING PR · LINEAGE CHECK Upstream system + new column → Ingestion pipeline → Our table + new column widened: int → bigint LINEAGE BLAST RADIUS What breaks downstream? Widened column RECORDED IN LINEAGE Registered job Registered job Walks every downstream job the lineage tool has on record. Its view is only as complete as that record. STAGING PERFORMANCE TEST Will it run in time? Runs in staging with: Production data volume Mocked data Cluster sized right? Freshness SLA met? Catches cluster sizing and freshness SLA risk before the PR merges. Who Reviews the Bots? 8/26 ### The agent starts with intent, then reads the code The PR description and commit messages help the agent interpret the diff. It traces the change into pipeline code and access grants across the repo. If the code contradicts the author’s account, the code wins. Layer 1 handles live queries. [10:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=658s) [The diff, PR description and commit messages feed an agent that reads, searches and lists repository files. Two callouts explain that intent cannot dismiss a finding and that Layer 1 handles live checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/dae88669c97e72967165605cebfd143aa94580e372b86498cf702c8161d0cf8d.webp) LAYER 2 · THE AGENT The agent starts with intent, then reads the code The diff what changed PR description what the author meant Commit messages how it got there Agent FILE TOOLS · THE REPO ONLY read search list Pipeline code and access grants across the repo, so it can follow a change beyond the lines in the diff Intent explains a change. It never explains away a real finding. No live queries Not the catalog, lineage tool, or production state. Live facts stay Layer 1's job. Who Reviews the Bots? 10/26 ### The rules cover gaps, not facts scripts prove The agent uses different rules for different changes. For schema changes, it searches pipeline definitions and access grants in the repository to find downstream consumers missing from lineage records. [11:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=707s) [Below the three rule categories, an arrow labeled “code walk” leads from pipeline definitions and access grants in the repository to consumers that lineage never recorded.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5128cd5d2f3479f4b035112a44a1bdcc6ecc4a9ae36b994187636b3635811ea.webp) LAYER 2 · THE AGENT The rules cover gaps, not facts scripts prove AGENT RULES · SET PER CHANGE TYPE, KNOWING WHAT LAYER 1 COVERS Query anti-patterns Sensitive data asks Consumers outside lineage WHAT THE AGENT DOES INSTEAD THE REPO Pipeline definitions Access grants code walk Consumers lineage never recorded Slow, tedious work for a person. The agent does it on every relevant diff. Who Reviews the Bots? 11/26 ### Why lineage missed it Query engines A and B publish their dependencies to the lineage tool. Engine C’s dashed red arrow marks a dependency it never published, so a downstream job that relies on the table can be missing from the catalog. [13:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=794s) [Solid arrows connect query engines A and B to a lineage tool. Engine C has a dashed red arrow marked “not published.” Each engine must publish its own lineage.](https://assets.ai.engineer/sites/aiecode2025/sha256/f95ceb03f42ebd9ef95d24f08aee41b49bbf4aea22fac2699aa7477cc5ed3936.webp) Why lineage missed it LAYER 2 · THE AGENT WHY LINEAGE HAS GAPS Query engine A Query engine B Query engine C Lineage tool not published Each engine has to publish its own lineage. No single catalog tracks everything. Who Reviews the Bots? 13/26 ### “None found” doesn't mean “none exist” The agent searched the repository and found an unregistered job that reads the table. Ad hoc queries and dashboards can read the same table outside the repo, so an empty search result applies only to the places the agent checked. [13:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=839s) [For a table change, the agent can search pipeline code, access grants and jobs in the repository. External ad hoc queries and dashboards remain outside its view. The green callout limits a negative finding to the places searched.](https://assets.ai.engineer/sites/aiecode2025/sha256/f4e1cfe45b534c866f520f03793734f0dbeaae135b3b25c25a555620321b5d5e.webp) “None found” doesn't mean “none exist” LAYER 2 · LIMITS RUNNING PR · CODE WALK LIMITS Upstream system + new column → Ingestion pipeline → Our table + new column widened: int → bigint INSIDE THE REPO · THE AGENT CAN SEARCH Pipeline code Access grants Jobs that read our tables This is where it found our unregistered job. OUTSIDE THE REPO · INVISIBLE TO IT Ad hoc queries Dashboards A consumer here never shows up in a repo search. WHAT THE AGENT SAYS “No consumers found in the places I searched.” WHAT IT NEVER CLAIMS “All clear. No consumers exist.” Who Reviews the Bots? 14/26 [14:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=862s) [Two panels show what a repository search covers. The agent can inspect pipeline code, access grants and jobs that read the tables. Ad hoc queries and dashboards can use the same tables outside its view.](https://assets.ai.engineer/sites/aiecode2025/sha256/11e4c0f06f362f1aaef52d4977ad6fbd7e2644bc5f9926152b502dce27d39465.webp) “None found” doesn't mean “none exist” LAYER 2 · LIMITS RUNNING PR · CODE WALK LIMITS Upstream system + new column → Ingestion pipeline → Our table + new column widened: int → bigint INSIDE THE REPO · THE AGENT CAN SEARCH Pipeline code Access grants Jobs that read our tables This is where it found our unregistered job. OUTSIDE THE REPO · INVISIBLE TO IT Ad hoc queries Dashboards A consumer here never shows up in a repo search. WHAT THE AGENT SAYS “No consumers found in the places I searched.” WHAT IT NEVER CLAIMS “All clear. No consumers exist.” Who Reviews the Bots? 14/26 ### A review ends with a decision The agent scores the risk. If it finds no serious risk, it can approve. Otherwise, a human gets the line comments and the agent’s reasoning. The pull request merges when Layer 1 checks pass and either the agent or the human approves. [15:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=904s) [The agent comments on specific lines, explains its reasoning and scores the risk. It either approves or escalates to a person. A pull request merges when Layer 1 checks pass and the agent or a human reviewer approves.](https://assets.ai.engineer/sites/aiecode2025/sha256/5006e71d7f6e634258091c46aa4c715cbdb1df2f71ed61e74dd45faa28784c48.webp) A review ends with a decision LAYER 2 · THE AGENT'S OUTPUT Agent reviews the change REVIEW COMMENTS On the specific lines that matter each with its reasoning RISK SCORE How severe is what it found? APPROVE no serious risk found ESCALATE TO A HUMAN with context to investigate WHEN DOES A PR MERGE? Layer 1 checks pass + The review approves by the agent, or by a person after escalation → PR merges Who Reviews the Bots? 15/26 ### Update the downstream first, then merge The agent flagged an unregistered job that could fail when writing bigint values to an int column. A reviewer confirmed the job ran in production, so the team widened the destination column to bigint before merging the PR. [15:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=937s) [A diagram traces a widened column to an unregistered job. Three boxes show the agent flagging a write-failure risk, a person confirming the job runs in production, and the downstream table’s column being widened to bigint.](https://assets.ai.engineer/sites/aiecode2025/sha256/b189cea2cb4d778a31b53b7a5073a69071fee7e4ad6c0c3491acf14758a6baa9.webp) Update the downstream first, then merge THE RUNNING EXAMPLE RUNNING PR · HUMAN DECISION SOURCE TABLE Upstream system + new column Ingestion pipeline DATA LAKE TABLE Our table + new column widened: int → bigint WHO READS THE WIDENED COLUMN? Registered consumers all look fine Unregistered job target widened first AGENT ESCALATES Write-failure risk, flagged with its reasoning A PERSON CHECKS The job is live in production. The agent can't see that. DOWNSTREAM FIRST The other table's column is widened to bigint Who Reviews the Bots? 16/26 ### Update the downstream first, then merge The agent flagged an unregistered job whose writes could fail. A reviewer confirmed the job ran in production. The team widened its target column to bigint first, then merged the pull request and deployed without incident. [16:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=995s) [Four boxes trace the example from a flagged write-failure risk through confirmation that the downstream job is live. Its target column is widened to bigint first. The final box says the pull request then merges and deployment finishes without incident.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ffb98603fbf495d1d5951c7c17320633fe2c19f5b93e7f6b30c861df7cd1706.webp) Update the downstream first, then merge THE RUNNING EXAMPLE RUNNING PR · HUMAN DECISION SOURCE TABLE Upstream system + new column Ingestion pipeline DATA LAKE TABLE Our table + new column widened: int → bigint WHO READS THE WIDENED COLUMN? Registered consumers all look fine Unregistered job target widened first AGENT ESCALATES Write-failure risk, flagged with its reasoning A PERSON CHECKS The job is live in production. The agent can't see that. DOWNSTREAM FIRST The other table's column is widened to bigint THEN OUR PR MERGES Deployed safely. No incident. Who Reviews the Bots? 16/26 ### Some calls always stay with a person The agent flags grants of access to sensitive data and migrations that change what the data means. A person decides both. The agent can approve other changes when the checks pass and it is confident in its verdict. [17:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1043s) [Two amber cards assign sensitive data grants and changes to the meaning of data to a human reviewer. A green panel says the agent can approve other changes when checks pass and it is confident in its verdict.](https://assets.ai.engineer/sites/aiecode2025/sha256/54ed0c3e526c748110dc5576ea375553e0a1c956c1aaaad84f5de315f6b0e813.webp) HUMAN BOUNDARIES Some calls always stay with a person The reviewer Human boundaries How it learns Results Lessons THE AGENT CAN APPROVE Everything else when the checks pass and the agent is confident in its verdict ALWAYS A PERSON · THE AGENT FLAGS, NEVER CLEARS Sensitive data grants Who can reach sensitive data Migration intent Changes what the data means, not just the mechanics Who Reviews the Bots? 17/26 ### Some calls always stay with a person Even when checks pass, a person decides who gets sensitive data access, whether a migration should change what the data means, and how to proceed when the agent is unsure. The agent sends its risk assessment and reasoning with each case. [17:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1048s) [Three cards identify decisions that require a person: grants of access to sensitive data, migrations that change what data means, and verdicts the agent lacks confidence in. The agent sends its risk assessment and reasoning with each case.](https://assets.ai.engineer/sites/aiecode2025/sha256/6cd5f4ac7c91c5e6192d8fa073a3b92078dfc045a1ced79d388280c8110f8790.webp) HUMAN BOUNDARIES Some calls always stay with a person The reviewer Human boundaries How it learns Results Lessons THE AGENT CAN APPROVE Everything else when the checks pass and the agent is confident in its verdict ALWAYS A PERSON · THE AGENT FLAGS, NEVER CLEARS Sensitive data grants Who can reach sensitive data Migration intent Changes what the data means, not just the mechanics Low-confidence verdicts Unsure never gets rounded up to approve Even when nothing concrete comes back wrong, these go to a person, with the agent's risk and reasoning attached. Who Reviews the Bots? 17/26 ### Humans teach the rules, by hand The agent explains its verdict in a review comment. People reply with why they agree or disagree. Once a week, the person on call reads the replies collected each day and edits the rules the agent uses for its next review. [18:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1101s) [Four connected boxes show the agent posting a verdict and reason, a person explaining their agreement or disagreement, daily collection of replies, and weekly rule edits by the person on call. An arrow carries the updated rules back to the next review.](https://assets.ai.engineer/sites/aiecode2025/sha256/bcc85823e57190913826357fca08b23fa5eff03ef5551cba1d4cf8f5ea7ba0a2.webp) LEARNING FROM FEEDBACK Humans teach the rules, by hand The reviewer Human boundaries How it learns Results Lessons THE AGENT Verdict + reason in its review comments A PERSON REPLIES Agree or disagree and says why EVERY DAY Feedback collected replies gathered ONCE A WEEK Oncall updates rules reviewed and edited by hand Updated rules shape the next review MANUAL TODAY Nothing retrains itself. A person reads the feedback and edits the rules each week. Who Reviews the Bots? 18/26 ### Context can remove a false alarm The agent flagged a proposed non-sensitive label because the source column contained personal information. The missing context was where encryption happened: the pipeline encrypted the column before writing it to the data lake. [18:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1135s) [A three-box diagram follows a column tagged as personal information from the source database through an ingestion pipeline. The pipeline encrypts the column before writing it to the data lake, where the proposed label is non-sensitive.](https://assets.ai.engineer/sites/aiecode2025/sha256/e029e01aecb68d03b72ce6c77abc05f7422b97226c7b9422e5558613552a192c.webp) Context can remove a false alarm LEARNING FROM FEEDBACK SEPARATE EXAMPLE · NOT THE RUNNING PR SOURCE DATABASE A column tagged PII personal information upstream INGESTION PIPELINE Encrypts the column before it writes to the lake DATA LAKE Encrypted column proposed label: non-sensitive Who Reviews the Bots? 19/26 ### Grade misses and false escalations A miss lets a bad change ship. A false escalation sends a sound pull request to a human reviewer. Escalating every pull request may avoid misses, but it leaves reviewers with the same workload. [20:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1241s) [A two-by-two matrix compares the agent’s decision with the quality of a pull request. Approving a bad change is a miss. Escalating a sound change costs a reviewer time. The other two cells show correct approvals and escalations.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b0637e3170bac66f7af9e57e9372384810395d374cf19bb33eb971dc2253193.webp) Grade misses and false escalations GRADING THE REVIEWER As of Aug 31, 2026 PR was fine PR was bad Agent approved Correct approve — nobody's time spent MISS — a bad change ships: the dangerous one Agent escalated FALSE ESCALATION — a fine PR costs a person's time Correct escalate — a person catches it Who Reviews the Bots? 20/26 ### Grade misses and false escalations A miss lets a bad PR ship. As of Aug 31, 2026, the miss rate was about 2%, mostly from gaps in the rules. Missing context caused most false escalations, sending sound PRs to human reviewers. [20:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1254s) [A matrix compares agent approvals and escalations with whether a PR was fine or bad. Approving a bad PR is a miss. Escalating a fine PR costs reviewer time. The slide reports about 2% misses, mostly from gaps in the rules, and attributes most false escalations to missing context.](https://assets.ai.engineer/sites/aiecode2025/sha256/cbf7648b532075f413be003d5abead278974ef0c4caef95bf6e91d4d25bec09e.webp) Grade misses and false escalations GRADING THE REVIEWER As of Aug 31, 2026 PR was fine PR was bad Agent approved Correct approve — nobody's time spent MISS — a bad change ships: the dangerous one Agent escalated FALSE ESCALATION — a fine PR costs a person's time Correct escalate — a person catches it ~2% miss rate, mostly from gaps in the rules False escalations: no single clean number. Mostly missing context. Misses become new rules. False escalations shrink when the agent gets the context it was missing. Who Reviews the Bots? 20/26 ### A miss becomes a new rule A privacy retag moved columns from the open copy to a restricted copy, breaking a downstream reader. The new rule tells reviewers to search the code for readers that still use those columns in the open copy before the retag merges. [21:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1311s) [Retagging columns as personal information moved them from an open copy to a restricted copy. A downstream reader still needed them in the open copy. The new review rule calls for searching the code for those readers before the change merges.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4ada87d0e603e01fc49886a696824d7cb36bf2a43cd336d918000c2f1bbff78.webp) A miss becomes a new rule GRADING THE REVIEWER SEPARATE MISS · NOT THE RUNNING PR OPEN COPY Other columns Gone from the open copy privacy retag moves them RESTRICTED COPY · AFTER THE RETAG Retagged as personal information DOWNSTREAM READER Reads those columns from the open copy THE MISS The retag looked fine and merged. The reader's columns were gone. THE NEW RULE When a column is retagged sensitive, search the code for anyone still reading it from the copy that's losing it. Who Reviews the Bots? 21/26 ### The PR in three beats The scripts passed a change from int to bigint. The agent traced the code and found an unregistered job that still wrote to an int column. A reviewer confirmed the job was live. The downstream table was fixed before the PR merged. [23:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1385s) [Script checks passed for a PR that widened a column from int to bigint. The agent found an unregistered job writing to an int column and escalated. A person confirmed the job was active, and the downstream table was updated before the PR merged.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b79c5db270e83bbca7efb1d635526bde406de2a1ea832f3d7067058ead63558.webp) The PR in three beats KEY TAKEAWAYS RUNNING PR · THREE BEATS SOURCE TABLE Upstream system + new column Ingestion pipeline DATA LAKE TABLE Our table + new column widened: int → bigint WHO READS THE WIDENED COLUMN? Registered consumers all look fine Unregistered job writes to an int column SCRIPTS The contract, the tags, the registered lineage, and performance all passed. AGENT The code walk found a job that lineage never knew about. It flagged the whole chain and escalated. PERSON A person confirmed the job was live. The downstream table was updated first, and only then did the PR merge. No incident. Every correction a person makes can become a rule. Who Reviews the Bots? 22/26 ## Full transcript [0:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=0s) Picture a pull request. It changes a [0:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=3s) Picture a pull request. It changes a [0:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=3s) Picture a pull request. It changes a schema in one of our data pipelines. It [0:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=5s) schema in one of our data pipelines. It [0:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=6s) schema in one of our data pipelines. It goes green in CI. The tests pass. The [0:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=8s) goes green in CI. The tests pass. The [0:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=8s) goes green in CI. The tests pass. The linder is happy. And everything about it [0:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=11s) linder is happy. And everything about it [0:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=11s) linder is happy. And everything about it looks clean. So, is it safe to merge? [0:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=15s) looks clean. So, is it safe to merge? [0:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=15s) looks clean. So, is it safe to merge? Because a change like this can still [0:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=17s) Because a change like this can still [0:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=17s) Because a change like this can still break something downstream. Maybe it [0:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=19s) break something downstream. Maybe it [0:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=19s) break something downstream. Maybe it blows a freshness SLA. Or maybe it [0:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=22s) blows a freshness SLA. Or maybe it [0:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=22s) blows a freshness SLA. Or maybe it breaks a consumer three teams away who [0:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=24s) breaks a consumer three teams away who [0:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=24s) breaks a consumer three teams away who never saw it coming. Or maybe it exposes [0:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=26s) never saw it coming. Or maybe it exposes [0:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=26s) never saw it coming. Or maybe it exposes data that never supposed to be [0:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=28s) data that never supposed to be [0:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=28s) data that never supposed to be reachable. That gap between CI past and [0:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=32s) reachable. That gap between CI past and [0:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=32s) reachable. That gap between CI past and this change was actually safe is the gap [0:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=34s) this change was actually safe is the gap [0:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=34s) this change was actually safe is the gap my team has to close every day. Hello [0:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=38s) my team has to close every day. Hello [0:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=38s) my team has to close every day. Hello everyone, I'm Jas. I'm an engineer on [0:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=40s) everyone, I'm Jas. I'm an engineer on [0:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=40s) everyone, I'm Jas. I'm an engineer on the data platform team at Robin Hood. I [0:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=43s) the data platform team at Robin Hood. I [0:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=43s) the data platform team at Robin Hood. I design and build the distributed data [0:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=45s) design and build the distributed data [0:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=45s) design and build the distributed data injection and processing frameworks [0:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=46s) injection and processing frameworks [0:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=46s) injection and processing frameworks behind our data lake. The ones teams [0:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=49s) behind our data lake. The ones teams [0:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=49s) behind our data lake. The ones teams across the company rely on to make [0:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=51s) across the company rely on to make [0:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=51s) across the company rely on to make decisions with data. [0:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=53s) decisions with data. [0:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=54s) decisions with data. Notice the title of this talk. The bots [0:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=56s) Notice the title of this talk. The bots [0:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=56s) Notice the title of this talk. The bots in it are the ones opening the pull [0:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=58s) in it are the ones opening the pull [0:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=58s) in it are the ones opening the pull requests and as you will see the ones [1:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=61s) requests and as you will see the ones [1:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=61s) requests and as you will see the ones reviewing them as well. [1:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=64s) reviewing them as well. [1:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=64s) reviewing them as well. Today I want to show you how we answer [1:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=66s) Today I want to show you how we answer [1:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=66s) Today I want to show you how we answer that question when the PRs are coming in [1:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=68s) that question when the PRs are coming in [1:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=68s) that question when the PRs are coming in faster than any of us can read them. [1:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=74s) So here's the problem in one sentence. [1:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=76s) So here's the problem in one sentence. [1:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=76s) So here's the problem in one sentence. Pull requests to our production data [1:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=78s) Pull requests to our production data [1:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=78s) Pull requests to our production data pipelines now arrive at machine speed. [1:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=81s) pipelines now arrive at machine speed. [1:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=81s) pipelines now arrive at machine speed. But reviewing them safely still runs at [1:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=84s) But reviewing them safely still runs at [1:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=84s) But reviewing them safely still runs at human speed. [1:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=86s) human speed. [1:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=86s) human speed. First, a quick word on the scope of this [1:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=88s) First, a quick word on the scope of this [1:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=88s) First, a quick word on the scope of this talk because it shapes everything else [1:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=90s) talk because it shapes everything else [1:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=90s) talk because it shapes everything else in this talk. [1:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=92s) in this talk. [1:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=92s) in this talk. AI coding assistants have made pull [1:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=94s) AI coding assistants have made pull [1:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=94s) AI coding assistants have made pull requests cheap to open. A new query, a [1:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=98s) requests cheap to open. A new query, a [1:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=98s) requests cheap to open. A new query, a schema change, or an access grant now [1:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=100s) schema change, or an access grant now [1:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=100s) schema change, or an access grant now takes just a few minutes. And an [1:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=103s) takes just a few minutes. And an [1:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=103s) takes just a few minutes. And an assistant can open pull requests far [1:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=105s) assistant can open pull requests far [1:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=105s) assistant can open pull requests far faster than any team of humans ever [1:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=108s) faster than any team of humans ever [1:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=108s) faster than any team of humans ever could. [1:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=110s) could. [1:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=110s) could. The PRs I'm talking about today are the [1:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=112s) The PRs I'm talking about today are the [1:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=112s) The PRs I'm talking about today are the PRs to our data pipelines like schema [1:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=115s) PRs to our data pipelines like schema [1:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=115s) PRs to our data pipelines like schema changes, pipeline code, configuration [1:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=118s) changes, pipeline code, configuration [1:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=118s) changes, pipeline code, configuration changes, and grants that give someone [2:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=120s) changes, and grants that give someone [2:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=120s) changes, and grants that give someone access to data. [2:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=123s) access to data. [2:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=123s) access to data. Data platform is the team that owns [2:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=125s) Data platform is the team that owns [2:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=125s) Data platform is the team that owns reviewing that kind of PR. So when those [2:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=127s) reviewing that kind of PR. So when those [2:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=127s) reviewing that kind of PR. So when those PRs pile up, they pile up on us. And the [2:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=130s) PRs pile up, they pile up on us. And the [2:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=130s) PRs pile up, they pile up on us. And the reviewer I'm about to describe is the [2:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=132s) reviewer I'm about to describe is the [2:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=132s) reviewer I'm about to describe is the system we built to handle them. [2:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=135s) system we built to handle them. [2:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=135s) system we built to handle them. So the people who feel this are a small [2:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=137s) So the people who feel this are a small [2:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=137s) So the people who feel this are a small group. The engineers who carry the [2:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=139s) group. The engineers who carry the [2:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=139s) group. The engineers who carry the context to say what is safe to merge [2:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=142s) context to say what is safe to merge [2:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=142s) context to say what is safe to merge into production and everyone who is [2:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=144s) into production and everyone who is [2:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=144s) into production and everyone who is waiting on them. [2:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=147s) So why is this a problem? [2:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=150s) So why is this a problem? [2:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=150s) So why is this a problem? AI open PRs just keep arriving. [2:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=154s) AI open PRs just keep arriving. [2:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=154s) AI open PRs just keep arriving. And to be fair, the safety net we [2:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=156s) And to be fair, the safety net we [2:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=156s) And to be fair, the safety net we already had works. CI checks together [2:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=160s) already had works. CI checks together [2:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=160s) already had works. CI checks together with a human review have always caught a [2:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=162s) with a human review have always caught a [2:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=162s) with a human review have always caught a lot of real bugs like broken code, [2:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=165s) lot of real bugs like broken code, [2:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=165s) lot of real bugs like broken code, inefficient queries and risks like [2:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=168s) inefficient queries and risks like [2:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=168s) inefficient queries and risks like downstream breakage or sensitive data [2:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=170s) downstream breakage or sensitive data [2:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=170s) downstream breakage or sensitive data leaks. But the human half only works at [2:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=173s) leaks. But the human half only works at [2:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=173s) leaks. But the human half only works at the human speed and only with the [2:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=175s) the human speed and only with the [2:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=175s) the human speed and only with the context a person happens to carry in [2:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=177s) context a person happens to carry in [2:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=177s) context a person happens to carry in their head. Who reads this table? How [3:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=180s) their head. Who reads this table? How [3:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=180s) their head. Who reads this table? How big is it? And how sensitive is the data [3:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=183s) big is it? And how sensitive is the data [3:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=183s) big is it? And how sensitive is the data in it? The PRs are coming in at the [3:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=186s) in it? The PRs are coming in at the [3:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=186s) in it? The PRs are coming in at the machine speed, but the number of people [3:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=188s) machine speed, but the number of people [3:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=188s) machine speed, but the number of people who understand what is safe to merge [3:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=190s) who understand what is safe to merge [3:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=190s) who understand what is safe to merge into a production data pipeline did not [3:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=192s) into a production data pipeline did not [3:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=192s) into a production data pipeline did not grow at all. That's the mismatch. The [3:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=195s) grow at all. That's the mismatch. The [3:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=195s) grow at all. That's the mismatch. The queue builds up in front of the few [3:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=197s) queue builds up in front of the few [3:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=197s) queue builds up in front of the few people with the context and that's where [3:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=199s) people with the context and that's where [3:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=199s) people with the context and that's where things start to slip. [3:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=202s) things start to slip. [3:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=202s) things start to slip. So the problem we set out to solve was [3:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=204s) So the problem we set out to solve was [3:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=204s) So the problem we set out to solve was this. Build a PR reviewer using AI that [3:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=208s) this. Build a PR reviewer using AI that [3:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=208s) this. Build a PR reviewer using AI that can keep up with the speed at which PRs [3:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=211s) can keep up with the speed at which PRs [3:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=211s) can keep up with the speed at which PRs are being opened. [3:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=214s) are being opened. [3:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=214s) are being opened. So here's the whole answer before we get [3:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=216s) So here's the whole answer before we get [3:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=216s) So here's the whole answer before we get into the details. Three duties, three [3:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=219s) into the details. Three duties, three [3:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=219s) into the details. Three duties, three owners. [3:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=220s) owners. [3:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=220s) owners. Scripts prove what can be proven. An [3:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=223s) Scripts prove what can be proven. An [3:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=223s) Scripts prove what can be proven. An agent judges what needs context and [3:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=226s) agent judges what needs context and [3:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=226s) agent judges what needs context and people own the calls only they should [3:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=229s) people own the calls only they should [3:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=229s) people own the calls only they should make. We took a hybrid approach. Scripts [3:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=232s) make. We took a hybrid approach. Scripts [3:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=232s) make. We took a hybrid approach. Scripts for the facts and AI for judgment. We [3:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=235s) for the facts and AI for judgment. We [3:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=235s) for the facts and AI for judgment. We did not want AI reviewing every change [3:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=238s) did not want AI reviewing every change [3:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=238s) did not want AI reviewing every change when some questions already have an [4:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=240s) when some questions already have an [4:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=240s) when some questions already have an authoritative answer. [4:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=242s) authoritative answer. [4:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=242s) authoritative answer. And that's layer one. Deterministic [4:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=246s) And that's layer one. Deterministic [4:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=246s) And that's layer one. Deterministic scripts check a source of truth like the [4:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=248s) scripts check a source of truth like the [4:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=248s) scripts check a source of truth like the schema registry or a catalog. A failed [4:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=251s) schema registry or a catalog. A failed [4:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=251s) schema registry or a catalog. A failed check turns CI red and blocks the PR [4:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=254s) check turns CI red and blocks the PR [4:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=254s) check turns CI red and blocks the PR until it is fixed. [4:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=257s) until it is fixed. [4:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=257s) until it is fixed. Layer two is the agent. The agents rules [4:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=260s) Layer two is the agent. The agents rules [4:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=260s) Layer two is the agent. The agents rules tell it what layer 1 already checks. So [4:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=263s) tell it what layer 1 already checks. So [4:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=263s) tell it what layer 1 already checks. So it focuses on the risks those checks [4:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=265s) it focuses on the risks those checks [4:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=265s) it focuses on the risks those checks cannot catch. It reviews the PR, scores [4:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=268s) cannot catch. It reviews the PR, scores [4:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=268s) cannot catch. It reviews the PR, scores the risk and approves or escalates to a [4:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=271s) the risk and approves or escalates to a [4:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=271s) the risk and approves or escalates to a person. Both these layers review the [4:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=274s) person. Both these layers review the [4:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=274s) person. Both these layers review the same PR. Merge requires passing checks [4:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=277s) same PR. Merge requires passing checks [4:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=277s) same PR. Merge requires passing checks and an approving review. [4:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=280s) and an approving review. [4:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=280s) and an approving review. And this also keeps AI costs down. We [4:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=283s) And this also keeps AI costs down. We [4:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=283s) And this also keeps AI costs down. We use AI for judgment, not to rediscover [4:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=286s) use AI for judgment, not to rediscover [4:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=286s) use AI for judgment, not to rediscover facts a script can look up. [4:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=290s) facts a script can look up. [4:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=290s) facts a script can look up. So here is where we are going today. [4:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=293s) So here is where we are going today. [4:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=293s) So here is where we are going today. We have covered the problem and the [4:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=294s) We have covered the problem and the [4:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=294s) We have covered the problem and the approach. Now I'll walk you through the [4:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=296s) approach. Now I'll walk you through the [4:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=296s) approach. Now I'll walk you through the hybrid reviewer we built and the [4:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=298s) hybrid reviewer we built and the [4:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=298s) hybrid reviewer we built and the decisions we deliberately keep with the [5:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=300s) decisions we deliberately keep with the [5:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=300s) decisions we deliberately keep with the people. After that how the reviewer [5:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=303s) people. After that how the reviewer [5:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=304s) people. After that how the reviewer learns from the human feedback and what [5:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=306s) learns from the human feedback and what [5:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=306s) learns from the human feedback and what results we have seen so far. And we'll [5:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=308s) results we have seen so far. And we'll [5:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=308s) results we have seen so far. And we'll finish with the lessons we would pass on [5:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=310s) finish with the lessons we would pass on [5:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=310s) finish with the lessons we would pass on to you. [5:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=312s) to you. [5:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=312s) to you. Three things to listen for as we go. [5:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=315s) Three things to listen for as we go. [5:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=315s) Three things to listen for as we go. Scripts for facts and agent for the [5:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=317s) Scripts for facts and agent for the [5:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=317s) Scripts for facts and agent for the judgment and people for the protected [5:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=320s) judgment and people for the protected [5:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=320s) judgment and people for the protected calls. I'll come back to all three at [5:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=322s) calls. I'll come back to all three at [5:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=322s) calls. I'll come back to all three at the end. [5:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=324s) the end. [5:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=324s) the end. So to make this concrete, let me show [5:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=327s) So to make this concrete, let me show [5:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=327s) So to make this concrete, let me show you the one PR will follow through the [5:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=329s) you the one PR will follow through the [5:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=329s) you the one PR will follow through the rest of this talk. [5:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=333s) This PR makes two changes to a table in [5:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=336s) This PR makes two changes to a table in [5:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=336s) This PR makes two changes to a table in our data lake. It adds a new column and [5:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=339s) our data lake. It adds a new column and [5:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=339s) our data lake. It adds a new column and widens an existing column from integer [5:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=341s) widens an existing column from integer [5:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=341s) widens an existing column from integer to big integer. It's the kind of PR we [5:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=343s) to big integer. It's the kind of PR we [5:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=343s) to big integer. It's the kind of PR we see all the time. On the left is an [5:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=346s) see all the time. On the left is an [5:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=346s) see all the time. On the left is an upstream source table. Our injection [5:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=348s) upstream source table. Our injection [5:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=348s) upstream source table. Our injection pipeline reads from it and writes into a [5:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=351s) pipeline reads from it and writes into a [5:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=351s) pipeline reads from it and writes into a table in the data lake. call it our [5:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=353s) table in the data lake. call it our [5:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=353s) table in the data lake. call it our table. [5:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=355s) table. [5:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=355s) table. The first change, the source has gained [5:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=357s) The first change, the source has gained [5:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=357s) The first change, the source has gained a new column and this PR makes injection [6:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=361s) a new column and this PR makes injection [6:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=361s) a new column and this PR makes injection bring that new column into our lake [6:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=363s) bring that new column into our lake [6:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=363s) bring that new column into our lake table. [6:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=365s) table. [6:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=365s) table. The second change is to a column of a [6:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=367s) The second change is to a column of a [6:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=367s) The second change is to a column of a table already has [6:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=369s) table already has [6:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=369s) table already has its type gets widened from integer to [6:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=372s) its type gets widened from integer to [6:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=372s) its type gets widened from integer to big integer so it can hold bigger values [6:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=375s) big integer so it can hold bigger values [6:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=376s) big integer so it can hold bigger values on its own. That sounds completely [6:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=377s) on its own. That sounds completely [6:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=377s) on its own. That sounds completely harmless. Widening a type is about the [6:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=380s) harmless. Widening a type is about the [6:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=380s) harmless. Widening a type is about the safest schema change you can make, [6:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=382s) safest schema change you can make, [6:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=382s) safest schema change you can make, right? [6:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=384s) right? [6:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=384s) right? Well, whenever a column changes, the [6:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=386s) Well, whenever a column changes, the [6:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=386s) Well, whenever a column changes, the real question is who reads it [6:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=388s) real question is who reads it [6:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=388s) real question is who reads it downstream? Some of those readers are [6:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=390s) downstream? Some of those readers are [6:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=390s) downstream? Some of those readers are registered in our lineage tool. So, we [6:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=393s) registered in our lineage tool. So, we [6:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=393s) registered in our lineage tool. So, we know about them. [6:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=395s) know about them. [6:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=395s) know about them. But keep a second question in the back [6:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=396s) But keep a second question in the back [6:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=396s) But keep a second question in the back of your mind. Are those all of them? [6:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=400s) of your mind. Are those all of them? [6:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=400s) of your mind. Are those all of them? Hold on to this picture. We will come [6:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=403s) Hold on to this picture. We will come [6:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=403s) Hold on to this picture. We will come back to this exact diagram each time the [6:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=405s) back to this exact diagram each time the [6:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=405s) back to this exact diagram each time the PR moves forward through the reviewer. [6:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=409s) So let's start with layer one. Layer one [6:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=412s) So let's start with layer one. Layer one [6:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=412s) So let's start with layer one. Layer one is the scripts and their whole job is [6:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=414s) is the scripts and their whole job is [6:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=414s) is the scripts and their whole job is checking facts in the live systems. [6:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=417s) checking facts in the live systems. [6:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=417s) checking facts in the live systems. Things the code alone cannot tell you. [7:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=420s) Things the code alone cannot tell you. [7:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=420s) Things the code alone cannot tell you. Here are the two of those checks and [7:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=423s) Here are the two of those checks and [7:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=423s) Here are the two of those checks and what they said about our running PR. [7:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=427s) what they said about our running PR. [7:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=427s) what they said about our running PR. First data contracts. When a PR changes [7:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=430s) First data contracts. When a PR changes [7:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=430s) First data contracts. When a PR changes a schema, a script compares the schema [7:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=433s) a schema, a script compares the schema [7:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=433s) a schema, a script compares the schema and keys against a source of truth, the [7:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=436s) and keys against a source of truth, the [7:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=436s) and keys against a source of truth, the schema registry. [7:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=438s) schema registry. [7:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=438s) schema registry. for our PR. The question is simple. Does [7:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=440s) for our PR. The question is simple. Does [7:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=440s) for our PR. The question is simple. Does that new column actually exist upstream [7:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=443s) that new column actually exist upstream [7:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=443s) that new column actually exist upstream as registered in the registry? It does. [7:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=446s) as registered in the registry? It does. [7:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=446s) as registered in the registry? It does. So that check passes. [7:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=449s) So that check passes. [7:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=449s) So that check passes. Second require tags. Every table needs [7:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=453s) Second require tags. Every table needs [7:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=453s) Second require tags. Every table needs an owner and any PII has to be labeled [7:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=455s) an owner and any PII has to be labeled [7:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=455s) an owner and any PII has to be labeled in the catalog. The script checks the [7:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=458s) in the catalog. The script checks the [7:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=458s) in the catalog. The script checks the catalog directly. Does our table have an [7:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=461s) catalog directly. Does our table have an [7:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=461s) catalog directly. Does our table have an owner? Yes. Is its PII labeled? Yes. [7:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=465s) owner? Yes. Is its PII labeled? Yes. [7:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=466s) owner? Yes. Is its PII labeled? Yes. pass again. [7:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=468s) pass again. [7:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=468s) pass again. And if either of those had failed, there [7:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=470s) And if either of those had failed, there [7:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=470s) And if either of those had failed, there is no debate and no model involved. CI [7:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=473s) is no debate and no model involved. CI [7:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=473s) is no debate and no model involved. CI goes red like any other failing checks [7:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=476s) goes red like any other failing checks [7:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=476s) goes red like any other failing checks and the PR is blocked until it is fixed. [8:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=480s) and the PR is blocked until it is fixed. [8:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=480s) and the PR is blocked until it is fixed. Contracts and tags are two of the layer [8:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=482s) Contracts and tags are two of the layer [8:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=482s) Contracts and tags are two of the layer 1 checks. Next, the two that look at [8:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=485s) 1 checks. Next, the two that look at [8:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=485s) 1 checks. Next, the two that look at downstream impact and performance. So [8:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=488s) downstream impact and performance. So [8:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=488s) downstream impact and performance. So these are the other two layer 1 checks. [8:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=490s) these are the other two layer 1 checks. [8:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=490s) these are the other two layer 1 checks. They look past the table itself, what [8:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=493s) They look past the table itself, what [8:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=493s) They look past the table itself, what happens downstream and whether the job [8:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=495s) happens downstream and whether the job [8:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=495s) happens downstream and whether the job will still run in time. [8:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=499s) will still run in time. [8:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=499s) will still run in time. First, the lineage blast radius. When a [8:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=502s) First, the lineage blast radius. When a [8:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=502s) First, the lineage blast radius. When a column changes, like the one our PR [8:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=504s) column changes, like the one our PR [8:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=504s) column changes, like the one our PR widens, a script walks every downstream [8:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=507s) widens, a script walks every downstream [8:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=507s) widens, a script walks every downstream job the lineage tool has on record for [8:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=509s) job the lineage tool has on record for [8:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=509s) job the lineage tool has on record for that column and asks a simple question. [8:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=512s) that column and asks a simple question. [8:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=512s) that column and asks a simple question. Does any of them break? It's a real [8:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=514s) Does any of them break? It's a real [8:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=514s) Does any of them break? It's a real check against a real system. But notice [8:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=517s) check against a real system. But notice [8:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=517s) check against a real system. But notice the boundary I have drawn. It can only [8:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=520s) the boundary I have drawn. It can only [8:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=520s) the boundary I have drawn. It can only see what is recorded in lineage. Its [8:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=523s) see what is recorded in lineage. Its [8:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=523s) see what is recorded in lineage. Its view of downstream is exactly as [8:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=526s) view of downstream is exactly as [8:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=526s) view of downstream is exactly as complete as that record. No more. [8:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=529s) complete as that record. No more. [8:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=530s) complete as that record. No more. Second, the performance. A correct [8:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=532s) Second, the performance. A correct [8:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=532s) Second, the performance. A correct change can still be too slow or too [8:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=534s) change can still be too slow or too [8:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=534s) change can still be too slow or too expensive. So the CI performance test [8:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=537s) expensive. So the CI performance test [8:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=537s) expensive. So the CI performance test runs the change in staging at the same [8:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=539s) runs the change in staging at the same [9:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=540s) runs the change in staging at the same data volume as production using mock [9:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=542s) data volume as production using mock [9:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=542s) data volume as production using mock data. [9:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=544s) data. [9:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=544s) data. So that tells us two things before the [9:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=546s) So that tells us two things before the [9:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=546s) So that tells us two things before the PR merges. whether the cluster is sized [9:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=548s) PR merges. whether the cluster is sized [9:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=548s) PR merges. whether the cluster is sized right for the job and whether the job [9:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=550s) right for the job and whether the job [9:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=550s) right for the job and whether the job will still make its freshness SLA. Those [9:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=554s) will still make its freshness SLA. Those [9:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=554s) will still make its freshness SLA. Those are the kind of problems that otherwise [9:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=556s) are the kind of problems that otherwise [9:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=556s) are the kind of problems that otherwise show up weeks later as an incident. So [9:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=559s) show up weeks later as an incident. So [9:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=559s) show up weeks later as an incident. So that's the full set of layer 1 checks. [9:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=562s) that's the full set of layer 1 checks. [9:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=562s) that's the full set of layer 1 checks. Let's see what they said about our PR. [9:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=567s) So back to our running PR. Same picture [9:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=570s) So back to our running PR. Same picture [9:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=570s) So back to our running PR. Same picture as before. Here's what the layer one [9:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=572s) as before. Here's what the layer one [9:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=572s) as before. Here's what the layer one said about it. [9:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=575s) said about it. [9:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=575s) said about it. The data contract check went to the [9:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=577s) The data contract check went to the [9:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=577s) The data contract check went to the schema registry and asked whether the [9:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=579s) schema registry and asked whether the [9:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=579s) schema registry and asked whether the new column really exists upstream as [9:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=581s) new column really exists upstream as [9:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=581s) new column really exists upstream as register. It does and the check passes. [9:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=586s) register. It does and the check passes. [9:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=586s) register. It does and the check passes. The leanage blast radius check walked [9:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=588s) The leanage blast radius check walked [9:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=588s) The leanage blast radius check walked every registered consumer of the widen [9:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=590s) every registered consumer of the widen [9:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=590s) every registered consumer of the widen column. The jobs the lenage tool has on [9:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=593s) column. The jobs the lenage tool has on [9:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=593s) column. The jobs the lenage tool has on record. None of them break when the [9:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=595s) record. None of them break when the [9:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=595s) record. None of them break when the column goes from integer to big integer. [9:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=597s) column goes from integer to big integer. [9:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=597s) column goes from integer to big integer. They all look fine. Along with that tags [10:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=601s) They all look fine. Along with that tags [10:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=601s) They all look fine. Along with that tags pass, performance passes. Every check is [10:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=603s) pass, performance passes. Every check is [10:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=604s) pass, performance passes. Every check is green. [10:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=605s) green. [10:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=605s) green. As far as the scripts can see, this PR [10:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=607s) As far as the scripts can see, this PR [10:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=607s) As far as the scripts can see, this PR is safe. But I chose those words [10:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=609s) is safe. But I chose those words [10:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=610s) is safe. But I chose those words carefully. As far as the scripts can [10:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=612s) carefully. As far as the scripts can [10:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=612s) carefully. As far as the scripts can see, everything on the slide is a [10:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=615s) see, everything on the slide is a [10:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=615s) see, everything on the slide is a statement about the registered picture, [10:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=617s) statement about the registered picture, [10:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=617s) statement about the registered picture, the jobs and the columns our systems [10:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=619s) the jobs and the columns our systems [10:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=619s) the jobs and the columns our systems have a record of. And keep that in mind. [10:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=623s) have a record of. And keep that in mind. [10:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=623s) have a record of. And keep that in mind. It is not the whole story. [10:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=626s) It is not the whole story. [10:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=626s) It is not the whole story. Now let's look at the agent with the [10:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=628s) Now let's look at the agent with the [10:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=628s) Now let's look at the agent with the diff. What actually changed? [10:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=631s) diff. What actually changed? [10:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=631s) diff. What actually changed? The PR description and the commit [10:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=633s) The PR description and the commit [10:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=633s) The PR description and the commit messages explain the author's intent, [10:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=635s) messages explain the author's intent, [10:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=635s) messages explain the author's intent, but intent never explains away a real [10:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=638s) but intent never explains away a real [10:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=638s) but intent never explains away a real finding. If the description says it is [10:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=641s) finding. If the description says it is [10:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=641s) finding. If the description says it is safe and the code says otherwise, the [10:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=643s) safe and the code says otherwise, the [10:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=643s) safe and the code says otherwise, the code wins. [10:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=646s) code wins. [10:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=646s) code wins. File tools let the agent read, search, [10:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=648s) File tools let the agent read, search, [10:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=648s) File tools let the agent read, search, and list the files across the [10:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=649s) and list the files across the [10:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=649s) and list the files across the repository. It follows changes beyond [10:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=652s) repository. It follows changes beyond [10:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=652s) repository. It follows changes beyond the diff into pipeline definitions and [10:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=654s) the diff into pipeline definitions and [10:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=654s) the diff into pipeline definitions and access grants looking for the [10:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=656s) access grants looking for the [10:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=656s) access grants looking for the relationships the author may have [10:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=658s) relationships the author may have [10:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=658s) relationships the author may have missed. [11:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=661s) missed. [11:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=661s) missed. These are the repository repository [11:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=663s) These are the repository repository [11:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=663s) These are the repository repository tools only. The agent cannot query the [11:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=665s) tools only. The agent cannot query the [11:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=665s) tools only. The agent cannot query the live catalog lineage tool or production [11:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=668s) live catalog lineage tool or production [11:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=668s) live catalog lineage tool or production state. Life facts stay layer 1's job. [11:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=672s) state. Life facts stay layer 1's job. [11:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=672s) state. Life facts stay layer 1's job. The agent reasons over the code. So why [11:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=675s) The agent reasons over the code. So why [11:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=675s) The agent reasons over the code. So why walk that code for a schema change? [11:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=677s) walk that code for a schema change? [11:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=678s) walk that code for a schema change? Because registered lineage is not the [11:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=680s) Because registered lineage is not the [11:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=680s) Because registered lineage is not the whole picture. [11:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=683s) whole picture. [11:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=683s) whole picture. The rules define where the judgment [11:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=686s) The rules define where the judgment [11:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=686s) The rules define where the judgment belongs. They are configured by the [11:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=688s) belongs. They are configured by the [11:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=688s) belongs. They are configured by the change type in the PR knowing what layer [11:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=690s) change type in the PR knowing what layer [11:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=690s) change type in the PR knowing what layer 1 already covers. They target query anti [11:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=693s) 1 already covers. They target query anti [11:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=693s) 1 already covers. They target query anti patterns, sensitive data asks and [11:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=696s) patterns, sensitive data asks and [11:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=696s) patterns, sensitive data asks and consumers outside lineage. [11:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=699s) consumers outside lineage. [11:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=699s) consumers outside lineage. For example, an unpartitioned cross join [11:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=701s) For example, an unpartitioned cross join [11:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=701s) For example, an unpartitioned cross join can run correctly and still hammer a [11:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=703s) can run correctly and still hammer a [11:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=703s) can run correctly and still hammer a cluster. A grant can look ordinary and [11:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=706s) cluster. A grant can look ordinary and [11:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=706s) cluster. A grant can look ordinary and still open sensitive data. [11:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=708s) still open sensitive data. [11:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=708s) still open sensitive data. And for the schema changes, the agent [11:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=711s) And for the schema changes, the agent [11:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=711s) And for the schema changes, the agent walks the pipeline definitions and [11:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=713s) walks the pipeline definitions and [11:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=713s) walks the pipeline definitions and access grants in the repository to find [11:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=715s) access grants in the repository to find [11:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=715s) access grants in the repository to find consumers that were not registered. It's [11:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=718s) consumers that were not registered. It's [11:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=718s) consumers that were not registered. It's a tedious search uh a person would [12:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=720s) a tedious search uh a person would [12:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=720s) a tedious search uh a person would otherwise do by hand. [12:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=722s) otherwise do by hand. [12:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=722s) otherwise do by hand. Let's return to our PR and see what that [12:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=724s) Let's return to our PR and see what that [12:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=724s) Let's return to our PR and see what that search found. [12:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=727s) search found. [12:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=727s) search found. So back to our running PR. Same picture. [12:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=730s) So back to our running PR. Same picture. [12:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=730s) So back to our running PR. Same picture. Layer 1 said every registered consumer [12:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=733s) Layer 1 said every registered consumer [12:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=733s) Layer 1 said every registered consumer of the vid column looks fine. Now the [12:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=736s) of the vid column looks fine. Now the [12:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=736s) of the vid column looks fine. Now the agent walks the code [12:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=738s) agent walks the code [12:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=738s) agent walks the code and it finds a job that lineage never [12:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=741s) and it finds a job that lineage never [12:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=741s) and it finds a job that lineage never knew about. It's right there in the [12:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=743s) knew about. It's right there in the [12:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=743s) knew about. It's right there in the repository. It reads our widen column [12:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=746s) repository. It reads our widen column [12:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=746s) repository. It reads our widen column from our late table but it was never [12:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=748s) from our late table but it was never [12:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=748s) from our late table but it was never registered in the lineage which is [12:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=750s) registered in the lineage which is [12:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=750s) registered in the lineage which is exactly why the layer 1 check could not [12:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=752s) exactly why the layer 1 check could not [12:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=752s) exactly why the layer 1 check could not see it. [12:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=754s) see it. [12:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=754s) see it. And that job takes the value and writes [12:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=756s) And that job takes the value and writes [12:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=756s) And that job takes the value and writes it into another table. And in that other [12:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=759s) it into another table. And in that other [12:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=759s) it into another table. And in that other table, the column is still an integer. [12:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=763s) table, the column is still an integer. [12:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=763s) table, the column is still an integer. Now follow it through. Our column is a [12:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=765s) Now follow it through. Our column is a [12:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=765s) Now follow it through. Our column is a big integer. Now the first time a value [12:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=768s) big integer. Now the first time a value [12:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=768s) big integer. Now the first time a value comes through that's bigger than an [12:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=770s) comes through that's bigger than an [12:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=770s) comes through that's bigger than an integer can hold that right fails. A [12:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=774s) integer can hold that right fails. A [12:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=774s) integer can hold that right fails. A valid schema change green in CI passes [12:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=777s) valid schema change green in CI passes [12:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=777s) valid schema change green in CI passes every registered check and a downstream [12:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=779s) every registered check and a downstream [13:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=780s) every registered check and a downstream job still breaks. [13:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=783s) job still breaks. [13:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=783s) job still breaks. The agent flagged exactly this chain in [13:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=785s) The agent flagged exactly this chain in [13:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=785s) The agent flagged exactly this chain in its review comment with the reasoning. [13:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=787s) its review comment with the reasoning. [13:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=787s) its review comment with the reasoning. So a person could see the whole path [13:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=789s) So a person could see the whole path [13:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=789s) So a person could see the whole path from our column to the failing right. [13:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=792s) from our column to the failing right. [13:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=792s) from our column to the failing right. We'll get to what happened next in a [13:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=793s) We'll get to what happened next in a [13:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=794s) We'll get to what happened next in a moment. [13:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=796s) moment. [13:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=796s) moment. So why did the lineage miss that job? [13:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=799s) So why did the lineage miss that job? [13:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=799s) So why did the lineage miss that job? Because the enterprise data world is [13:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=800s) Because the enterprise data world is [13:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=800s) Because the enterprise data world is messy. Workloads run across different [13:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=803s) messy. Workloads run across different [13:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=803s) messy. Workloads run across different query engines and each engine has to [13:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=805s) query engines and each engine has to [13:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=805s) query engines and each engine has to publish its own lineage. If one does not [13:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=808s) publish its own lineage. If one does not [13:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=808s) publish its own lineage. If one does not publish, that relationship is missing [13:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=810s) publish, that relationship is missing [13:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=810s) publish, that relationship is missing from the picture. There's a single [13:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=812s) from the picture. There's a single [13:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=812s) from the picture. There's a single catalog that automatically sees [13:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=814s) catalog that automatically sees [13:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=814s) catalog that automatically sees everything. So the registered picture is [13:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=816s) everything. So the registered picture is [13:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=816s) everything. So the registered picture is useful but it is not the whole picture. [13:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=821s) useful but it is not the whole picture. [13:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=821s) useful but it is not the whole picture. Now I don't want to oversell the [13:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=823s) Now I don't want to oversell the [13:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=823s) Now I don't want to oversell the codewalk. It's genuinely useful but it [13:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=826s) codewalk. It's genuinely useful but it [13:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=826s) codewalk. It's genuinely useful but it cannot see everything. [13:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=828s) cannot see everything. [13:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=828s) cannot see everything. The agent can search what is in the [13:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=830s) The agent can search what is in the [13:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=830s) The agent can search what is in the repo. Pipeline code access grants and [13:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=833s) repo. Pipeline code access grants and [13:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=833s) repo. Pipeline code access grants and the jobs defined there that read our [13:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=835s) the jobs defined there that read our [13:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=836s) the jobs defined there that read our tables. That's where it found our [13:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=838s) tables. That's where it found our [13:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=838s) tables. That's where it found our unregistered job. But plenty of data use [14:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=842s) unregistered job. But plenty of data use [14:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=842s) unregistered job. But plenty of data use lives outside the codebase. Ad hawk [14:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=844s) lives outside the codebase. Ad hawk [14:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=844s) lives outside the codebase. Ad hawk queries that someone runs by hand. [14:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=846s) queries that someone runs by hand. [14:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=846s) queries that someone runs by hand. Dashboards, those are completely [14:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=848s) Dashboards, those are completely [14:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=848s) Dashboards, those are completely invisible to a repository search. A [14:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=851s) invisible to a repository search. A [14:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=851s) invisible to a repository search. A consumer out there never shows up, no [14:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=853s) consumer out there never shows up, no [14:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=853s) consumer out there never shows up, no matter how carefully the agent looks. So [14:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=855s) matter how carefully the agent looks. So [14:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=856s) matter how carefully the agent looks. So we make the agent honest about that [14:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=857s) we make the agent honest about that [14:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=857s) we make the agent honest about that limit. If it does not find anything, it [14:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=860s) limit. If it does not find anything, it [14:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=860s) limit. If it does not find anything, it does not get to say no downstream [14:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=862s) does not get to say no downstream [14:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=862s) does not get to say no downstream consumers exist. It says it found none [14:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=865s) consumers exist. It says it found none [14:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=865s) consumers exist. It says it found none in the places it searched. None found is [14:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=868s) in the places it searched. None found is [14:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=868s) in the places it searched. None found is not the same as none exist and the agent [14:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=871s) not the same as none exist and the agent [14:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=871s) not the same as none exist and the agent never gets to claim an allcle. [14:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=876s) So the agent has walked the code. What [14:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=878s) So the agent has walked the code. What [14:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=878s) So the agent has walked the code. What does it actually produce? A review the [14:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=881s) does it actually produce? A review the [14:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=881s) does it actually produce? A review the same shape a good human reviewer would [14:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=883s) same shape a good human reviewer would [14:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=883s) same shape a good human reviewer would give you. [14:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=885s) give you. [14:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=885s) give you. First review comments on the specific [14:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=887s) First review comments on the specific [14:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=887s) First review comments on the specific lines that matter each with its [14:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=890s) lines that matter each with its [14:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=890s) lines that matter each with its reasoning. Not awake looks risky but [14:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=893s) reasoning. Not awake looks risky but [14:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=893s) reasoning. Not awake looks risky but this line changes this column. And here [14:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=895s) this line changes this column. And here [14:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=896s) this line changes this column. And here is the job that reads it. [14:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=898s) is the job that reads it. [14:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=898s) is the job that reads it. Second, a risk score. How severe is what [15:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=901s) Second, a risk score. How severe is what [15:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=901s) Second, a risk score. How severe is what it found. That's what drives the next [15:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=904s) it found. That's what drives the next [15:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=904s) it found. That's what drives the next step. Then it makes a call. If there is [15:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=907s) step. Then it makes a call. If there is [15:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=907s) step. Then it makes a call. If there is no serious risk, it can approve. If [15:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=909s) no serious risk, it can approve. If [15:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=909s) no serious risk, it can approve. If there is a risk that needs a person, it [15:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=912s) there is a risk that needs a person, it [15:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=912s) there is a risk that needs a person, it escalates to a human and the comments [15:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=914s) escalates to a human and the comments [15:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=914s) escalates to a human and the comments and reasoning go with it. So the person [15:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=917s) and reasoning go with it. So the person [15:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=917s) and reasoning go with it. So the person has the context to investigate instead [15:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=919s) has the context to investigate instead [15:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=919s) has the context to investigate instead of starting from scratch. [15:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=922s) of starting from scratch. [15:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=922s) of starting from scratch. And to be clear about what it takes to [15:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=924s) And to be clear about what it takes to [15:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=924s) And to be clear about what it takes to merge, layer 1's checks have to pass and [15:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=928s) merge, layer 1's checks have to pass and [15:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=928s) merge, layer 1's checks have to pass and the review has to approve either the [15:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=930s) the review has to approve either the [15:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=930s) the review has to approve either the agent or after an escalation a person. [15:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=933s) agent or after an escalation a person. [15:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=933s) agent or after an escalation a person. Both have to say yes. Now let's see what [15:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=936s) Both have to say yes. Now let's see what [15:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=936s) Both have to say yes. Now let's see what that looked like for our PR. [15:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=940s) that looked like for our PR. [15:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=940s) that looked like for our PR. Here is how our running PR actually [15:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=942s) Here is how our running PR actually [15:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=942s) Here is how our running PR actually ended. Same picture. The registered [15:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=945s) ended. Same picture. The registered [15:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=945s) ended. Same picture. The registered consumers look fine and the core walk [15:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=947s) consumers look fine and the core walk [15:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=948s) consumers look fine and the core walk found an unregistered job that writes [15:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=950s) found an unregistered job that writes [15:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=950s) found an unregistered job that writes our widened column into a table where it [15:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=952s) our widened column into a table where it [15:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=952s) our widened column into a table where it is still an integer [15:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=955s) is still an integer [15:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=955s) is still an integer because the agent found a real [15:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=956s) because the agent found a real [15:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=956s) because the agent found a real downstream break. It scored the risk [15:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=958s) downstream break. It scored the risk [15:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=958s) downstream break. It scored the risk high and escalated to a human with the [16:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=961s) high and escalated to a human with the [16:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=961s) high and escalated to a human with the whole chain spelled out in its review [16:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=963s) whole chain spelled out in its review [16:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=963s) whole chain spelled out in its review comment. Now the human does the part the [16:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=966s) comment. Now the human does the part the [16:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=966s) comment. Now the human does the part the agent cannot. The agent only reasons [16:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=969s) agent cannot. The agent only reasons [16:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=969s) agent cannot. The agent only reasons over the repo. It cannot see whether the [16:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=971s) over the repo. It cannot see whether the [16:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=971s) over the repo. It cannot see whether the job is actually running in production. [16:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=973s) job is actually running in production. [16:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=973s) job is actually running in production. human reviewer checked and confirmed [16:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=976s) human reviewer checked and confirmed [16:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=976s) human reviewer checked and confirmed that the consumer the agent found was [16:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=977s) that the consumer the agent found was [16:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=978s) that the consumer the agent found was live and active. So this was not a [16:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=981s) live and active. So this was not a [16:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=981s) live and active. So this was not a theoretical risk. The PR waited [16:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=985s) theoretical risk. The PR waited [16:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=985s) theoretical risk. The PR waited the f the fix was about order not about [16:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=988s) the f the fix was about order not about [16:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=988s) the f the fix was about order not about rejecting the change. The downstream [16:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=991s) rejecting the change. The downstream [16:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=991s) rejecting the change. The downstream table was updated first so its column [16:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=993s) table was updated first so its column [16:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=993s) table was updated first so its column could handle the wider type [16:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=996s) could handle the wider type [16:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=996s) could handle the wider type and only then did our PR merge and [16:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=998s) and only then did our PR merge and [16:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=998s) and only then did our PR merge and deploy. No incident. The scripts in [16:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1001s) deploy. No incident. The scripts in [16:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1001s) deploy. No incident. The scripts in layer 1 said safe. The agent found the [16:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1004s) layer 1 said safe. The agent found the [16:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1004s) layer 1 said safe. The agent found the gap and a human made the call. [16:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1009s) gap and a human made the call. [16:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1009s) gap and a human made the call. And that was not a one-time thing. Most [16:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1011s) And that was not a one-time thing. Most [16:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1012s) And that was not a one-time thing. Most changes can be cleared by the agent. [16:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1014s) changes can be cleared by the agent. [16:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1014s) changes can be cleared by the agent. When the checks pass and the agent is [16:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1016s) When the checks pass and the agent is [16:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1016s) When the checks pass and the agent is confident in its verdict, it approves on [16:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1018s) confident in its verdict, it approves on [16:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1018s) confident in its verdict, it approves on its own. But there's a whole category of [17:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1020s) its own. But there's a whole category of [17:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1020s) its own. But there's a whole category of changes where the agent never gets to [17:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1022s) changes where the agent never gets to [17:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1022s) changes where the agent never gets to say approved. And no matter how [17:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1024s) say approved. And no matter how [17:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1024s) say approved. And no matter how confident it sounds, [17:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1026s) confident it sounds, [17:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1026s) confident it sounds, grants touching sensitive data are one. [17:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1030s) grants touching sensitive data are one. [17:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1030s) grants touching sensitive data are one. Who can reach sensitive data is a [17:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1031s) Who can reach sensitive data is a [17:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1032s) Who can reach sensitive data is a decision we want a person to own every [17:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1034s) decision we want a person to own every [17:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1034s) decision we want a person to own every time. [17:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1036s) time. [17:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1036s) time. A migration that looks like it changes [17:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1038s) A migration that looks like it changes [17:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1038s) A migration that looks like it changes intent, not just mechanics, is another. [17:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1041s) intent, not just mechanics, is another. [17:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1041s) intent, not just mechanics, is another. If a change alters what the data means [17:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1044s) If a change alters what the data means [17:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1044s) If a change alters what the data means rather than just how it is moved or [17:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1046s) rather than just how it is moved or [17:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1046s) rather than just how it is moved or stored, a person decides. [17:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1050s) stored, a person decides. [17:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1050s) stored, a person decides. And if the agent itself is not confident [17:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1052s) And if the agent itself is not confident [17:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1052s) And if the agent itself is not confident in its own verdict, that's the third. It [17:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1055s) in its own verdict, that's the third. It [17:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1055s) in its own verdict, that's the third. It does not get to round up to approve just [17:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1057s) does not get to round up to approve just [17:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1057s) does not get to round up to approve just because nothing concrete came back [17:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1059s) because nothing concrete came back [17:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1059s) because nothing concrete came back wrong. In all the all the three cases, [17:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1062s) wrong. In all the all the three cases, [17:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1062s) wrong. In all the all the three cases, its job is only able to surface the risk [17:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1065s) its job is only able to surface the risk [17:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1065s) its job is only able to surface the risk and its reasoning. The decision stays [17:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1067s) and its reasoning. The decision stays [17:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1067s) and its reasoning. The decision stays with a person. [17:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1070s) with a person. [17:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1070s) with a person. That is one PR. Now, how does this hold [17:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1073s) That is one PR. Now, how does this hold [17:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1073s) That is one PR. Now, how does this hold up across every PR we see? And how does [17:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1075s) up across every PR we see? And how does [17:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1075s) up across every PR we see? And how does the reviewer get better over time? [17:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1079s) the reviewer get better over time? [17:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1079s) the reviewer get better over time? How do we know the flags are useful or [18:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1081s) How do we know the flags are useful or [18:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1082s) How do we know the flags are useful or whether a rule change helps? We start [18:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1085s) whether a rule change helps? We start [18:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1085s) whether a rule change helps? We start with the agents verdict and its [18:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1086s) with the agents verdict and its [18:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1086s) with the agents verdict and its reasoning in the review comments. [18:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1089s) reasoning in the review comments. [18:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1089s) reasoning in the review comments. A person replies agreeing or disagreeing [18:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1092s) A person replies agreeing or disagreeing [18:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1092s) A person replies agreeing or disagreeing and explaining why. That explanation is [18:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1095s) and explaining why. That explanation is [18:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1095s) and explaining why. That explanation is what makes the feedback useful rather [18:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1097s) what makes the feedback useful rather [18:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1097s) what makes the feedback useful rather than just a vote. [18:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1099s) than just a vote. [18:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1099s) than just a vote. Those replies are collected every day. [18:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1103s) Those replies are collected every day. [18:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1103s) Those replies are collected every day. Once a week, the on call reads the [18:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1105s) Once a week, the on call reads the [18:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1105s) Once a week, the on call reads the feedback and updates the review rules by [18:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1107s) feedback and updates the review rules by [18:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1107s) feedback and updates the review rules by hand. Those rules shape the next review. [18:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1111s) hand. Those rules shape the next review. [18:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1111s) hand. Those rules shape the next review. I know this is manual today. Nothing [18:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1113s) I know this is manual today. Nothing [18:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1113s) I know this is manual today. Nothing retrains itself and no rule changes [18:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1115s) retrains itself and no rule changes [18:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1115s) retrains itself and no rule changes automatically. But we have an idea to [18:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1118s) automatically. But we have an idea to [18:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1118s) automatically. But we have an idea to let an incident investigation agent help [18:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1121s) let an incident investigation agent help [18:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1121s) let an incident investigation agent help with that weekly pass. But it is not [18:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1123s) with that weekly pass. But it is not [18:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1123s) with that weekly pass. But it is not built. Without the feedback, every rule [18:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1126s) built. Without the feedback, every rule [18:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1126s) built. Without the feedback, every rule change would be a guess. So let me show [18:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1128s) change would be a guess. So let me show [18:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1128s) change would be a guess. So let me show you an example where a human corrected [18:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1130s) you an example where a human corrected [18:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1130s) you an example where a human corrected the agent and we use that feedback to [18:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1133s) the agent and we use that feedback to [18:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1133s) the agent and we use that feedback to improve its next review. [18:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1137s) improve its next review. [18:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1137s) improve its next review. So here's a real one. And note the label [19:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1140s) So here's a real one. And note the label [19:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1140s) So here's a real one. And note the label in the corner. This is a separate [19:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1142s) in the corner. This is a separate [19:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1142s) in the corner. This is a separate example, not our schema change PR. [19:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1146s) example, not our schema change PR. [19:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1146s) example, not our schema change PR. A column was marked as personal [19:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1148s) A column was marked as personal [19:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1148s) A column was marked as personal information in the source database. [19:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1150s) information in the source database. [19:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1150s) information in the source database. But by the time it lands in the data [19:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1152s) But by the time it lands in the data [19:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1152s) But by the time it lands in the data lake, the injection pipeline has already [19:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1154s) lake, the injection pipeline has already [19:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1154s) lake, the injection pipeline has already encrypted it. So the PR proposed [19:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1157s) encrypted it. So the PR proposed [19:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1157s) encrypted it. So the PR proposed labeling it nonsensitive in the data [19:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1159s) labeling it nonsensitive in the data [19:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1159s) labeling it nonsensitive in the data lake even though the upstream data is [19:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1161s) lake even though the upstream data is [19:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1161s) lake even though the upstream data is still PII. [19:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1163s) still PII. [19:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1163s) still PII. The agent flagged this. From where it [19:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1166s) The agent flagged this. From where it [19:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1166s) The agent flagged this. From where it stood, the upstream says PII, so a [19:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1168s) stood, the upstream says PII, so a [19:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1168s) stood, the upstream says PII, so a nonsensitive label looked wrong. The [19:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1171s) nonsensitive label looked wrong. The [19:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1171s) nonsensitive label looked wrong. The author pushed back and asked the agent [19:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1174s) author pushed back and asked the agent [19:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1174s) author pushed back and asked the agent to look at how similar columns on the [19:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1175s) to look at how similar columns on the [19:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1175s) to look at how similar columns on the other tables were already classified [19:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1177s) other tables were already classified [19:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1178s) other tables were already classified instead of going purely off the upstream [19:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1180s) instead of going purely off the upstream [19:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1180s) instead of going purely off the upstream tagging. [19:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1182s) tagging. [19:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1182s) tagging. A human agreed with the author. The [19:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1185s) A human agreed with the author. The [19:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1185s) A human agreed with the author. The encryption before write made the [19:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1186s) encryption before write made the [19:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1186s) encryption before write made the nonsensitive label look okay for this [19:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1189s) nonsensitive label look okay for this [19:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1189s) nonsensitive label look okay for this column. And that kind of correction is [19:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1191s) column. And that kind of correction is [19:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1191s) column. And that kind of correction is exactly the signal that teaches the [19:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1193s) exactly the signal that teaches the [19:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1193s) exactly the signal that teaches the rules to get smarter because the next [19:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1196s) rules to get smarter because the next [19:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1196s) rules to get smarter because the next time a similar column comes through that [19:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1198s) time a similar column comes through that [19:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1198s) time a similar column comes through that context can stop a needless escalation. [20:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1202s) context can stop a needless escalation. [20:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1202s) context can stop a needless escalation. One caution though, this depends on the [20:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1204s) One caution though, this depends on the [20:04](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1204s) One caution though, this depends on the specific pipeline encrypting before the [20:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1207s) specific pipeline encrypting before the [20:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1207s) specific pipeline encrypting before the right. It's not a blanket rule that [20:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1209s) right. It's not a blanket rule that [20:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1209s) right. It's not a blanket rule that encryption always makes data [20:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1211s) encryption always makes data [20:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1211s) encryption always makes data nonsensitive. [20:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1214s) So how do we know whether the reviewer [20:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1216s) So how do we know whether the reviewer [20:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1216s) So how do we know whether the reviewer is actually doing a good job? [20:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1218s) is actually doing a good job? [20:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1218s) is actually doing a good job? We look at two kinds of mistakes. Bad [20:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1221s) We look at two kinds of mistakes. Bad [20:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1221s) We look at two kinds of mistakes. Bad PRs it approved and fine PRs it [20:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1224s) PRs it approved and fine PRs it [20:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1224s) PRs it approved and fine PRs it unnecessarily send to a person. That [20:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1227s) unnecessarily send to a person. That [20:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1227s) unnecessarily send to a person. That matrix separates four possibilities. [20:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1231s) matrix separates four possibilities. [20:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1231s) matrix separates four possibilities. A fine PR approved and a bad PR [20:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1233s) A fine PR approved and a bad PR [20:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1233s) A fine PR approved and a bad PR escalated are the two correct outcomes. [20:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1236s) escalated are the two correct outcomes. [20:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1236s) escalated are the two correct outcomes. A bad PR approved is a miss. That's [20:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1239s) A bad PR approved is a miss. That's [20:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1239s) A bad PR approved is a miss. That's dangerous because the change ships. [20:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1242s) dangerous because the change ships. [20:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1242s) dangerous because the change ships. A fine PR escalated is a false [20:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1244s) A fine PR escalated is a false [20:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1244s) A fine PR escalated is a false escalation that costs a person's time. A [20:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1248s) escalation that costs a person's time. A [20:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1248s) escalation that costs a person's time. A gate that escalates everything may avoid [20:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1251s) gate that escalates everything may avoid [20:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1251s) gate that escalates everything may avoid misses, but it has not reduced the [20:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1253s) misses, but it has not reduced the [20:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1253s) misses, but it has not reduced the review load. [20:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1255s) review load. [20:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1256s) review load. As of August 31st, 2026, our miss rate [20:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1258s) As of August 31st, 2026, our miss rate [20:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1258s) As of August 31st, 2026, our miss rate was about 2%. [21:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1261s) was about 2%. [21:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1261s) was about 2%. Mostly from the gaps in the rules rather [21:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1263s) Mostly from the gaps in the rules rather [21:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1263s) Mostly from the gaps in the rules rather than the failures of the judgment. We [21:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1266s) than the failures of the judgment. We [21:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1266s) than the failures of the judgment. We don't have a clean single false [21:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1267s) don't have a clean single false [21:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1267s) don't have a clean single false escalation number. But the dominant [21:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1269s) escalation number. But the dominant [21:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1270s) escalation number. But the dominant cause is missing context like the [21:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1271s) cause is missing context like the [21:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1272s) cause is missing context like the encrypted column example that I just [21:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1274s) encrypted column example that I just [21:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1274s) encrypted column example that I just gave you. [21:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1276s) gave you. [21:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1276s) gave you. Misses give us new rules and false [21:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1278s) Misses give us new rules and false [21:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1278s) Misses give us new rules and false escalation show us context the reviewer [21:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1280s) escalation show us context the reviewer [21:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1280s) escalation show us context the reviewer needs. So here is one of those misses. [21:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1284s) needs. So here is one of those misses. [21:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1284s) needs. So here is one of those misses. Again know the label. This is a separate [21:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1286s) Again know the label. This is a separate [21:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1286s) Again know the label. This is a separate example, not our schema change PR. [21:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1289s) example, not our schema change PR. [21:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1289s) example, not our schema change PR. We have an open copy of the sum of some [21:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1292s) We have an open copy of the sum of some [21:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1292s) We have an open copy of the sum of some data. some of its columns are about to [21:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1294s) data. some of its columns are about to [21:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1294s) data. some of its columns are about to be rettagged and there is a downstream [21:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1297s) be rettagged and there is a downstream [21:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1297s) be rettagged and there is a downstream reader that reads those columns from the [21:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1299s) reader that reads those columns from the [21:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1299s) reader that reads those columns from the open copy. [21:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1301s) open copy. [21:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1301s) open copy. Privacy retag those columns as personal [21:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1303s) Privacy retag those columns as personal [21:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1303s) Privacy retag those columns as personal information. When that happens, they [21:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1306s) information. When that happens, they [21:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1306s) information. When that happens, they move into a more restricted copy of the [21:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1308s) move into a more restricted copy of the [21:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1308s) move into a more restricted copy of the data and they are gone from the open [21:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1310s) data and they are gone from the open [21:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1310s) data and they are gone from the open copy. [21:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1312s) copy. [21:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1312s) copy. The retag looked fine and it merged. But [21:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1315s) The retag looked fine and it merged. But [21:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1315s) The retag looked fine and it merged. But what nobody checked was whether anything [21:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1317s) what nobody checked was whether anything [21:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1317s) what nobody checked was whether anything downstream was still reading those [21:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1318s) downstream was still reading those [21:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1318s) downstream was still reading those columns from the old open copy. [22:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1321s) columns from the old open copy. [22:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1321s) columns from the old open copy. something was. And once the columns were [22:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1323s) something was. And once the columns were [22:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1323s) something was. And once the columns were gone, that reader broke. That's a miss. [22:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1327s) gone, that reader broke. That's a miss. [22:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1327s) gone, that reader broke. That's a miss. A PR that should have been escalated got [22:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1329s) A PR that should have been escalated got [22:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1329s) A PR that should have been escalated got approved. [22:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1331s) approved. [22:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1331s) approved. It wasn't a failure of judgment. It was [22:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1333s) It wasn't a failure of judgment. It was [22:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1333s) It wasn't a failure of judgment. It was a gap in the rules. A coroner case [22:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1336s) a gap in the rules. A coroner case [22:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1336s) a gap in the rules. A coroner case nobody had written down. So, it became a [22:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1338s) nobody had written down. So, it became a [22:18](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1338s) nobody had written down. So, it became a new rule. When a column gets retagged as [22:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1341s) new rule. When a column gets retagged as [22:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1341s) new rule. When a column gets retagged as sensitive, search the codebase for [22:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1343s) sensitive, search the codebase for [22:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1343s) sensitive, search the codebase for anyone still reading it from the copy [22:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1345s) anyone still reading it from the copy [22:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1345s) anyone still reading it from the copy that's about to lose it. And that's how [22:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1347s) that's about to lose it. And that's how [22:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1347s) that's about to lose it. And that's how the rules get better. the misses get [22:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1350s) the rules get better. the misses get [22:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1350s) the rules get better. the misses get found and get turned into new rules. [22:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1354s) found and get turned into new rules. [22:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1354s) found and get turned into new rules. So let's pull it all together. You [22:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1357s) So let's pull it all together. You [22:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1357s) So let's pull it all together. You remember the pull request from the [22:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1358s) remember the pull request from the [22:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1358s) remember the pull request from the start, the one that was green in CI. [22:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1361s) start, the one that was green in CI. [22:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1362s) start, the one that was green in CI. Three things happened to it. [22:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1364s) Three things happened to it. [22:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1364s) Three things happened to it. First, scripts proved what a source of [22:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1367s) First, scripts proved what a source of [22:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1367s) First, scripts proved what a source of truth could prove. The contracts, the [22:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1370s) truth could prove. The contracts, the [22:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1370s) truth could prove. The contracts, the tags, the registered lineage and [22:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1372s) tags, the registered lineage and [22:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1372s) tags, the registered lineage and performance all passed. [22:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1374s) performance all passed. [22:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1374s) performance all passed. Second, the agent read the code and [22:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1377s) Second, the agent read the code and [22:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1377s) Second, the agent read the code and found what the scripts could not see. A [22:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1379s) found what the scripts could not see. A [22:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1379s) found what the scripts could not see. A job that Lineage never knew about, [23:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1381s) job that Lineage never knew about, [23:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1381s) job that Lineage never knew about, quietly waiting to fail on a wider [23:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1383s) quietly waiting to fail on a wider [23:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1383s) quietly waiting to fail on a wider column, it flagged the whole chain and [23:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1385s) column, it flagged the whole chain and [23:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1385s) column, it flagged the whole chain and escalated. Third, a person confirmed the [23:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1389s) escalated. Third, a person confirmed the [23:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1389s) escalated. Third, a person confirmed the job was live. Uh, the downstream table [23:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1391s) job was live. Uh, the downstream table [23:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1391s) job was live. Uh, the downstream table was fixed first and only then did the PR [23:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1393s) was fixed first and only then did the PR [23:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1394s) was fixed first and only then did the PR merge. No incident. And behind all of [23:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1397s) merge. No incident. And behind all of [23:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1397s) merge. No incident. And behind all of that, the reviewer keeps getting better [23:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1400s) that, the reviewer keeps getting better [23:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1400s) that, the reviewer keeps getting better because every correction a person makes [23:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1402s) because every correction a person makes [23:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1402s) because every correction a person makes can become a rule. These results are as [23:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1405s) can become a rule. These results are as [23:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1405s) can become a rule. These results are as of August 31st, 2026. So you might ask, [23:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1409s) of August 31st, 2026. So you might ask, [23:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1409s) of August 31st, 2026. So you might ask, Jas, does it all work? [23:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1412s) Jas, does it all work? [23:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1412s) Jas, does it all work? Yes, about 60% of the data pipeline PRs [23:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1415s) Yes, about 60% of the data pipeline PRs [23:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1415s) Yes, about 60% of the data pipeline PRs merged without a human review. [23:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1418s) merged without a human review. [23:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1418s) merged without a human review. We have seen about 75% less manual [23:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1420s) We have seen about 75% less manual [23:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1420s) We have seen about 75% less manual review effort. The agent specific [23:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1423s) review effort. The agent specific [23:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1423s) review effort. The agent specific comments and reasoning help focus the [23:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1425s) comments and reasoning help focus the [23:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1425s) comments and reasoning help focus the review rather than making a person [23:47](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1427s) review rather than making a person [23:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1428s) review rather than making a person reconstruct every risk from scratch. [23:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1431s) reconstruct every risk from scratch. [23:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1431s) reconstruct every risk from scratch. The time saved comes to about 132 [23:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1435s) The time saved comes to about 132 [23:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1435s) The time saved comes to about 132 engineering hours a month. Incidents [23:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1437s) engineering hours a month. Incidents [23:57](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1437s) engineering hours a month. Incidents tied to these PRs are down about 30%. [24:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1441s) tied to these PRs are down about 30%. [24:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1441s) tied to these PRs are down about 30%. And that does not mean human free [24:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1443s) And that does not mean human free [24:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1443s) And that does not mean human free review. The protected decisions still [24:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1446s) review. The protected decisions still [24:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1446s) review. The protected decisions still reach a person by design. The goal is [24:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1449s) reach a person by design. The goal is [24:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1449s) reach a person by design. The goal is less review work, not removing [24:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1451s) less review work, not removing [24:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1451s) less review work, not removing accountability. And those are the [24:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1454s) accountability. And those are the [24:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1454s) accountability. And those are the results. Now, let's look at the [24:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1456s) results. Now, let's look at the [24:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1456s) results. Now, let's look at the architecture choices we got wrong. [24:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1460s) Here is what we got wrong today. Layer 1 [24:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1463s) Here is what we got wrong today. Layer 1 [24:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1463s) Here is what we got wrong today. Layer 1 and layer 2 are separate systems [24:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1466s) and layer 2 are separate systems [24:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1466s) and layer 2 are separate systems triggered independently and running in [24:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1468s) triggered independently and running in [24:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1468s) triggered independently and running in parallel. Each produces its own result [24:31](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1471s) parallel. Each produces its own result [24:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1472s) parallel. Each produces its own result that seemed like an obvious design when [24:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1473s) that seemed like an obvious design when [24:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1473s) that seemed like an obvious design when we built it, [24:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1475s) we built it, [24:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1475s) we built it, but neither sees the others pass fail or [24:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1478s) but neither sees the others pass fail or [24:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1478s) but neither sees the others pass fail or outage state. Layer 2 knows what the [24:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1481s) outage state. Layer 2 knows what the [24:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1481s) outage state. Layer 2 knows what the scripts are configured to cover, not [24:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1483s) scripts are configured to cover, not [24:43](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1483s) scripts are configured to cover, not what happened when they ran. Merge still [24:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1486s) what happened when they ran. Merge still [24:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1486s) what happened when they ran. Merge still needs passing checks and an approving [24:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1488s) needs passing checks and an approving [24:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1488s) needs passing checks and an approving review, but no single system has a full [24:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1491s) review, but no single system has a full [24:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1491s) review, but no single system has a full runtime picture. [24:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1493s) runtime picture. [24:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1493s) runtime picture. The agent also cannot call live layer 1 [24:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1496s) The agent also cannot call live layer 1 [24:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1496s) The agent also cannot call live layer 1 checks. Its tools only read, search and [24:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1499s) checks. Its tools only read, search and [24:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1499s) checks. Its tools only read, search and list the repository files. If a judgment [25:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1502s) list the repository files. If a judgment [25:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1502s) list the repository files. If a judgment needs a live fact from a live production [25:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1505s) needs a live fact from a live production [25:05](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1505s) needs a live fact from a live production system, it cannot fetch that fact [25:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1507s) system, it cannot fetch that fact [25:07](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1507s) system, it cannot fetch that fact itself. [25:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1509s) itself. [25:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1509s) itself. When one site cannot finish, we wait for [25:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1512s) When one site cannot finish, we wait for [25:12](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1512s) When one site cannot finish, we wait for a human. There is no automatic fallback. [25:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1515s) a human. There is no automatic fallback. [25:15](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1515s) a human. There is no automatic fallback. These are current limitations and they [25:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1517s) These are current limitations and they [25:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1517s) These are current limitations and they motivate the proposal. On the next [25:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1519s) motivate the proposal. On the next [25:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1519s) motivate the proposal. On the next slide, platform team is exploring one [25:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1522s) slide, platform team is exploring one [25:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1522s) slide, platform team is exploring one entry point. This is a proposal and not [25:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1525s) entry point. This is a proposal and not [25:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1525s) entry point. This is a proposal and not what's running today. The left panel is [25:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1527s) what's running today. The left panel is [25:27](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1527s) what's running today. The left panel is the current design and the right is [25:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1530s) the current design and the right is [25:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1530s) the current design and the right is where we want to go. A classifier looks [25:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1533s) where we want to go. A classifier looks [25:33](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1533s) where we want to go. A classifier looks at the whole PR then decides which [25:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1535s) at the whole PR then decides which [25:35](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1535s) at the whole PR then decides which changes to review separately and which [25:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1537s) changes to review separately and which [25:37](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1537s) changes to review separately and which to review together. [25:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1539s) to review together. [25:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1539s) to review together. This is a whole PR decision, not [25:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1541s) This is a whole PR decision, not [25:41](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1541s) This is a whole PR decision, not independent routing of each file. [25:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1544s) independent routing of each file. [25:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1544s) independent routing of each file. The agent would also get the layer 1 [25:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1546s) The agent would also get the layer 1 [25:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1546s) The agent would also get the layer 1 checks as collable tools alongside its [25:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1549s) checks as collable tools alongside its [25:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1549s) checks as collable tools alongside its repository tools. When judgment needs a [25:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1552s) repository tools. When judgment needs a [25:52](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1552s) repository tools. When judgment needs a live fact, it could request the [25:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1553s) live fact, it could request the [25:54](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1554s) live fact, it could request the appropriate check instead of being [25:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1555s) appropriate check instead of being [25:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1555s) appropriate check instead of being limited to code alone. [25:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1558s) limited to code alone. [25:58](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1558s) limited to code alone. The aim here is one system with the [26:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1560s) The aim here is one system with the [26:00](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1560s) The aim here is one system with the whole PR in view. We haven't built this [26:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1562s) whole PR in view. We haven't built this [26:02](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1562s) whole PR in view. We haven't built this yet. So, it's a direction, not a result. [26:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1566s) yet. So, it's a direction, not a result. [26:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1566s) yet. So, it's a direction, not a result. Let me close with what we would carry [26:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1568s) Let me close with what we would carry [26:08](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1568s) Let me close with what we would carry forward. [26:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1570s) forward. [26:10](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1570s) forward. So, here is the whole talk in three [26:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1571s) So, here is the whole talk in three [26:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1571s) So, here is the whole talk in three lines. [26:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1573s) lines. [26:13](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1573s) lines. Deterministic evaluations. First, if [26:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1576s) Deterministic evaluations. First, if [26:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1576s) Deterministic evaluations. First, if something can be proven against a source [26:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1577s) something can be proven against a source [26:17](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1577s) something can be proven against a source of truth, let a script prove it. Don't [26:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1580s) of truth, let a script prove it. Don't [26:20](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1580s) of truth, let a script prove it. Don't spend a judgment call on a yes or no [26:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1582s) spend a judgment call on a yes or no [26:22](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1582s) spend a judgment call on a yes or no fact. It's not just cleaner, it's [26:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1585s) fact. It's not just cleaner, it's [26:25](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1585s) fact. It's not just cleaner, it's cheaper, too. Every fact a script proves [26:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1588s) cheaper, too. Every fact a script proves [26:28](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1588s) cheaper, too. Every fact a script proves is review work that never needs AI. So, [26:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1590s) is review work that never needs AI. So, [26:30](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1590s) is review work that never needs AI. So, the cost of running this stays down. [26:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1594s) the cost of running this stays down. [26:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1594s) the cost of running this stays down. Flag never clear. Not for every PR, but [26:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1598s) Flag never clear. Not for every PR, but [26:38](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1598s) Flag never clear. Not for every PR, but for the changes that actually scare [26:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1600s) for the changes that actually scare [26:40](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1600s) for the changes that actually scare people, like sensitive grants, migration [26:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1602s) people, like sensitive grants, migration [26:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1602s) people, like sensitive grants, migration intent, and low confidence verdicts. [26:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1605s) intent, and low confidence verdicts. [26:45](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1605s) intent, and low confidence verdicts. There, the agent's job stops at [26:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1608s) There, the agent's job stops at [26:48](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1608s) There, the agent's job stops at surfacing the risk and the reasoning [26:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1610s) surfacing the risk and the reasoning [26:50](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1610s) surfacing the risk and the reasoning behind it. The decision stays with the [26:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1613s) behind it. The decision stays with the [26:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1613s) behind it. The decision stays with the person every time. [26:55](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1615s) person every time. [26:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1616s) person every time. Count the false escalations. A gate that [26:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1619s) Count the false escalations. A gate that [26:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1619s) Count the false escalations. A gate that escalates everything is not actually [27:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1621s) escalates everything is not actually [27:01](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1621s) escalates everything is not actually saving anyone any time. It's just moving [27:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1623s) saving anyone any time. It's just moving [27:03](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1623s) saving anyone any time. It's just moving the work around. measure that cost the [27:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1626s) the work around. measure that cost the [27:06](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1626s) the work around. measure that cost the same way you measure a miss. So I [27:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1629s) same way you measure a miss. So I [27:09](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1629s) same way you measure a miss. So I started with a pull request that was [27:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1631s) started with a pull request that was [27:11](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1631s) started with a pull request that was green in CI and a problem PRs arriving [27:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1634s) green in CI and a problem PRs arriving [27:14](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1634s) green in CI and a problem PRs arriving at machine speed and too few people with [27:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1636s) at machine speed and too few people with [27:16](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1636s) at machine speed and too few people with the context to review them safely. We [27:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1639s) the context to review them safely. We [27:19](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1639s) the context to review them safely. We built a hybrid reviewer scripts for the [27:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1641s) built a hybrid reviewer scripts for the [27:21](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1641s) built a hybrid reviewer scripts for the facts and agent for judgment and people [27:23](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1643s) facts and agent for judgment and people [27:24](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1644s) facts and agent for judgment and people for the protected calls. We walked one [27:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1646s) for the protected calls. We walked one [27:26](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1646s) for the protected calls. We walked one PR through all three saw it get lighter [27:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1649s) PR through all three saw it get lighter [27:29](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1649s) PR through all three saw it get lighter without going human free and saw how it [27:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1652s) without going human free and saw how it [27:32](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1652s) without going human free and saw how it learns from the corrections. We told you [27:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1654s) learns from the corrections. We told you [27:34](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1654s) learns from the corrections. We told you what we got wrong and we left you with [27:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1656s) what we got wrong and we left you with [27:36](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1656s) what we got wrong and we left you with three lessons. Prove what you can [27:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1659s) three lessons. Prove what you can [27:39](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1659s) three lessons. Prove what you can cheaply. Flag and never clear the scary [27:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1662s) cheaply. Flag and never clear the scary [27:42](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1662s) cheaply. Flag and never clear the scary changes. And count the false [27:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1664s) changes. And count the false [27:44](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1664s) changes. And count the false escalations. [27:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1666s) escalations. [27:46](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1666s) escalations. So who reviews the bots? Scripts for [27:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1669s) So who reviews the bots? Scripts for [27:49](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1669s) So who reviews the bots? Scripts for everything a source of truth can prove, [27:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1671s) everything a source of truth can prove, [27:51](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1671s) everything a source of truth can prove, an agent for everything that needs [27:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1673s) an agent for everything that needs [27:53](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1673s) an agent for everything that needs context and judgment, and humans for the [27:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1676s) context and judgment, and humans for the [27:56](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1676s) context and judgment, and humans for the calls only they get to make. Thank you [27:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1679s) calls only they get to make. Thank you [27:59](https://www.youtube.com/watch?v=MQiojLnK-E0&t=1679s) calls only they get to make. Thank you for listening.