60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood

0:00 Picture a pull request. It changes a schema in one of our data pipelines. It goes green in CI. The tests pass. The linder is happy. And everything about it looks clean. So, is it safe to merge?

0:15 Because a change like this can still break something downstream. Maybe it blows a freshness SLA. Or maybe it breaks a consumer three teams away who never saw it coming. Or maybe it exposes

0:26 data that never supposed to be reachable. That gap between CI past and this change was actually safe is the gap my team has to close every day. Hello everyone, I'm Jas. I'm an engineer on

0:40 the data platform team at Robin Hood. I design and build the distributed data injection and processing frameworks behind our data lake. The ones teams across the company rely on to make

0:51 decisions with data. Notice the title of this talk. The bots in it are the ones opening the pull requests and as you will see the ones reviewing them as well.

1:04 Today I want to show you how we answer that question when the PRs are coming in faster than any of us can read them. So here's the problem in one sentence. Pull requests to our production data

1:18 pipelines now arrive at machine speed. But reviewing them safely still runs at human speed. First, a quick word on the scope of this talk because it shapes everything else

1:30 in this talk. AI coding assistants have made pull requests cheap to open. A new query, a schema change, or an access grant now takes just a few minutes. And an

1:43 assistant can open pull requests far faster than any team of humans ever could. The PRs I'm talking about today are the PRs to our data pipelines like schema

1:55 changes, pipeline code, configuration changes, and grants that give someone access to data. Data platform is the team that owns reviewing that kind of PR. So when those

2:07 PRs pile up, they pile up on us. And the reviewer I'm about to describe is the system we built to handle them. So the people who feel this are a small group. The engineers who carry the

2:19 context to say what is safe to merge into production and everyone who is waiting on them. So why is this a problem? AI open PRs just keep arriving.

2:34 And to be fair, the safety net we already had works. CI checks together with a human review have always caught a lot of real bugs like broken code, inefficient queries and risks like

2:48 downstream breakage or sensitive data leaks. But the human half only works at the human speed and only with the context a person happens to carry in their head. Who reads this table? How

3:00 big is it? And how sensitive is the data in it? The PRs are coming in at the machine speed, but the number of people who understand what is safe to merge into a production data pipeline did not

3:12 grow at all. That's the mismatch. The queue builds up in front of the few people with the context and that's where things start to slip. So the problem we set out to solve was

3:24 this. Build a PR reviewer using AI that can keep up with the speed at which PRs are being opened. So here's the whole answer before we get into the details. Three duties, three

3:39 owners. Scripts prove what can be proven. An agent judges what needs context and people own the calls only they should make. We took a hybrid approach. Scripts

3:52 for the facts and AI for judgment. We did not want AI reviewing every change when some questions already have an authoritative answer. And that's layer one. Deterministic

4:06 scripts check a source of truth like the schema registry or a catalog. A failed check turns CI red and blocks the PR until it is fixed. Layer two is the agent. The agents rules

4:20 tell it what layer 1 already checks. So it focuses on the risks those checks cannot catch. It reviews the PR, scores the risk and approves or escalates to a person. Both these layers review the

4:34 same PR. Merge requires passing checks and an approving review. And this also keeps AI costs down. We use AI for judgment, not to rediscover facts a script can look up.

4:50 So here is where we are going today. We have covered the problem and the approach. Now I'll walk you through the hybrid reviewer we built and the decisions we deliberately keep with the

5:00 people. After that how the reviewer learns from the human feedback and what results we have seen so far. And we'll finish with the lessons we would pass on to you.

5:12 Three things to listen for as we go. Scripts for facts and agent for the judgment and people for the protected calls. I'll come back to all three at the end.

5:24 So to make this concrete, let me show you the one PR will follow through the rest of this talk. This PR makes two changes to a table in our data lake. It adds a new column and

5:39 widens an existing column from integer to big integer. It's the kind of PR we see all the time. On the left is an upstream source table. Our injection pipeline reads from it and writes into a

5:51 table in the data lake. call it our table. The first change, the source has gained a new column and this PR makes injection bring that new column into our lake

6:03 table. The second change is to a column of a table already has its type gets widened from integer to big integer so it can hold bigger values

6:16 on its own. That sounds completely harmless. Widening a type is about the safest schema change you can make, right? Well, whenever a column changes, the

6:26 real question is who reads it downstream? Some of those readers are registered in our lineage tool. So, we know about them. But keep a second question in the back

6:36 of your mind. Are those all of them? Hold on to this picture. We will come back to this exact diagram each time the PR moves forward through the reviewer. So let's start with layer one. Layer one

6:52 is the scripts and their whole job is checking facts in the live systems. Things the code alone cannot tell you. Here are the two of those checks and what they said about our running PR.

7:07 First data contracts. When a PR changes a schema, a script compares the schema and keys against a source of truth, the schema registry. for our PR. The question is simple. Does

7:20 that new column actually exist upstream as registered in the registry? It does. So that check passes. Second require tags. Every table needs an owner and any PII has to be labeled

7:35 in the catalog. The script checks the catalog directly. Does our table have an owner? Yes. Is its PII labeled? Yes. pass again. And if either of those had failed, there

7:50 is no debate and no model involved. CI goes red like any other failing checks and the PR is blocked until it is fixed. Contracts and tags are two of the layer 1 checks. Next, the two that look at

8:05 downstream impact and performance. So these are the other two layer 1 checks. They look past the table itself, what happens downstream and whether the job will still run in time.

8:19 First, the lineage blast radius. When a column changes, like the one our PR widens, a script walks every downstream job the lineage tool has on record for that column and asks a simple question.

8:32 Does any of them break? It's a real check against a real system. But notice the boundary I have drawn. It can only see what is recorded in lineage. Its view of downstream is exactly as

8:46 complete as that record. No more. Second, the performance. A correct change can still be too slow or too expensive. So the CI performance test runs the change in staging at the same

9:00 data volume as production using mock data. So that tells us two things before the PR merges. whether the cluster is sized right for the job and whether the job

9:10 will still make its freshness SLA. Those are the kind of problems that otherwise show up weeks later as an incident. So that's the full set of layer 1 checks. Let's see what they said about our PR.

9:27 So back to our running PR. Same picture as before. Here's what the layer one said about it. The data contract check went to the schema registry and asked whether the

9:39 new column really exists upstream as register. It does and the check passes. The leanage blast radius check walked every registered consumer of the widen column. The jobs the lenage tool has on

9:53 record. None of them break when the column goes from integer to big integer. They all look fine. Along with that tags pass, performance passes. Every check is green.

10:05 As far as the scripts can see, this PR is safe. But I chose those words carefully. As far as the scripts can see, everything on the slide is a statement about the registered picture,

10:17 the jobs and the columns our systems have a record of. And keep that in mind. It is not the whole story. Now let's look at the agent with the diff. What actually changed?

10:31 The PR description and the commit messages explain the author's intent, but intent never explains away a real finding. If the description says it is safe and the code says otherwise, the

10:43 code wins. File tools let the agent read, search, and list the files across the repository. It follows changes beyond the diff into pipeline definitions and

10:54 access grants looking for the relationships the author may have missed. These are the repository repository tools only. The agent cannot query the

11:05 live catalog lineage tool or production state. Life facts stay layer 1's job. The agent reasons over the code. So why walk that code for a schema change? Because registered lineage is not the

11:20 whole picture. The rules define where the judgment belongs. They are configured by the change type in the PR knowing what layer 1 already covers. They target query anti

11:33 patterns, sensitive data asks and consumers outside lineage. For example, an unpartitioned cross join can run correctly and still hammer a cluster. A grant can look ordinary and

11:46 still open sensitive data. And for the schema changes, the agent walks the pipeline definitions and access grants in the repository to find consumers that were not registered. It's

11:58 a tedious search uh a person would otherwise do by hand. Let's return to our PR and see what that search found. So back to our running PR. Same picture.

12:10 Layer 1 said every registered consumer of the vid column looks fine. Now the agent walks the code and it finds a job that lineage never knew about. It's right there in the

12:23 repository. It reads our widen column from our late table but it was never registered in the lineage which is exactly why the layer 1 check could not see it.

12:34 And that job takes the value and writes it into another table. And in that other table, the column is still an integer. Now follow it through. Our column is a big integer. Now the first time a value

12:48 comes through that's bigger than an integer can hold that right fails. A valid schema change green in CI passes every registered check and a downstream job still breaks.

13:03 The agent flagged exactly this chain in its review comment with the reasoning. So a person could see the whole path from our column to the failing right. We'll get to what happened next in a

13:14 moment. So why did the lineage miss that job? Because the enterprise data world is messy. Workloads run across different query engines and each engine has to

13:25 publish its own lineage. If one does not publish, that relationship is missing from the picture. There's a single catalog that automatically sees everything. So the registered picture is

13:36 useful but it is not the whole picture. Now I don't want to oversell the codewalk. It's genuinely useful but it cannot see everything. The agent can search what is in the

13:50 repo. Pipeline code access grants and the jobs defined there that read our tables. That's where it found our unregistered job. But plenty of data use lives outside the codebase. Ad hawk

14:04 queries that someone runs by hand. Dashboards, those are completely invisible to a repository search. A consumer out there never shows up, no matter how carefully the agent looks. So

14:16 we make the agent honest about that limit. If it does not find anything, it does not get to say no downstream consumers exist. It says it found none in the places it searched. None found is

14:28 not the same as none exist and the agent never gets to claim an allcle. So the agent has walked the code. What does it actually produce? A review the same shape a good human reviewer would

14:43 give you. First review comments on the specific lines that matter each with its reasoning. Not awake looks risky but this line changes this column. And here

14:56 is the job that reads it. Second, a risk score. How severe is what it found. That's what drives the next step. Then it makes a call. If there is no serious risk, it can approve. If

15:09 there is a risk that needs a person, it escalates to a human and the comments and reasoning go with it. So the person has the context to investigate instead of starting from scratch.

15:22 And to be clear about what it takes to merge, layer 1's checks have to pass and the review has to approve either the agent or after an escalation a person. Both have to say yes. Now let's see what

15:36 that looked like for our PR. Here is how our running PR actually ended. Same picture. The registered consumers look fine and the core walk found an unregistered job that writes

15:50 our widened column into a table where it is still an integer because the agent found a real downstream break. It scored the risk high and escalated to a human with the

16:01 whole chain spelled out in its review comment. Now the human does the part the agent cannot. The agent only reasons over the repo. It cannot see whether the job is actually running in production.

16:13 human reviewer checked and confirmed that the consumer the agent found was live and active. So this was not a theoretical risk. The PR waited the f the fix was about order not about

16:28 rejecting the change. The downstream table was updated first so its column could handle the wider type and only then did our PR merge and deploy. No incident. The scripts in

16:41 layer 1 said safe. The agent found the gap and a human made the call. And that was not a one-time thing. Most changes can be cleared by the agent. When the checks pass and the agent is

16:56 confident in its verdict, it approves on its own. But there's a whole category of changes where the agent never gets to say approved. And no matter how confident it sounds,

17:06 grants touching sensitive data are one. Who can reach sensitive data is a decision we want a person to own every time. A migration that looks like it changes

17:18 intent, not just mechanics, is another. If a change alters what the data means rather than just how it is moved or stored, a person decides. And if the agent itself is not confident

17:32 in its own verdict, that's the third. It does not get to round up to approve just because nothing concrete came back wrong. In all the all the three cases, its job is only able to surface the risk

17:45 and its reasoning. The decision stays with a person. That is one PR. Now, how does this hold up across every PR we see? And how does the reviewer get better over time?

17:59 How do we know the flags are useful or whether a rule change helps? We start with the agents verdict and its reasoning in the review comments. A person replies agreeing or disagreeing

18:12 and explaining why. That explanation is what makes the feedback useful rather than just a vote. Those replies are collected every day. Once a week, the on call reads the

18:25 feedback and updates the review rules by hand. Those rules shape the next review. I know this is manual today. Nothing retrains itself and no rule changes automatically. But we have an idea to

18:38 let an incident investigation agent help with that weekly pass. But it is not built. Without the feedback, every rule change would be a guess. So let me show you an example where a human corrected

18:50 the agent and we use that feedback to improve its next review. So here's a real one. And note the label in the corner. This is a separate example, not our schema change PR.

19:06 A column was marked as personal information in the source database. But by the time it lands in the data lake, the injection pipeline has already encrypted it. So the PR proposed

19:17 labeling it nonsensitive in the data lake even though the upstream data is still PII. The agent flagged this. From where it stood, the upstream says PII, so a

19:28 nonsensitive label looked wrong. The author pushed back and asked the agent to look at how similar columns on the other tables were already classified instead of going purely off the upstream

19:40 tagging. A human agreed with the author. The encryption before write made the nonsensitive label look okay for this column. And that kind of correction is

19:51 exactly the signal that teaches the rules to get smarter because the next time a similar column comes through that context can stop a needless escalation. One caution though, this depends on the

20:04 specific pipeline encrypting before the right. It's not a blanket rule that encryption always makes data nonsensitive. So how do we know whether the reviewer

20:16 is actually doing a good job? We look at two kinds of mistakes. Bad PRs it approved and fine PRs it unnecessarily send to a person. That matrix separates four possibilities.

20:31 A fine PR approved and a bad PR escalated are the two correct outcomes. A bad PR approved is a miss. That's dangerous because the change ships. A fine PR escalated is a false

20:44 escalation that costs a person's time. A gate that escalates everything may avoid misses, but it has not reduced the review load. As of August 31st, 2026, our miss rate

20:58 was about 2%. Mostly from the gaps in the rules rather than the failures of the judgment. We don't have a clean single false escalation number. But the dominant

21:10 cause is missing context like the encrypted column example that I just gave you. Misses give us new rules and false escalation show us context the reviewer

21:20 needs. So here is one of those misses. Again know the label. This is a separate example, not our schema change PR. We have an open copy of the sum of some data. some of its columns are about to

21:34 be rettagged and there is a downstream reader that reads those columns from the open copy. Privacy retag those columns as personal information. When that happens, they

21:46 move into a more restricted copy of the data and they are gone from the open copy. The retag looked fine and it merged. But what nobody checked was whether anything

21:57 downstream was still reading those columns from the old open copy. something was. And once the columns were gone, that reader broke. That's a miss. A PR that should have been escalated got

22:09 approved. It wasn't a failure of judgment. It was a gap in the rules. A coroner case nobody had written down. So, it became a new rule. When a column gets retagged as

22:21 sensitive, search the codebase for anyone still reading it from the copy that's about to lose it. And that's how the rules get better. the misses get found and get turned into new rules.

22:34 So let's pull it all together. You remember the pull request from the start, the one that was green in CI. Three things happened to it. First, scripts proved what a source of

22:47 truth could prove. The contracts, the tags, the registered lineage and performance all passed. Second, the agent read the code and found what the scripts could not see. A

22:59 job that Lineage never knew about, quietly waiting to fail on a wider column, it flagged the whole chain and escalated. Third, a person confirmed the job was live. Uh, the downstream table

23:11 was fixed first and only then did the PR merge. No incident. And behind all of that, the reviewer keeps getting better because every correction a person makes can become a rule. These results are as

23:25 of August 31st, 2026. So you might ask, Jas, does it all work? Yes, about 60% of the data pipeline PRs merged without a human review. We have seen about 75% less manual

23:40 review effort. The agent specific comments and reasoning help focus the review rather than making a person reconstruct every risk from scratch. The time saved comes to about 132

23:55 engineering hours a month. Incidents tied to these PRs are down about 30%. And that does not mean human free review. The protected decisions still reach a person by design. The goal is

24:09 less review work, not removing accountability. And those are the results. Now, let's look at the architecture choices we got wrong. Here is what we got wrong today. Layer 1

24:23 and layer 2 are separate systems triggered independently and running in parallel. Each produces its own result that seemed like an obvious design when we built it,

24:35 but neither sees the others pass fail or outage state. Layer 2 knows what the scripts are configured to cover, not what happened when they ran. Merge still needs passing checks and an approving

24:48 review, but no single system has a full runtime picture. The agent also cannot call live layer 1 checks. Its tools only read, search and list the repository files. If a judgment

25:02 needs a live fact from a live production system, it cannot fetch that fact itself. When one site cannot finish, we wait for a human. There is no automatic fallback.

25:15 These are current limitations and they motivate the proposal. On the next slide, platform team is exploring one entry point. This is a proposal and not what's running today. The left panel is

25:27 the current design and the right is where we want to go. A classifier looks at the whole PR then decides which changes to review separately and which to review together.

25:39 This is a whole PR decision, not independent routing of each file. The agent would also get the layer 1 checks as collable tools alongside its repository tools. When judgment needs a

25:52 live fact, it could request the appropriate check instead of being limited to code alone. The aim here is one system with the whole PR in view. We haven't built this

26:02 yet. So, it's a direction, not a result. Let me close with what we would carry forward. So, here is the whole talk in three lines.

26:13 Deterministic evaluations. First, if something can be proven against a source of truth, let a script prove it. Don't spend a judgment call on a yes or no fact. It's not just cleaner, it's

26:25 cheaper, too. Every fact a script proves is review work that never needs AI. So, the cost of running this stays down. Flag never clear. Not for every PR, but for the changes that actually scare

26:40 people, like sensitive grants, migration intent, and low confidence verdicts. There, the agent's job stops at surfacing the risk and the reasoning behind it. The decision stays with the

26:53 person every time. Count the false escalations. A gate that escalates everything is not actually saving anyone any time. It's just moving the work around. measure that cost the

27:06 same way you measure a miss. So I started with a pull request that was green in CI and a problem PRs arriving at machine speed and too few people with the context to review them safely. We

27:19 built a hybrid reviewer scripts for the facts and agent for judgment and people for the protected calls. We walked one PR through all three saw it get lighter without going human free and saw how it

27:32 learns from the corrections. We told you what we got wrong and we left you with three lessons. Prove what you can cheaply. Flag and never clear the scary changes. And count the false

27:44 escalations. So who reviews the bots? Scripts for everything a source of truth can prove, an agent for everything that needs context and judgment, and humans for the

27:56 calls only they get to make. Thank you for listening.