# AI-Generated Code Is Already Competing With Human Code — Daksh Gupta, Greptile
Daksh Gupta, Greptile
AI Engineer World's Fair 2026
[Talk page](https://ai.engineer/talks/474j-n1Ltxc-ai-generated-code-is-already-competing-human) · [Video](https://www.youtube.com/watch?v=474j-n1Ltxc)
## AI-Generated Code Is Already Competing With Human Code
Daksh Gupta examines Greptile’s enterprise pull-request data, finds similar quality signals for human and agent code, and explains why growing code volume changes what validation must accomplish.
### From completing a line to producing a pull request
[0:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=12s)
An agent opening a hundred pull requests a day makes an impressive story. It also raises a practical question: can a company with real customers safely use those changes? Daksh Gupta, co-founder of Greptile, approaches that question with skepticism earned from programming before autonomous agents. Producing code quickly and producing code a business can merge are different accomplishments.
Greptile already sits between a proposed change and its acceptance. Its agents inspect changed files and related code to look for bugs. They can also install dependencies, start the application in a sandbox, mock inputs, and click through the running app. That position provides both code-review observations and a reason to care about whether the author is a person or an agent.
The shift in coding tools explains why the question has become urgent. Gupta traces a progression from tab completion, through multifile editing in 2024, to agents in 2025 that could take a task and create an entire pull request. Each step expands the unit of work delegated to software: first a continuation, then coordinated edits, then a proposed contribution ready for review. The remaining question is whether that contribution holds up in an existing enterprise codebase.
### An author field misses much of the agent activity
[3:02](https://www.youtube.com/watch?v=474j-n1Ltxc&t=182s)
Greptile reviews more than a million pull requests a month across thousands of companies, usually enterprises or businesses with established products and customers. Before comparing quality, Gupta had to divide that stream into likely agent-written and human-written changes. The first attempt—looking for agent names in GitHub author fields—identified fewer than 1% of pull requests.
Source frame: An author field misses much of the agent activity
That result prompted a wider search for traces left by the tools:
- **Author fields:** An agent named as the author provides a direct metadata signal.
- **Co-author footers:** PR descriptions may credit Claude, Cursor, or another tool even when the author field does not.
- **Branch prefixes:** Agent-created branch names provide another indication that a tool participated in producing the pull request.
Consider the classification change implied by the footer check. A PR whose author field names no agent would escape the first detector. If its description credits Claude, the expanded detector now has evidence of agent involvement. Nothing about the code has changed; the observable change is in how the PR is classified. Gupta takes an agent writing the description or branch name as a reasonable sign that it also wrote much of the implementation. That remains an inference: these signals do not establish full autonomy, and an unmarked PR need not be entirely human-written.
With the combined signals, roughly a quarter of reviewed PRs appeared largely or entirely AI-generated. Looking back over the preceding twelve months, Gupta found fewer than 1% showing such evidence early in that period. He describes the adoption curve as continuous: individual model releases did not stand out as obvious discontinuities. The striking observation is the spread of the workflow through companies, rather than a single release-day jump.
### What would make an agent-written PR worse?
[5:13](https://www.youtube.com/watch?v=474j-n1Ltxc&t=313s)
Authorship answers who likely produced a change. Quality requires a separate definition. Gupta starts with reverts: a change that a team subsequently undoes is a useful candidate for a bad PR. He tracks GitHub’s revert branch naming pattern, which includes the original PR number and name, to connect an undo operation to its source change.
Source frame: What would make an agent-written PR worse?
The reported rates are about one revert per thousand Codex PRs, about 3.5 per thousand Devin PRs, and about 2.5 per thousand human PRs. Humans sit between the two agents rather than clearly outperforming both. These are observational comparisons using reverts and automated review findings as quality proxies; they do not establish equivalent task difficulty or general superiority.
One plausible explanation was task allocation. People might give agents small, well-scoped changes while retaining complicated work themselves. More complicated PRs expose more opportunities for failure, so a raw revert comparison could penalize humans for doing harder jobs. Gupta therefore examines PR size alongside reverts. He reports that the size comparison did not provide strong evidence that human PRs were better. This addresses the proposed size explanation, although size alone cannot capture everything that makes a task difficult.
The next signal is the severity of issues Greptile flags. P0, P1, and P2 are the severity labels used in the reported review findings; the talk does not define their thresholds. Counting findings in these categories asks whether similar revert rates conceal different amounts of problematic code caught before acceptance. Three of the four tested agents produced fewer flagged P0 issues than humans. Across the severity categories, Gupta again reports broadly comparable quality.
Finally, review rounds measure the work needed to make a proposal mergeable. In the workflow described here, an agent reads review comments, addresses them, and commits a revision to the same branch. A higher-quality starting point should, in principle, need fewer cycles. Tracking the rounds between opening and merging a PR produced another comparison with little reported difference between human and agent contributions.
### Similar overall quality, different ways to fail
[8:24](https://www.youtube.com/watch?v=474j-n1Ltxc&t=504s)
A similar total number of problems can hide a different mix of problems. Greptile makes about four comments per PR on average, giving Gupta several million comments across the preceding months. Searching that corpus for phrases such as SQL injection and N+1 query lets him compare specific failure categories across authors.
Source frame: Similar overall quality, different ways to fail
The comparison sets human frequency to 1× separately for each error category:
- **SQL injection:** Claude’s reported relative frequency is about 1.5× the human baseline.
- **Authentication bypass:** Devin’s reported relative frequency is about half the human baseline.
The ratios compare each agent with humans within a category. They do not say that SQL injection and authentication bypass occur equally often, or that one agent is uniformly safer. Because the analysis searches review-comment language, these are patterns in flagged findings rather than independently confirmed incident rates.
This changes the useful question from “Are agents bad at coding?” to which failures a particular tool tends to introduce. The aggregate comparisons weakened Gupta’s initial skepticism about enterprise use; the category comparisons preserve a reason to inspect the resulting code carefully. Agents can make meaningful contributions while retaining distinctive weaknesses.
### More code turns validation into the bottleneck
[9:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=594s)
Once agents can contribute useful code, the next constraint is the capacity to validate it. Gupta describes a large gap between typical Greptile users and the most prolific users, whose monthly output reaches hundreds or thousands of changes. Exact activity figures are uncertain because the recording and its description use different units and percentile figures. The operational concern is the same: manual review, testing, and external QA must keep pace with a much faster stream of proposed changes.
Greptile’s response starts with what must be established between a PR and a safe deployment. Gupta proposes three questions:
- **Present behavior:** Does the change violate the application’s user contract—the behavior users should be able to rely on?
- **Future risk:** Does the change make a later violation of that contract more likely?
- **Author intent:** Does the implementation accomplish what its author meant to change?
The questions separate three ways a change can disappoint. It can break existing behavior now, leave behind a risk that becomes a failure later, or preserve existing behavior while failing to deliver the requested improvement. This gives validation a purpose beyond accumulating review comments: establish whether the proposed change is safe and whether it does the intended job.
### Inspect the surrounding code, then try the running app
[11:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=692s)
How can validation connect a changed file to what users actually experience? Greptile combines two avenues described across the talk. Code analysis follows the change into related files, extending inspection beyond the diff. Sandbox execution installs dependencies, starts the app, supplies mocked inputs, and lets browser agents interact with it. One avenue examines the implementation and its relationships; the other looks for failures in running behavior.
The flow below answers where a proposed change becomes evidence for a revision. Findings from code inspection and browser interaction become review comments. The coding agent can then address those comments and commit a new version to the PR branch. The observable change is a revised proposal, and further review determines whether more work is needed before merge.
Gupta expects this combination to discover many issues and increase confidence in merging. The talk supplies the approach rather than a measured coverage guarantee, and it does not spell out how each of the three validation questions is conclusively answered. Its closing adoption signal is that almost a fifth of the PRs Greptile reviews are merged without human review or human testing. Greptile wants that share to grow while maintaining code quality. The ambition is to make validation scale with code production, so generating a useful change does not simply move an unmanageable workload onto reviewers.
From proposed code to a revised pull request
Code inspection and sandbox interaction provide findings for the same review-and-revision loop. The loop can repeat before merge.
- Pull request — A proposed code change enters review.
- Inspect code relationships — Agents examine changed files and related files for bugs.
- Run the app in a sandbox — Install dependencies, start the app, mock inputs, and use browser agents.
- Review findings — Comments identify issues for the coding agent to address.
- Commit a revision — The agent addresses comments and updates the same PR branch.
- Merge — Review rounds continue until the PR is ready to merge.
- Pull request → Inspect code relationships (Read changed and related files)
- Pull request → Run the app in a sandbox (Exercise running behavior)
- Inspect code relationships → Review findings (Flag issues)
- Run the app in a sandbox → Review findings (Discover failures)
- Review findings → Commit a revision (Address feedback)
- Commit a revision → Pull request (Another review round)
- Pull request → Merge (When ready)
## Key takeaways
- Agent authorship needs multiple signals. Author fields alone identified fewer than 1% of PRs; adding co-author footers and branch prefixes raised the estimate to roughly a quarter, with full autonomy still inferred.
- Greptile’s observational comparisons found broadly similar human and agent results across reverts, PR-size analysis, flagged issue severity, and review rounds before merge.
- Overall quality can conceal distinct failure patterns. Comment-based comparisons put Claude’s SQL-injection frequency at about 1.5× the human baseline and Devin’s authentication-bypass frequency at about half.
- Validation should examine existing user behavior, future failure risk, and author intent. Greptile combines inspection of changed and related code with sandboxed browser interaction to support that work.
## Resources
- [Original recording](https://www.youtube.com/watch?v=474j-n1Ltxc) — The complete AI Engineer recording, with paragraph citations tied to its source timestamps.
- [Greptile](https://greptile.com/) — The code-review product discussed in the talk; link supplied in the recording description.
- [Daksh Gupta](https://dakshgupta.com/) — The speaker website supplied in the recording description.
## Slides
### What is Greptile?
Greptile agents review pull requests using related code and sandbox tests. The example flags a removed await: the function can return success before the charge resolves, even if it fails.
[0:19](https://www.youtube.com/watch?v=474j-n1Ltxc&t=19s) [Greptile describes AI agents that review and test pull requests, alongside an example code-review comment.](https://assets.ai.engineer/sites/aiecode2025/sha256/a884faedee041fc409a242f55936234bebf15262639ea9c383477731c1fa60ea.webp)
What is Greptile?
AI agents that review and test pull requests.
greptile
commented on api/billing/charges.ts
43 const customer = await db.customers.find(id);
44 - await stripe.charge(customer.token, amount);
44 + stripe.charge(customer.token, amount);
45 return { ok: true };
P1 Charge may fail silently
The await was removed from stripe.charge(), so the function returns { ok: true } before the charge resolves — including when it fails.
Fix in Cursor
Fix in Claude
Fix in Codex
greptile
### Evolution of AI coding agents
The workflow moves from accepting autocomplete suggestions to reviewing multi-file edits, then to giving an agent a ticket and receiving a pull request.
[0:56](https://www.youtube.com/watch?v=474j-n1Ltxc&t=56s) [Timeline from autocomplete in 2021–2023 to multi-file agents in 2024 and end-to-end agents from 2025 onward.](https://assets.ai.engineer/sites/aiecode2025/sha256/e41ab86e3f66791f9bc16e5318ce842530e5a71984d8addf053ef8e29678c899.webp)
2021–2023
Autocomplete suggests a line; humans tap tab to accept it.
const total = items.reduce((a, b) => a + b.price, 0)
2024
Multi-file agents edit across a repo, with humans still doing review.
utils/format.ts
[illegible]
api/cart.ts
[illegible]
checkout.tsx
- {total}
+ {fmt(total)}
2025–today
End-to-end agents: human writes a ticket, AI ships the PR.
ENG-247
Format cart totals to 2 decimal places
Bug P2
#1842
fix(cart): format totals to 2dp
+12 -4 3 files
Open
greptile
### Identifying fully-vibed PRs is hard
Gupta introduces three signals for finding AI-generated PRs in Greptile’s review data, a task he found surprisingly difficult.
[3:01](https://www.youtube.com/watch?v=474j-n1Ltxc&t=181s) [Section slide introducing three signals used to identify fully-vibed pull requests.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b3a5b87d51005541fbff9c81719451aa2fb9003ad3d9680b146418f38766a40.webp)
Identifying fully-vibed PRs is hard
Three signals we used to find them.
greptile
### GitHub author field
Gupta found fewer than 1% of PRs named Codex, Claude or Cursor as author. He expected a higher share of code to be fully AI-generated, so he looked for other signals.
[3:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=212s) [The first signal is the GitHub author field, illustrated by a grid of 100 pull requests with one attributed to an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/01b5ba64e9d2eb53067f621bfb4bf15a36d22d4bae3bff832f359fad51d64000.webp)
Signal 1
GitHub author field
100 PRs · 1 attributed to an agent
human-attributed
agent-attributed
greptile
### PR description footer
Gupta uses AI attribution in PR footers as another signal of potentially AI-generated code. The example footer credits Claude Code and lists Claude as a co-author.
[3:55](https://www.youtube.com/watch?v=474j-n1Ltxc&t=235s) [The second signal is a pull request description footer indicating that an AI coding tool generated or co-authored the change.](https://assets.ai.engineer/sites/aiecode2025/sha256/cefb45cb6a027c3144c6b415730ec55093e50424982897c18338ee09421f6d26.webp)
Signal 2
PR description footer
feat: add idempotency keys to webhooks
opened 2 hours ago by alex-h
Adds idempotency keys to all outbound webhook deliveries so retries don’t double-fire downstream handlers.
Tested against the staging Stripe environment.
—
🤖 Generated with Claude Code
Co-Authored-By: Claude
Interface text:
greptile
### Branch name prefix
Gupta uses agent-prefixed branch names as a signal that a PR was largely AI-generated. The example highlights Codex and Cursor prefixes.
[4:11](https://www.youtube.com/watch?v=474j-n1Ltxc&t=251s) [A branch list highlights examples with Codex and Cursor prefixes.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4ad1858df2289dbd6704cab8e5f4a3e9db312d4f4ab7426cb6a6ad2b1127176.webp)
Signal 3
Branch name prefix
Branches
5 of 1,284
feature/payment-retry — 2 days ago
codex/add-rate-limit-middleware — 1 day ago
fix/login-redirect — 5 hours ago
cursor/refactor-billing-types — 3 hours ago
chore/bump-deps — 12 min ago
greptile
### 27.6% of PRs had strong evidence of full AI authorship
Gupta treats agent-written branch names or PR descriptions as signals of largely AI-generated PRs. Of the PRs Greptile reviewed in April, 27.6% had strong evidence of being fully vibed.
[4:34](https://www.youtube.com/watch?v=474j-n1Ltxc&t=274s) [Large 27.6% figure beside a statement about April PRs showing strong evidence of being fully vibed.](https://assets.ai.engineer/sites/aiecode2025/sha256/90dbfdaf01ff799c276542da1065ac59eee5eeea71d611300baafe575ac2b50b.webp)
27.6%
of PRs in April had strong evidence of being fully vibed
greptile
### Share of PRs with evidence of full AI authorship
The share of merged PRs with evidence of full AI authorship rose from 0.86% in February 2025 to 27.6% in April 2026. Signals include bot authors, co-authored-by footers, and codex/ branch prefixes.
[4:44](https://www.youtube.com/watch?v=474j-n1Ltxc&t=284s) [Monthly bar chart rises from 0.86% in February 2025 to 27.6% in April 2026.](https://assets.ai.engineer/sites/aiecode2025/sha256/cad093cd681f0cdd5bd36b0fc284c5203d012f26ed7b112ece16356d1a4d76e6.webp)
[ FIG. 01 · SHARE OF PRs WITH EVIDENCE OF FULL AI AUTHORSHIP ]
Y-axis: % OF MERGED PRs
Y-axis ticks: 0%, 10%, 20%, 30%
Timeline: Feb 2025, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec, Jan 2026, Feb, Mar, Apr
X-axis: MONTH
0.86%
27.6%
monthly share
PRs with bot author, co-authored-by footer, or codex/ branch prefix
greptile
### Are these PRs any good?
Gupta turns to the quality of fully AI-generated pull requests, asking whether they are any good.
[5:11](https://www.youtube.com/watch?v=474j-n1Ltxc&t=311s) [Slide asks whether these PRs are any good.](https://assets.ai.engineer/sites/aiecode2025/sha256/48e6fdc9ede140f1c179dd705a4df36d0ec60c11cb00a3bb9813403fd8885e59.webp)
Are these PRs any good?
greptile
### What does it mean for a PR to be good?
Before judging AI-generated PRs, Gupta asks how to define a good pull request.
[5:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=321s) [Slide asks what it means for a PR to be good.](https://assets.ai.engineer/sites/aiecode2025/sha256/1c93b6c38a22a853b889d9e282e0bca89da76f5617df403463e312c7e8b54737.webp)
what does it mean for a
PR to be good?
greptile
### Revert rates
Gupta uses revert rates as a proxy for bad PRs, tracking GitHub branches named for the reverted PR. The example undoes a webhook change after production webhooks started double-firing.
[5:29](https://www.youtube.com/watch?v=474j-n1Ltxc&t=329s) [Defines a revert as a follow-up commit undoing a merged PR and shows an example revert.](https://assets.ai.engineer/sites/aiecode2025/sha256/059b6198021a4e8d762ca39d90550f437832c7173813356fbc3ec0a9d1a34130.webp)
Method 1
Revert rates
A revert is a follow-up commit that undoes a merged PR — usually because it broke something in production.
Pull requests · #4821
merged
Revert "feat: add idempotency keys to webhooks"
opened by alex-h · 18 minutes after merge
branch revert-4807-feat/idempotency-keys
reverts #4807
Reverts the changes from #4807. Production webhooks started double-firing 12 minutes after deploy.
greptile
### Revert rates by author
Codex had the lowest revert rate at 1.19 per 1,000 merged PRs; Devin had the highest at 3.50. Humans fell between the two at 2.72.
[5:53](https://www.youtube.com/watch?v=474j-n1Ltxc&t=353s) [Bar chart comparing reverts per 1,000 merged PRs by author; Codex is lowest at 1.19 and Devin highest at 3.50.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ffb34ba554465a4fa643333266551c038a85bea4f131cf2ea7407aa8d8767ef.webp)
Revert rates by author
Codex 1.19
Claude 1.80
human 2.72
Cursor BG 3.41
Devin 3.50
X-axis: REVERTS PER 1,000 MERGED PRs
X-axis ticks: 0, 1, 2, 3, 4
greptile
### Revert rate by PR size
Gupta compares reverts per 1,000 merged PRs: versus human PRs, AI-generated PRs are 12% higher for one file, tied for 2–3 files, and 17% and 28% lower for 4–10 and 10+ files.
[6:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=388s) [Grouped bars compare human and AI-generated PR revert rates across four file-count ranges.](https://assets.ai.engineer/sites/aiecode2025/sha256/36e926db786d1cbedb430f18957758133a9068570fc7e852de25d764c3fb7a08.webp)
Revert rate by PR size
Y-axis: REVERTS PER 1,000 MERGED PRs
Y-axis ticks: 0, 1, 2, 3, 4
X-axis: FILES CHANGED
Legend: human; AI-generated
1 file: human 2.99; AI-generated 3.36; +12%
2–3 files: human 2.99; AI-generated 3.02; tied
4–10 files: human 2.88; AI-generated 2.38; −17%
10+ files: human 3.57; AI-generated 2.57; −28%
greptile
### Greptile comments
Greptile’s review comments provide a second signal of code quality. The examples contrast a P0 access-control issue with a P2 cosmetic date-formatting suggestion.
[6:49](https://www.youtube.com/watch?v=474j-n1Ltxc&t=409s) [Two illustrative code-review comments contrast a critical access-control issue with a cosmetic date-formatting suggestion.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac34c7d494650e0124b81da21b11358a2250b0fccbf7104bb42f5f998094ee5e.webp)
Method 2
Greptile comments
greptile
commented on auth/session.ts
P0 · CRITICAL
88 if (token === user.session_token) {
89 return grantAccess(user, "admin");
90 }
String comparison on session tokens is vulnerable to timing attacks, and any authenticated user is granted admin scope. This will leak privilege the moment it ships.
greptile
commented on utils/format.ts
P2 · NIT
14 function formatDate(d) {
15 return d.toString();
16 }
Consider using toLocaleDateString so the rendered date respects the user's locale. Cosmetic only.
greptile
### Greptile comments on AI-generated PRs
Three of four agents have fewer P0 comments than the human reference; two are below it for P1 and P2. Cursor BG is above the human reference in all three categories.
[7:10](https://www.youtube.com/watch?v=474j-n1Ltxc&t=430s) [Three bar charts compare P0, P1 and P2 comment values for four coding agents against human reference values.](https://assets.ai.engineer/sites/aiecode2025/sha256/72eded3ee8b218691b849ca69a8f759d718e7ac0aa097bb4a4bea74512adff90.webp)
Method 2
Greptile comments on AI-generated PRs
P0
Devin 0.038
Codex 0.041
Claude 0.078
Cursor BG 0.145
human · 0.099
P1
Devin 1.47
Codex 1.31
Claude 2.34
Cursor BG 2.83
human · 1.94
P2
Devin 2.64
Codex 2.81
Claude 3.75
Cursor BG 4.80
human · 2.88
greptile
### Review rounds
Greptile reviews a pull request, the agent addresses comments and pushes changes, and the cycle repeats until merge. The speaker tracks these rounds as a measure of iteration.
[7:34](https://www.youtube.com/watch?v=474j-n1Ltxc&t=454s) [A review round is defined as a push-and-re-review cycle, illustrated by a pull request timeline.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf84ec98f046c459daf60937c2372648b25cc2e09c182e0404a9b921680cef66.webp)
Method 3
Review rounds
A round is a push-and-re-review cycle.
Pull requests · #4832
merged
feat: add rate limit middleware
opened by coding-agent · 3 rounds before merge
coding-agent opened
greptile reviewed
coding-agent pushed
greptile reviewed
coding-agent pushed
merged
greptile
### Number of review rounds
Mean review cycles to merge ranged from 2.11 for Devin to 2.46 for Cursor BG and Codex, with humans at 2.21. Gupta describes the differences as small.
[8:02](https://www.youtube.com/watch?v=474j-n1Ltxc&t=482s) [Bar chart compares mean review cycles to merge, ranging from 2.11 for Devin to 2.46 for Cursor BG and Codex.](https://assets.ai.engineer/sites/aiecode2025/sha256/01b4976d68271dfde7bb218fbe68bdc91d7758468956726c13ef79f4e2517761.webp)
Method 3
Number of review rounds
Author:
Devin: 2.11
Claude: 2.19
human: 2.21
Cursor BG: 2.46
Codex: 2.46
Mean review cycles to merge
X-axis ticks: 2.0, 2.2, 2.4, 2.6
greptile
### Are agent-generated PRs bad in different ways?
Gupta investigates whether agents fail differently from humans by scanning several million Greptile comments from recent months for terms such as SQL injection and N+1 query.
[8:22](https://www.youtube.com/watch?v=474j-n1Ltxc&t=502s) [Slide asks whether agent-generated pull requests fail differently from human-written ones.](https://assets.ai.engineer/sites/aiecode2025/sha256/451933c1cb0f242c17ed9f807366473cc31a46c5afebeb6096f805b56a8e7a44.webp)
Are agent-generated PRs bad in different ways than human-written PRs?
Greptile
### Patterns of failure
Greptile’s review-comment term frequencies vary by agent. For N+1 queries, Cursor BG reaches 3.45× the human baseline, while Devin is at 0.45×.
[8:56](https://www.youtube.com/watch?v=474j-n1Ltxc&t=536s) [Heatmap compares failure categories across Claude, Codex, Devin and Cursor BG, with values relative to a human baseline.](https://assets.ai.engineer/sites/aiecode2025/sha256/236aaed40f72ce0a6d0e8ae43f299aa3f0e581f4ae2c73e304d11efa45a18fba.webp)
Patterns of failure
Claude | Codex | Devin | Cursor BG
sql injection: 1.50× | 1.25× | 0.70× | 1.70×
xss: 1.57× | 0.86× | 0.86× | 1.43×
auth bypass: 1.50× | 1.00× | 0.50× | 1.67×
IDOR / missing tenant check: 1.75× | 0.88× | 0.69× | 1.31×
secret in logs: 1.34× | 1.34× | 0.94× | 1.65×
n+1 query: 1.27× | 0.64× | 0.45× | 3.45×
regression: 1.25× | 1.34× | 0.89× | 2.37×
off-by-one: 1.64× | 0.55× | 0.64× | 2.27×
test missing: 0.96× | 1.13× | 0.93× | 2.37×
stale comment: 1.69× | 0.38× | 0.88× | 0.69×
greptile
### Autonomous agents are good, actually
Despite his initial skepticism, Gupta says the evidence suggests end-to-end coding agents can probably contribute meaningfully in enterprise coding environments.
[9:36](https://www.youtube.com/watch?v=474j-n1Ltxc&t=576s) [Slide states that autonomous agents are good.](https://assets.ai.engineer/sites/aiecode2025/sha256/67829bf51aadd2cb8b8f193a79e6dbf3e7c47975e394b54f53efa173f9cc6965.webp)
Autonomous agents are good, actually
greptile
### Code review needs to adapt for the agentic era
Gupta says the median Greptile user writes 50 pull requests a month, versus 500 at the 90th percentile, and asks how code validation can scale to match.
[9:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=594s) [Slide states that code review needs to adapt for the agentic era.](https://assets.ai.engineer/sites/aiecode2025/sha256/da3ef30e70e51a2bcacf0b9fcd0ee1e035e9398f762f0a7feb1c7f290d0af904.webp)
Code review needs
to adapt for the
agentic era
greptile
### Daksh Gupta
Greptile co-founder Daksh Gupta closes the talk and thanks the audience.
[12:07](https://www.youtube.com/watch?v=474j-n1Ltxc&t=727s) [Greptile closing slide with the speaker’s name and professional contact details.](https://assets.ai.engineer/sites/aiecode2025/sha256/60d15076492dc9e2702ed42a9fc8285e8a79011429447dd65de5d506efd604d5.webp)
Daksh Gupta
greptile
[redacted] · X [redacted]
25 / 25
### The AI Code Reviewer
Greptile invites attendees to visit its booth. Gupta also offers a free trial and asks for feedback.
[12:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=734s) [Greptile promotional slide with a QR code, colorful lines and a booth invitation.](https://assets.ai.engineer/sites/aiecode2025/sha256/202bfd3e25412ca4a72401e95c1e4c6bcaac189edaf8253263e7175bc9fe9760.webp)
greptile
The AI Code Reviewer.
Visit our booth or reach us at sales@greptile.com
### AI Engineer World's Fair
[12:24](https://www.youtube.com/watch?v=474j-n1Ltxc&t=744s) [AI Engineer World's Fair title in white on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/f51dffd91bee63d0cc7a6e86fc80bd6fc283ddf826917b010b45cdd42415a829.webp)
AI Engineer
World’s Fair
HTTPS://AI.ENGINEER
## Full transcript
[0:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=12s) How's everyone doing?
[0:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=14s) Awesome. Uh, my name is Du. I'm one of
[0:16](https://www.youtube.com/watch?v=474j-n1Ltxc&t=16s) the co-founders of a company called
[0:18](https://www.youtube.com/watch?v=474j-n1Ltxc&t=18s) Gravile. And at Grapile, we're working
[0:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=21s) on AI agents that validate poll requests
[0:25](https://www.youtube.com/watch?v=474j-n1Ltxc&t=25s) with full context of the codebase. The
[0:27](https://www.youtube.com/watch?v=474j-n1Ltxc&t=27s) way Graval does this is every time
[0:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=28s) there's a pull request, we have a swarm
[0:30](https://www.youtube.com/watch?v=474j-n1Ltxc&t=30s) of agents and they go and look at every
[0:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=32s) file that's changed, every file that's
[0:35](https://www.youtube.com/watch?v=474j-n1Ltxc&t=35s) related to the files that changed to
[0:36](https://www.youtube.com/watch?v=474j-n1Ltxc&t=36s) figure out if there's bugs. And it also
[0:38](https://www.youtube.com/watch?v=474j-n1Ltxc&t=38s) spins up your code in a sandbox,
[0:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=40s) installs the dependencies, spins up
[0:42](https://www.youtube.com/watch?v=474j-n1Ltxc&t=42s) local host, clicks around to try to
[0:44](https://www.youtube.com/watch?v=474j-n1Ltxc&t=44s) break things, mocks inputs, and
[0:47](https://www.youtube.com/watch?v=474j-n1Ltxc&t=47s) everything else to figure out if the
[0:48](https://www.youtube.com/watch?v=474j-n1Ltxc&t=48s) code is broken. But I'm going to talk
[0:50](https://www.youtube.com/watch?v=474j-n1Ltxc&t=50s) about something a little bit different,
[0:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=52s) which is fully autonomous coding agents.
[0:56](https://www.youtube.com/watch?v=474j-n1Ltxc&t=56s) So
[0:57](https://www.youtube.com/watch?v=474j-n1Ltxc&t=57s) I first moved to San Francisco three
[0:59](https://www.youtube.com/watch?v=474j-n1Ltxc&t=59s) years ago to work on AI coding. And this
[1:00](https://www.youtube.com/watch?v=474j-n1Ltxc&t=60s) is because GBT 3.5 which came out in
[1:03](https://www.youtube.com/watch?v=474j-n1Ltxc&t=63s) 2022 was the first model that seemed
[1:06](https://www.youtube.com/watch?v=474j-n1Ltxc&t=66s) like it was really good at programming
[1:08](https://www.youtube.com/watch?v=474j-n1Ltxc&t=68s) at the time. Code complete tab complete
[1:11](https://www.youtube.com/watch?v=474j-n1Ltxc&t=71s) was like the main paradigm for coding.
[1:13](https://www.youtube.com/watch?v=474j-n1Ltxc&t=73s) The most exciting thing that was
[1:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=74s) happening was you had tab complete from
[1:15](https://www.youtube.com/watch?v=474j-n1Ltxc&t=75s) cursor and you had co-pilots code
[1:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=77s) complete and that was like the way in
[1:19](https://www.youtube.com/watch?v=474j-n1Ltxc&t=79s) which AI coding works. In 2024, for the
[1:23](https://www.youtube.com/watch?v=474j-n1Ltxc&t=83s) first time, multifile editing started to
[1:26](https://www.youtube.com/watch?v=474j-n1Ltxc&t=86s) work. Cursor was the first to do this
[1:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=88s) and then a couple of other products
[1:29](https://www.youtube.com/watch?v=474j-n1Ltxc&t=89s) followed. But now, for the first time,
[1:31](https://www.youtube.com/watch?v=474j-n1Ltxc&t=91s) AI could simultaneously edit multiple
[1:33](https://www.youtube.com/watch?v=474j-n1Ltxc&t=93s) files at once. But things got really
[1:36](https://www.youtube.com/watch?v=474j-n1Ltxc&t=96s) interesting in 2025 because for the
[1:37](https://www.youtube.com/watch?v=474j-n1Ltxc&t=97s) first time, we had autonomous agents.
[1:39](https://www.youtube.com/watch?v=474j-n1Ltxc&t=99s) They could be given a task and they
[1:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=100s) could go and fulfill the task and create
[1:42](https://www.youtube.com/watch?v=474j-n1Ltxc&t=102s) entire pull requests all at once. And
[1:43](https://www.youtube.com/watch?v=474j-n1Ltxc&t=103s) this came to a precipice in December of
[1:45](https://www.youtube.com/watch?v=474j-n1Ltxc&t=105s) last year when coding agents became
[1:48](https://www.youtube.com/watch?v=474j-n1Ltxc&t=108s) literally completely autonomous. And
[1:50](https://www.youtube.com/watch?v=474j-n1Ltxc&t=110s) anyone that's working in AI coding knows
[1:51](https://www.youtube.com/watch?v=474j-n1Ltxc&t=111s) that December when the new models came
[1:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=112s) out was sort of this like watershed
[1:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=114s) moment in the history of AI coding
[1:56](https://www.youtube.com/watch?v=474j-n1Ltxc&t=116s) because these products for the first
[1:57](https://www.youtube.com/watch?v=474j-n1Ltxc&t=117s) time were like truly autonomous in how
[1:59](https://www.youtube.com/watch?v=474j-n1Ltxc&t=119s) they programmed. And all over Twitter
[2:02](https://www.youtube.com/watch?v=474j-n1Ltxc&t=122s) were all these crazy stories of people
[2:04](https://www.youtube.com/watch?v=474j-n1Ltxc&t=124s) that were having these agents spin off
[2:05](https://www.youtube.com/watch?v=474j-n1Ltxc&t=125s) and open a 100 pull requests a day. They
[2:08](https://www.youtube.com/watch?v=474j-n1Ltxc&t=128s) were experimenting with polyphasic sleep
[2:10](https://www.youtube.com/watch?v=474j-n1Ltxc&t=130s) so they could stay awake to re-trigger
[2:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=132s) their agents from time to time. And as
[2:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=134s) someone that started programming before
[2:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=137s) agents, I was very skeptical that real
[2:19](https://www.youtube.com/watch?v=474j-n1Ltxc&t=139s) companies could program this way. So I
[2:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=141s) was really really curious, are these
[2:23](https://www.youtube.com/watch?v=474j-n1Ltxc&t=143s) fully vi PRs really actually good? Is
[2:25](https://www.youtube.com/watch?v=474j-n1Ltxc&t=145s) this something that's sort of a Twitter
[2:27](https://www.youtube.com/watch?v=474j-n1Ltxc&t=147s) hype and you can have independent
[2:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=148s) developers and and maybe like small
[2:30](https://www.youtube.com/watch?v=474j-n1Ltxc&t=150s) startups with no real customers do this?
[2:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=152s) But anyone with real customers with an
[2:35](https://www.youtube.com/watch?v=474j-n1Ltxc&t=155s) actually commercially viable code base,
[2:38](https://www.youtube.com/watch?v=474j-n1Ltxc&t=158s) could they still use these end-to-end
[2:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=160s) coding agents?
[2:42](https://www.youtube.com/watch?v=474j-n1Ltxc&t=162s) And at Grubile, we're fortunate to work
[2:44](https://www.youtube.com/watch?v=474j-n1Ltxc&t=164s) with some very large companies. We work
[2:46](https://www.youtube.com/watch?v=474j-n1Ltxc&t=166s) with Nvidia, with Coinbase, with Scale,
[2:48](https://www.youtube.com/watch?v=474j-n1Ltxc&t=168s) with Data Dog, with American Express.
[2:49](https://www.youtube.com/watch?v=474j-n1Ltxc&t=169s) And we got really excited to see if
[2:53](https://www.youtube.com/watch?v=474j-n1Ltxc&t=173s) these coding agents were being used
[2:55](https://www.youtube.com/watch?v=474j-n1Ltxc&t=175s) really at these large companies and if
[2:56](https://www.youtube.com/watch?v=474j-n1Ltxc&t=176s) they were any good at doing real world
[2:59](https://www.youtube.com/watch?v=474j-n1Ltxc&t=179s) coding.
[3:02](https://www.youtube.com/watch?v=474j-n1Ltxc&t=182s) So, as an amateur data scientist, I
[3:04](https://www.youtube.com/watch?v=474j-n1Ltxc&t=184s) decided to go and dive into our data. We
[3:06](https://www.youtube.com/watch?v=474j-n1Ltxc&t=186s) review more than a million poll requests
[3:07](https://www.youtube.com/watch?v=474j-n1Ltxc&t=187s) a month and we tend to get a lot of
[3:10](https://www.youtube.com/watch?v=474j-n1Ltxc&t=190s) interesting data on what's good and bad
[3:11](https://www.youtube.com/watch?v=474j-n1Ltxc&t=191s) about these poll requests. We do this
[3:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=192s) for thousands and thousands of companies
[3:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=194s) and they're usually enterprises or at
[3:16](https://www.youtube.com/watch?v=474j-n1Ltxc&t=196s) least companies with serious products
[3:18](https://www.youtube.com/watch?v=474j-n1Ltxc&t=198s) with real customers. And so we figured
[3:20](https://www.youtube.com/watch?v=474j-n1Ltxc&t=200s) actually studying this data would yield
[3:22](https://www.youtube.com/watch?v=474j-n1Ltxc&t=202s) some really interesting results. Was a
[3:24](https://www.youtube.com/watch?v=474j-n1Ltxc&t=204s) surprisingly hard to figure out from all
[3:25](https://www.youtube.com/watch?v=474j-n1Ltxc&t=205s) the poll requests that GR was reviewing
[3:27](https://www.youtube.com/watch?v=474j-n1Ltxc&t=207s) which of the poll requests was actually
[3:29](https://www.youtube.com/watch?v=474j-n1Ltxc&t=209s) generated by AI. Surprisingly difficult
[3:31](https://www.youtube.com/watch?v=474j-n1Ltxc&t=211s) to do this. The first thing that I tried
[3:34](https://www.youtube.com/watch?v=474j-n1Ltxc&t=214s) was I started looking at the GitHub
[3:35](https://www.youtube.com/watch?v=474j-n1Ltxc&t=215s) author field. GitHub every commit has an
[3:39](https://www.youtube.com/watch?v=474j-n1Ltxc&t=219s) author field and so went through the
[3:41](https://www.youtube.com/watch?v=474j-n1Ltxc&t=221s) authors and turns out that less than 1%
[3:43](https://www.youtube.com/watch?v=474j-n1Ltxc&t=223s) of pull requests had codeex or clawed or
[3:45](https://www.youtube.com/watch?v=474j-n1Ltxc&t=225s) cursor as the author. But it wasn't
[3:48](https://www.youtube.com/watch?v=474j-n1Ltxc&t=228s) intuitive to me that only but 1% of all
[3:50](https://www.youtube.com/watch?v=474j-n1Ltxc&t=230s) code was fully AI generated. It just
[3:51](https://www.youtube.com/watch?v=474j-n1Ltxc&t=231s) seemed intuitive that that number would
[3:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=232s) be much higher. So I started to look for
[3:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=234s) other signals. Now thankfully Claude and
[3:58](https://www.youtube.com/watch?v=474j-n1Ltxc&t=238s) Codex and some other products leave a PR
[4:00](https://www.youtube.com/watch?v=474j-n1Ltxc&t=240s) description in the footer. In the PR
[4:02](https://www.youtube.com/watch?v=474j-n1Ltxc&t=242s) description, they put something along
[4:03](https://www.youtube.com/watch?v=474j-n1Ltxc&t=243s) the lines of co-authored by claude or
[4:04](https://www.youtube.com/watch?v=474j-n1Ltxc&t=244s) co-authored by cursor and so on. And
[4:06](https://www.youtube.com/watch?v=474j-n1Ltxc&t=246s) this gave me a little bit more signal, a
[4:08](https://www.youtube.com/watch?v=474j-n1Ltxc&t=248s) little bit more data on which PRs were
[4:09](https://www.youtube.com/watch?v=474j-n1Ltxc&t=249s) ostensibly largely, if not completely AI
[4:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=252s) generated. The third thing I looked at
[4:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=254s) was branch name prefixes. If you use
[4:16](https://www.youtube.com/watch?v=474j-n1Ltxc&t=256s) Codex, you know that Codex names as
[4:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=257s) branches. And it seemed reasonable that
[4:19](https://www.youtube.com/watch?v=474j-n1Ltxc&t=259s) if someone was using these coding agents
[4:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=261s) in such a sense that they were writing
[4:23](https://www.youtube.com/watch?v=474j-n1Ltxc&t=263s) the branch names or writing the PR
[4:24](https://www.youtube.com/watch?v=474j-n1Ltxc&t=264s) descriptions that the PRs were largely
[4:26](https://www.youtube.com/watch?v=474j-n1Ltxc&t=266s) AI generated. That seemed like a
[4:27](https://www.youtube.com/watch?v=474j-n1Ltxc&t=267s) reasonable assumption to me. So now I
[4:29](https://www.youtube.com/watch?v=474j-n1Ltxc&t=269s) had a good set of signals to indicate
[4:30](https://www.youtube.com/watch?v=474j-n1Ltxc&t=270s) whether a poll request was in fact
[4:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=272s) completely or largely AI generated or
[4:34](https://www.youtube.com/watch?v=474j-n1Ltxc&t=274s) not. And I came to the result that about
[4:37](https://www.youtube.com/watch?v=474j-n1Ltxc&t=277s) a quarter of all the poll requests that
[4:38](https://www.youtube.com/watch?v=474j-n1Ltxc&t=278s) Graal was reviewing in any given month
[4:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=280s) were completely or at least largely
[4:43](https://www.youtube.com/watch?v=474j-n1Ltxc&t=283s) generated by AI.
[4:46](https://www.youtube.com/watch?v=474j-n1Ltxc&t=286s) Then I backtracked this data to the last
[4:48](https://www.youtube.com/watch?v=474j-n1Ltxc&t=288s) 12 months and it turns out that this
[4:49](https://www.youtube.com/watch?v=474j-n1Ltxc&t=289s) number is growing really fast. In fact,
[4:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=292s) early last year, fewer than 1% of pull
[4:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=294s) requests had any evidence of being
[4:55](https://www.youtube.com/watch?v=474j-n1Ltxc&t=295s) completely AI generated. So this number
[4:57](https://www.youtube.com/watch?v=474j-n1Ltxc&t=297s) is going very very fast as model
[4:59](https://www.youtube.com/watch?v=474j-n1Ltxc&t=299s) performance is getting better. What's
[5:00](https://www.youtube.com/watch?v=474j-n1Ltxc&t=300s) also interesting is that you can't tell
[5:03](https://www.youtube.com/watch?v=474j-n1Ltxc&t=303s) when new models came out on this chart.
[5:05](https://www.youtube.com/watch?v=474j-n1Ltxc&t=305s) It seemed like the progress is very
[5:06](https://www.youtube.com/watch?v=474j-n1Ltxc&t=306s) continuous. These things are just
[5:08](https://www.youtube.com/watch?v=474j-n1Ltxc&t=308s) generally diffusing into the economy at
[5:10](https://www.youtube.com/watch?v=474j-n1Ltxc&t=310s) a generally high pace.
[5:13](https://www.youtube.com/watch?v=474j-n1Ltxc&t=313s) So then came the next question.
[5:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=314s) Everyone's vibe coding. Everyone's
[5:16](https://www.youtube.com/watch?v=474j-n1Ltxc&t=316s) producing these PRs that are
[5:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=317s) antiendentic. Not human plus AI but
[5:19](https://www.youtube.com/watch?v=474j-n1Ltxc&t=319s) literally AI. The question is are these
[5:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=321s) PRs any good? First question, what does
[5:23](https://www.youtube.com/watch?v=474j-n1Ltxc&t=323s) it mean for PR to be good? That seemed
[5:25](https://www.youtube.com/watch?v=474j-n1Ltxc&t=325s) like actually a very important question
[5:26](https://www.youtube.com/watch?v=474j-n1Ltxc&t=326s) to answer before we got into judging
[5:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=328s) these AI generated PRs. And so I took a
[5:31](https://www.youtube.com/watch?v=474j-n1Ltxc&t=331s) few methods to try this. The first one I
[5:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=332s) tried was revert rates. If a pull
[5:34](https://www.youtube.com/watch?v=474j-n1Ltxc&t=334s) request reverted, it was probably pretty
[5:36](https://www.youtube.com/watch?v=474j-n1Ltxc&t=336s) bad. And so that seemed like a pretty
[5:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=340s) sincere sort of guess at what a bad PR
[5:42](https://www.youtube.com/watch?v=474j-n1Ltxc&t=342s) could be. And so I started tracking,
[5:44](https://www.youtube.com/watch?v=474j-n1Ltxc&t=344s) okay, which PRs were in fact reverted.
[5:46](https://www.youtube.com/watch?v=474j-n1Ltxc&t=346s) GitHub conveniently names its branches
[5:48](https://www.youtube.com/watch?v=474j-n1Ltxc&t=348s) with a revert dash PR number and PR
[5:51](https://www.youtube.com/watch?v=474j-n1Ltxc&t=351s) name. And so I was able to track the
[5:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=352s) rate at which these things were being
[5:53](https://www.youtube.com/watch?v=474j-n1Ltxc&t=353s) reverted. And I found some interesting
[5:55](https://www.youtube.com/watch?v=474j-n1Ltxc&t=355s) data. Codex PR is reverted about one out
[5:58](https://www.youtube.com/watch?v=474j-n1Ltxc&t=358s) of every thousand poll requests. Devon
[6:00](https://www.youtube.com/watch?v=474j-n1Ltxc&t=360s) once every three and a half times every
[6:02](https://www.youtube.com/watch?v=474j-n1Ltxc&t=362s) thousand poll requests. Humans right in
[6:04](https://www.youtube.com/watch?v=474j-n1Ltxc&t=364s) the middle at about two and a half. So
[6:05](https://www.youtube.com/watch?v=474j-n1Ltxc&t=365s) there doesn't seem to be very big
[6:07](https://www.youtube.com/watch?v=474j-n1Ltxc&t=367s) difference between the rate at which
[6:08](https://www.youtube.com/watch?v=474j-n1Ltxc&t=368s) poll requests were reverted from people
[6:09](https://www.youtube.com/watch?v=474j-n1Ltxc&t=369s) versus agents in my study. Now I was
[6:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=372s) very skeptical of this and I figured
[6:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=374s) okay this is probably because humans are
[6:16](https://www.youtube.com/watch?v=474j-n1Ltxc&t=376s) making agents do the easier work the
[6:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=377s) simpler well scoped pull requests and
[6:20](https://www.youtube.com/watch?v=474j-n1Ltxc&t=380s) the more complex work was being done by
[6:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=381s) humans and of course complex work had a
[6:23](https://www.youtube.com/watch?v=474j-n1Ltxc&t=383s) higher propensity to be reverted because
[6:24](https://www.youtube.com/watch?v=474j-n1Ltxc&t=384s) the poll requests were more complicated
[6:26](https://www.youtube.com/watch?v=474j-n1Ltxc&t=386s) had larger surface areas risks and so
[6:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=388s) on. So I decided to measure the average
[6:30](https://www.youtube.com/watch?v=474j-n1Ltxc&t=390s) size of PR and whether the revert rates
[6:33](https://www.youtube.com/watch?v=474j-n1Ltxc&t=393s) were sort of aligned for both of those
[6:35](https://www.youtube.com/watch?v=474j-n1Ltxc&t=395s) and I found really interesting data.
[6:36](https://www.youtube.com/watch?v=474j-n1Ltxc&t=396s) Turns out there's actually very little
[6:38](https://www.youtube.com/watch?v=474j-n1Ltxc&t=398s) correlation, if any, between the size of
[6:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=400s) PRs of humans were getting reverted
[6:41](https://www.youtube.com/watch?v=474j-n1Ltxc&t=401s) versus size of PRs from agents that were
[6:43](https://www.youtube.com/watch?v=474j-n1Ltxc&t=403s) being reverted. So this does not seem
[6:45](https://www.youtube.com/watch?v=474j-n1Ltxc&t=405s) like actually very strong evidence that
[6:47](https://www.youtube.com/watch?v=474j-n1Ltxc&t=407s) the human PRs are any better than the
[6:49](https://www.youtube.com/watch?v=474j-n1Ltxc&t=409s) agent PRs.
[6:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=412s) The second set of signals I started
[6:53](https://www.youtube.com/watch?v=474j-n1Ltxc&t=413s) looking at was Graile comments. Now, now
[6:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=414s) Grapple is reviewing all these poll
[6:56](https://www.youtube.com/watch?v=474j-n1Ltxc&t=416s) requests and Graile finds P 0's, P1's,
[6:59](https://www.youtube.com/watch?v=474j-n1Ltxc&t=419s) P2s across all these changes and it
[7:00](https://www.youtube.com/watch?v=474j-n1Ltxc&t=420s) seemed reasonable that if Grapile was
[7:03](https://www.youtube.com/watch?v=474j-n1Ltxc&t=423s) finding more bugs in the code that the
[7:05](https://www.youtube.com/watch?v=474j-n1Ltxc&t=425s) code was most likely worse. So then I
[7:07](https://www.youtube.com/watch?v=474j-n1Ltxc&t=427s) started tracking the counts of P 0, P1's
[7:10](https://www.youtube.com/watch?v=474j-n1Ltxc&t=430s) and P2s in all these changes. And
[7:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=432s) interestingly, once again, there was not
[7:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=434s) that much of a difference. In fact,
[7:16](https://www.youtube.com/watch?v=474j-n1Ltxc&t=436s) three out of the four agents that we
[7:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=437s) tested performed better than humans in
[7:20](https://www.youtube.com/watch?v=474j-n1Ltxc&t=440s) terms of the rate at which they were
[7:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=441s) producing P 0. they were producing fewer
[7:23](https://www.youtube.com/watch?v=474j-n1Ltxc&t=443s) P 0 than humans were. Similarly the case
[7:26](https://www.youtube.com/watch?v=474j-n1Ltxc&t=446s) with P1's and P2s. Broadly speaking,
[7:29](https://www.youtube.com/watch?v=474j-n1Ltxc&t=449s) human generated PRs were about equal in
[7:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=452s) quality to agent generated PRs based on
[7:33](https://www.youtube.com/watch?v=474j-n1Ltxc&t=453s) this data.
[7:43](https://www.youtube.com/watch?v=474j-n1Ltxc&t=463s) have the agent go and look at those
[7:46](https://www.youtube.com/watch?v=474j-n1Ltxc&t=466s) comments, address them, and make a new
[7:47](https://www.youtube.com/watch?v=474j-n1Ltxc&t=467s) commit on the pull request branch. That
[7:49](https://www.youtube.com/watch?v=474j-n1Ltxc&t=469s) is the most common way to use guptile.
[7:51](https://www.youtube.com/watch?v=474j-n1Ltxc&t=471s) And so it seems reasonable that if the
[7:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=472s) pull requests were higher quality, it
[7:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=474s) would require fewer iterations before
[7:56](https://www.youtube.com/watch?v=474j-n1Ltxc&t=476s) they would be merged. And so I started
[7:57](https://www.youtube.com/watch?v=474j-n1Ltxc&t=477s) tracking the number of iterations, the
[7:59](https://www.youtube.com/watch?v=474j-n1Ltxc&t=479s) number of review rounds between when the
[8:01](https://www.youtube.com/watch?v=474j-n1Ltxc&t=481s) poll request was opened and when it was
[8:02](https://www.youtube.com/watch?v=474j-n1Ltxc&t=482s) merged.
[8:04](https://www.youtube.com/watch?v=474j-n1Ltxc&t=484s) Sure enough, very little difference.
[8:06](https://www.youtube.com/watch?v=474j-n1Ltxc&t=486s) Devon's PRs 2.1, Codex PR is 2.45, four,
[8:10](https://www.youtube.com/watch?v=474j-n1Ltxc&t=490s) five number of review cycles to merge
[8:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=492s) and humans right in the middle. Once
[8:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=494s) again, very little if any statistical
[8:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=497s) difference between human generated and
[8:18](https://www.youtube.com/watch?v=474j-n1Ltxc&t=498s) AI generated pull request in terms of
[8:20](https://www.youtube.com/watch?v=474j-n1Ltxc&t=500s) how many iterations before they were
[8:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=501s) ready to merge.
[8:24](https://www.youtube.com/watch?v=474j-n1Ltxc&t=504s) So now that there wasn't any strong
[8:26](https://www.youtube.com/watch?v=474j-n1Ltxc&t=506s) evidence that agent generated PRs were
[8:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=508s) worse than human generated PRs, I got
[8:30](https://www.youtube.com/watch?v=474j-n1Ltxc&t=510s) curious if there were qualitative
[8:31](https://www.youtube.com/watch?v=474j-n1Ltxc&t=511s) differences. Maybe the types of ways in
[8:33](https://www.youtube.com/watch?v=474j-n1Ltxc&t=513s) which agents failed were different from
[8:34](https://www.youtube.com/watch?v=474j-n1Ltxc&t=514s) the types of ways that humans failed.
[8:36](https://www.youtube.com/watch?v=474j-n1Ltxc&t=516s) And so then I looked at the corpus of
[8:38](https://www.youtube.com/watch?v=474j-n1Ltxc&t=518s) reptiles comments. It makes on average
[8:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=520s) about four comments per pull request. So
[8:42](https://www.youtube.com/watch?v=474j-n1Ltxc&t=522s) we had this corpus of several million
[8:44](https://www.youtube.com/watch?v=474j-n1Ltxc&t=524s) comments across the last several months.
[8:46](https://www.youtube.com/watch?v=474j-n1Ltxc&t=526s) And so I started scanning them for
[8:47](https://www.youtube.com/watch?v=474j-n1Ltxc&t=527s) specific phrases and words. For
[8:49](https://www.youtube.com/watch?v=474j-n1Ltxc&t=529s) instance, SQL injection or N plus1
[8:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=532s) query. And I plotted the frequency with
[8:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=534s) which these terms occur in graphile
[8:55](https://www.youtube.com/watch?v=474j-n1Ltxc&t=535s) comments for these various agents.
[8:58](https://www.youtube.com/watch?v=474j-n1Ltxc&t=538s) And so I made this chart of patterns of
[9:01](https://www.youtube.com/watch?v=474j-n1Ltxc&t=541s) failure.
[9:03](https://www.youtube.com/watch?v=474j-n1Ltxc&t=543s) To interpret this chart, you can assume
[9:05](https://www.youtube.com/watch?v=474j-n1Ltxc&t=545s) that 1x is the human propensity for
[9:07](https://www.youtube.com/watch?v=474j-n1Ltxc&t=547s) producing that type of error across the
[9:08](https://www.youtube.com/watch?v=474j-n1Ltxc&t=548s) entire chart. And you can kind of see
[9:10](https://www.youtube.com/watch?v=474j-n1Ltxc&t=550s) that there's actually quite a lot of
[9:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=552s) variation in the types of failures that
[9:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=554s) these agents seem to produce. For
[9:16](https://www.youtube.com/watch?v=474j-n1Ltxc&t=556s) instance, Claude is one and a half times
[9:18](https://www.youtube.com/watch?v=474j-n1Ltxc&t=558s) more likely to produce a SQL injection
[9:20](https://www.youtube.com/watch?v=474j-n1Ltxc&t=560s) error than humans. Devon is about half
[9:23](https://www.youtube.com/watch?v=474j-n1Ltxc&t=563s) as likely as humans to produce a off
[9:26](https://www.youtube.com/watch?v=474j-n1Ltxc&t=566s) bypass issue.
[9:29](https://www.youtube.com/watch?v=474j-n1Ltxc&t=569s) I found it very interesting that there
[9:30](https://www.youtube.com/watch?v=474j-n1Ltxc&t=570s) was this much variation in how these
[9:31](https://www.youtube.com/watch?v=474j-n1Ltxc&t=571s) agents were performing and how different
[9:34](https://www.youtube.com/watch?v=474j-n1Ltxc&t=574s) their failure modes were from humans.
[9:38](https://www.youtube.com/watch?v=474j-n1Ltxc&t=578s) So it turns out that in spite of my
[9:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=580s) initial skepticism around the enterprise
[9:43](https://www.youtube.com/watch?v=474j-n1Ltxc&t=583s) usability of endto-end coding agents,
[9:46](https://www.youtube.com/watch?v=474j-n1Ltxc&t=586s) the evidence seems to suggest that
[9:47](https://www.youtube.com/watch?v=474j-n1Ltxc&t=587s) they're here and they probably can
[9:49](https://www.youtube.com/watch?v=474j-n1Ltxc&t=589s) contribute in real meaningful ways to
[9:51](https://www.youtube.com/watch?v=474j-n1Ltxc&t=591s) enterprise coding environments. And so
[9:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=594s) we started to think a little bit more
[9:55](https://www.youtube.com/watch?v=474j-n1Ltxc&t=595s) about what code review would look like
[9:56](https://www.youtube.com/watch?v=474j-n1Ltxc&t=596s) in such a world. Here's an interesting
[9:59](https://www.youtube.com/watch?v=474j-n1Ltxc&t=599s) stat. Today, Graal is used by several
[10:01](https://www.youtube.com/watch?v=474j-n1Ltxc&t=601s) tens of thousands of engineers every
[10:03](https://www.youtube.com/watch?v=474j-n1Ltxc&t=603s) single week to review all of their code.
[10:05](https://www.youtube.com/watch?v=474j-n1Ltxc&t=605s) So, we also know generally speaking the
[10:08](https://www.youtube.com/watch?v=474j-n1Ltxc&t=608s) number of pull requests that each of
[10:09](https://www.youtube.com/watch?v=474j-n1Ltxc&t=609s) these people are writing. The median
[10:11](https://www.youtube.com/watch?v=474j-n1Ltxc&t=611s) grubile user writes 50 pull requests a
[10:14](https://www.youtube.com/watch?v=474j-n1Ltxc&t=614s) month. So, call it about two plus per
[10:16](https://www.youtube.com/watch?v=474j-n1Ltxc&t=616s) workday. The 90th percentile writes 500
[10:19](https://www.youtube.com/watch?v=474j-n1Ltxc&t=619s) poll requests per month. That is a
[10:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=621s) drastic difference between the median
[10:22](https://www.youtube.com/watch?v=474j-n1Ltxc&t=622s) and the P90. The P99 is in the thousands
[10:25](https://www.youtube.com/watch?v=474j-n1Ltxc&t=625s) of pull requests. That is very
[10:27](https://www.youtube.com/watch?v=474j-n1Ltxc&t=627s) interesting in its own sense because
[10:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=628s) that means that the people on the
[10:30](https://www.youtube.com/watch?v=474j-n1Ltxc&t=630s) margins are actually producing poll
[10:31](https://www.youtube.com/watch?v=474j-n1Ltxc&t=631s) requests at the rate at which they're
[10:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=632s) coming up with new ideas. And beyond
[10:35](https://www.youtube.com/watch?v=474j-n1Ltxc&t=635s) being interesting, it also makes you
[10:37](https://www.youtube.com/watch?v=474j-n1Ltxc&t=637s) wonder well there existing systems for
[10:38](https://www.youtube.com/watch?v=474j-n1Ltxc&t=638s) validating this code which is manual
[10:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=640s) code review of course testing maybe you
[10:43](https://www.youtube.com/watch?v=474j-n1Ltxc&t=643s) have a QA firm that you work with
[10:44](https://www.youtube.com/watch?v=474j-n1Ltxc&t=644s) naturally can scale to that same degree.
[10:47](https://www.youtube.com/watch?v=474j-n1Ltxc&t=647s) And at Grav, we decided to take sort of
[10:48](https://www.youtube.com/watch?v=474j-n1Ltxc&t=648s) a first principles view at what really
[10:50](https://www.youtube.com/watch?v=474j-n1Ltxc&t=650s) good validation could look like. Instead
[10:52](https://www.youtube.com/watch?v=474j-n1Ltxc&t=652s) of saying that we wanted to automate QA
[10:53](https://www.youtube.com/watch?v=474j-n1Ltxc&t=653s) or automate testing or automate code
[10:55](https://www.youtube.com/watch?v=474j-n1Ltxc&t=655s) review, we took a step back and said,
[10:57](https://www.youtube.com/watch?v=474j-n1Ltxc&t=657s) what would we need to do for anyone to
[10:59](https://www.youtube.com/watch?v=474j-n1Ltxc&t=659s) be able to merge hundreds of pull
[11:01](https://www.youtube.com/watch?v=474j-n1Ltxc&t=661s) requests a month in an enterprise
[11:02](https://www.youtube.com/watch?v=474j-n1Ltxc&t=662s) environment where it really matters that
[11:04](https://www.youtube.com/watch?v=474j-n1Ltxc&t=664s) the code is correct? What would need to
[11:06](https://www.youtube.com/watch?v=474j-n1Ltxc&t=666s) happen between when the code was
[11:07](https://www.youtube.com/watch?v=474j-n1Ltxc&t=667s) expressed into a pull request and when
[11:09](https://www.youtube.com/watch?v=474j-n1Ltxc&t=669s) it was merged and deployed safely? We
[11:11](https://www.youtube.com/watch?v=474j-n1Ltxc&t=671s) figured we only actually had to answer
[11:13](https://www.youtube.com/watch?v=474j-n1Ltxc&t=673s) three questions. The first one is does
[11:15](https://www.youtube.com/watch?v=474j-n1Ltxc&t=675s) this change violate the user contract?
[11:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=677s) The second one, does it increase the
[11:19](https://www.youtube.com/watch?v=474j-n1Ltxc&t=679s) propensity of a future violation of the
[11:21](https://www.youtube.com/watch?v=474j-n1Ltxc&t=681s) user contract, whatever the user
[11:22](https://www.youtube.com/watch?v=474j-n1Ltxc&t=682s) contract might be for that application?
[11:24](https://www.youtube.com/watch?v=474j-n1Ltxc&t=684s) And third, does it fulfill the intent
[11:26](https://www.youtube.com/watch?v=474j-n1Ltxc&t=686s) that the author described? Did the poll
[11:28](https://www.youtube.com/watch?v=474j-n1Ltxc&t=688s) request do the thing that the author
[11:29](https://www.youtube.com/watch?v=474j-n1Ltxc&t=689s) wanted to do? And so we started
[11:32](https://www.youtube.com/watch?v=474j-n1Ltxc&t=692s) approaching this problem from base
[11:34](https://www.youtube.com/watch?v=474j-n1Ltxc&t=694s) ground level and said, okay, agents can
[11:36](https://www.youtube.com/watch?v=474j-n1Ltxc&t=696s) probably figure out if something's going
[11:38](https://www.youtube.com/watch?v=474j-n1Ltxc&t=698s) to violate the user contract and detect
[11:40](https://www.youtube.com/watch?v=474j-n1Ltxc&t=700s) bugs. If you let it spin up the code in
[11:42](https://www.youtube.com/watch?v=474j-n1Ltxc&t=702s) a sandbox, have it install the
[11:44](https://www.youtube.com/watch?v=474j-n1Ltxc&t=704s) dependencies, mock the inputs, run the
[11:46](https://www.youtube.com/watch?v=474j-n1Ltxc&t=706s) browser agents, you can probably start
[11:47](https://www.youtube.com/watch?v=474j-n1Ltxc&t=707s) to discover most of the issues that
[11:50](https://www.youtube.com/watch?v=474j-n1Ltxc&t=710s) could occur, you get a pretty high
[11:51](https://www.youtube.com/watch?v=474j-n1Ltxc&t=711s) degree of confidence on merge. Today,
[11:54](https://www.youtube.com/watch?v=474j-n1Ltxc&t=714s) almost a fifth of all the poll request
[11:55](https://www.youtube.com/watch?v=474j-n1Ltxc&t=715s) or gravile reviews are merged without
[11:57](https://www.youtube.com/watch?v=474j-n1Ltxc&t=717s) any human review or without any human
[11:58](https://www.youtube.com/watch?v=474j-n1Ltxc&t=718s) testing, which I find very interesting
[12:00](https://www.youtube.com/watch?v=474j-n1Ltxc&t=720s) and that is a number that we care a lot
[12:01](https://www.youtube.com/watch?v=474j-n1Ltxc&t=721s) about and want to bring higher and
[12:03](https://www.youtube.com/watch?v=474j-n1Ltxc&t=723s) higher, of course, within the guard
[12:04](https://www.youtube.com/watch?v=474j-n1Ltxc&t=724s) rails of producing really high quality
[12:06](https://www.youtube.com/watch?v=474j-n1Ltxc&t=726s) code.
[12:10](https://www.youtube.com/watch?v=474j-n1Ltxc&t=730s) Thank you so much. My name is D, one of
[12:12](https://www.youtube.com/watch?v=474j-n1Ltxc&t=732s) the co-founders of Graptile. We have a
[12:13](https://www.youtube.com/watch?v=474j-n1Ltxc&t=733s) booth here which you should come to. And
[12:15](https://www.youtube.com/watch?v=474j-n1Ltxc&t=735s) if you're interested in trying Graptile,
[12:17](https://www.youtube.com/watch?v=474j-n1Ltxc&t=737s) you can find us at gretile.com and try
[12:19](https://www.youtube.com/watch?v=474j-n1Ltxc&t=739s) it today for free. And we'd love to hear
[12:22](https://www.youtube.com/watch?v=474j-n1Ltxc&t=742s) your feedback. Thank you so much.