# The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans

Corey Gallon

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/26RtyAm9y_Q-dark-arts-web-automation-teaching-agents-use) · [Video](https://www.youtube.com/watch?v=26RtyAm9y_Q)

## The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans

Corey Gallon builds browser automation around reusable code, CDP input, and a verification loop, reserving model calls for the steps that actually need judgment.

Familiarity with browser JavaScript, DOM events, and the distinction between model decisions and tool execution will help you follow the examples.

### An account warning before the first demo

[0:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=29s)

While preparing his browser-automation demonstrations, Corey Gallon received what he describes as an OpenAI account-ban warning for cyber abuse. The work behind the warning was a set of agents solving browser challenges without a human in the loop.

![Dark slide with the opening claim in white and “ban me” highlighted in red; the speaker appears at lower left.](https://ai.engineer/api/talk-essay-media/5464d717028d9ec0d31ad4ed3392c0b0830a09a78e16aa09ea01dedd1e5f989a)

The opening claim: “OpenAI threatened to ban me for preparing this talk.”

The practical ambition is more familiar than the opening suggests: book things, send emails, and fill in forms. An agent can understand those tasks and still fail when a website rejects its interactions. Gallon’s premise is that driving Chrome through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) gives the agent access to the browser’s own input machinery. He describes that as making an agent resemble a person with a mouse; whether an entire website can distinguish automation is a broader question than how a click enters Chrome.

The implementation has three parts: a programmable shell CLI, CDP access through his [Chrome Agent](https://github.com/captivus/chrome-agent) tool, and an interaction loop that escalates through increasingly human-like techniques. The CLI captures repeatable work; CDP supplies perception and control; the loop decides whether an action actually accomplished anything.

### Keep the model out of repeatable steps

[2:40](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=160s)

Gallon starts by choosing shell tools over a model-mediated MCP workflow. His first point is that task capability need not decide the choice: he cites roughly 83% success for both CLI and MCP in an [Arize evaluation](https://arize.com/blog/mcp-vs-cli-skills-for-agents-what-our-eval-found-and-which-you-should-use/). That evaluation used **Opus 4.6** through the Claude Agent SDK on 25 GitHub tasks, with five repetitions per arm and LLM-judged correctness. Its results concern that task set, rather than browser automation.

**A successful sequence should become a program.** Once the steps are known, a shell command can replay them without asking a model to choose every next operation. Gallon contrasts that with an interaction pattern in which the model sits between successive tool calls. Reuse then improves both latency and the amount of information that must pass through the model.

Gallon reports seven CLI turns in under one minute versus 71 round trips and eight minutes for MCP on the same task. In the underlying Arize article, the task is milestone average time-to-close analysis, and the 71-call result is the MCP arm’s worst run—not an average. Only three calls in that run were actual MCP calls; the rest were bash parsing. The useful lesson is about how the work was organized: repeated model decisions and intermediate-data handling can dominate the task.

Gallon also attributes an up-to-75× token-cost advantage to CLI use. The matching [Anthropic code-execution example](https://www.anthropic.com/engineering/code-execution-with-mcp) reduces token usage from 150,000 to 2,000 by combining selective tool loading with code execution using MCP. It supports the value of moving deterministic work into code, but not a general CLI-versus-MCP price ratio. MCP itself does not require a model round trip for every operation. That distinction preserves the architectural point: **batch predictable operations, and spend inference on decisions.**

### See, hear, and operate through CDP

[4:10](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=250s)

Chrome’s familiar F12 DevTools panel is an entry point to the same protocol an agent can use. Chrome Agent exposes CDP and helps agents write code that replays browser interactions. Gallon describes the protocol at the time of the talk as having 57 domains, with hundreds of methods and events, which he groups into eight explanatory buckets. The surface changes frequently; the important design choice is to use a small working subset rather than make the agent reason over the entire protocol.

That subset gives the agent several different ways to understand a page:

| Capability | Browser channels | What they provide |
| --- | --- | --- |
| See | DOM, accessibility tree, screenshot | Structure, semantics, pixels |
| Hear | Network traffic, console, logs | Requests, responses, diagnostics |
| Operate | Clicks, keystrokes, navigation | Changes to browser state |

These channels are complementary. The DOM can expose a control’s structure, the accessibility tree can explain its role, and a screenshot can show what is actually visible. Network and console observations add evidence that the page responded to an action.

![Slide titled “See. Hear. Operate.” with three columns listing browser structure, network and console information, and interaction controls.](https://ai.engineer/api/talk-essay-media/7aa56f40903cc1a674426c0af389d6117b5adab8f9b88b066ad769cbf39c9c9e)

CDP capabilities organized into “See,” “Hear,” and “Operate.”

### Verify each move before climbing the ladder

[5:39](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=339s)

The basic loop is deliberately small:

1. **Sense:** inspect the current page through one or more available channels.
2. **Act:** perform one interaction, such as clicking, typing, or selecting.
3. **Verify:** observe the result through a different channel.

A click returning successfully is not evidence that a purchase entered the cart or a message was sent. After clicking, inspect the screen or the resulting network activity. Then start the next iteration from the state that actually exists.

When the expected change does not appear, Gallon climbs his three-rung interaction ladder. The rule is to use the lowest rung that works:

| Rung | Interaction technique | Reason to use it |
| --- | --- | --- |
| One | In-page APIs or synthetic JavaScript clicks | The page accepts direct programmatic interaction |
| Two | Browser input through CDP’s `Input` domain | Synthetic events are rejected |
| Three | Vision and human-like interaction behavior | Locating or operating the control requires more |

Rung one is the inexpensive default. Rung two changes how input reaches the page. Rung three adds capabilities such as interpreting pixels, following a mouse path, pausing, or varying movement. Escalation follows failed verification, rather than starting every task with the most elaborate technique.

Exploration and execution are separate phases. First, work through the loop manually, climbing only until the task succeeds. Then preserve the working path as code, an agent skill, or both. Code records the repeatable interactions; a skill can retain the instructions needed to use them. The next run should benefit from the previous discovery.

### Outlook: generate the message, replay the interactions

[8:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=509s)

Before the demonstrations, Gallon returns to the lawyer from his opening. He says the examples come from real browser-agent experience, but everything shown runs on infrastructure he owns and accounts he operates. That scope matters for the increasingly adversarial browser challenges that follow.

The first task is ordinary office work: send a batch of personalized emails through Outlook’s web client. The compose window accepts rung-one interaction. A synthetic click opens it, code fills the fields, and another synthetic click sends the message. Once that sequence is captured, a program can loop over recipients from one command. Gallon uses twenty or two hundred emails as examples of reusing the same sequence.

The model still has a job: personalizing the content. It does not need to rediscover how to open the composer or fill the subject field for each recipient. **Content generation varies; browser mechanics repeat.** The frame pairs the recipient loop with an Outlook message list and reading pane, showing those two responsibilities together.

![Slide titled “Outlook — batch send,” with a recipient loop on the left and an Outlook message list and reading pane on the right.](https://ai.engineer/api/talk-essay-media/7408869d19afc3633b7b8ab79df36d1e2bb3074a0fa05edba327ae7051b8bf28)

Outlook batch-send pseudocode beside the running browser demonstration.

Why use the web interface instead of a mail API? In Gallon’s corporate scenario, app registration and administrator approval put API access beyond an employee’s reach, while an existing web login already lets that employee send mail. The [Microsoft Graph permissions reference](https://learn.microsoft.com/en-us/graph/permissions-reference) distinguishes application `Mail.Send`, which requires admin consent, from delegated `Mail.Send`, which does not inherently require it; tenant policy can still constrain access. The example is therefore about a particular access barrier, not a universal requirement of Office 365.

Gallon calls the resulting web interface a universal, or permissionless, API. Here, that means avoiding a separate API integration while using the access of an authenticated account. It does not grant permissions beyond that account’s existing access.

### Demazon: a successful click can change nothing

[10:51](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=651s)

The next demonstration moves to a mock retailer, Demazon. The synthetic JavaScript click that worked in Outlook now does nothing to the Add to Cart button. There is no error: the page quietly ignores the event because it is untrusted. This is precisely why the loop needs verification beyond the action’s return value.

The relevant browser property is [`Event.isTrusted`](https://dom.spec.whatwg.org/). It distinguishes browser-dispatched events from script-dispatched ones; `HTMLElement.click()` produces an untrusted click. It is not proof that a human caused the event. A small JavaScript example makes the distinction observable without involving a shopping site:

```javascript
const button = document.createElement('button');
button.textContent = 'Inspect click trust';

const output = document.createElement('pre');
document.body.append(button, output);

button.addEventListener('click', (event) => {
  output.textContent += `isTrusted=${event.isTrusted}\n`;
});

button.click();
```

The programmatic call records `isTrusted=false`. Clicking the displayed button yourself records a browser-dispatched event. The flag describes event provenance, not the absence of automation elsewhere in the session.

Gallon then moves to rung two and uses CDP’s `Input` domain. In the demonstration, the resulting trusted click is accepted and the item enters the cart. That establishes the difference for this page’s event check, rather than proving that all bot detection has disappeared. The lower panel illustrates what the page’s logs might look like; it is not a retailer’s internal log feed. The visible cursor is also an added visualization of CDP input, not Gallon’s physical mouse moving across the screen.

### Turnstile: visible does not mean selectable

[12:44](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=764s)

Rung three begins with [Cloudflare Turnstile](https://developers.cloudflare.com/turnstile/). In Gallon’s demonstrated setup, the apparently simple checkbox is difficult to reach through ordinary page scripting. He describes three isolation boundaries: a closed shadow root, a cross-origin iframe, and another shadow root. The obstacle is initially locating and reaching the control, rather than interpreting a complicated visual puzzle.

His approach changes from selecting the checkbox as an element to locating it on the rendered page. The demonstration uses the iframe’s screen location and the checkbox’s relative position, then lets Chrome route a trusted coordinate interaction. Gallon reports completion without a human in the loop. The next challenges add another requirement: the agent must interpret what it sees, not merely reach a visible control.

### MTCaptcha and Lemin: perception and movement

[14:26](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=866s)

[MTCaptcha](https://www.mtcaptcha.com/) presents noisy characters. The agent takes a screenshot, uses its own vision capability to read the text, and enters the answer through trusted keyboard input in the cross-origin widget, one character at a time. The demonstration ends with server verification and token issuance. Recognition, input delivery, and confirmation are distinct steps: seeing the answer alone does not finish the task.

The next example, Lemin, is a jigsaw puzzle: identify the missing piece’s destination and drag the piece into the gap. Unlike the preceding isolation problem, Gallon says this example has neither a shadow root nor a cross-origin iframe. The piece is accessible in the page. The difficulty lies in the drag.

Gallon explains that this class of challenge samples a trail of mouse positions, including variation in speed and jitter. The demonstrated movement eases in, follows a slight curve, deliberately overshoots, and settles back into place. Vision supplies the destination; movement supplies the interaction behavior. His “moves like Jagger” joke captures the distinction: solving the geometry and satisfying the movement check are different problems. Turnstile, MTCaptcha, and Lemin have now exercised three different capabilities—reaching an isolated control, reading an image, and operating a control with a trajectory.

### reCAPTCHA v2: split the driver from the visual judgment

[16:36](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=996s)

The last demonstration is [reCAPTCHA v2](https://developers.google.com/recaptcha/docs/verify): a checkbox followed by a blurry image grid asking for objects such as crosswalks or traffic lights. This combines the earlier capabilities into a system with two responsibilities, which Gallon calls the **solver** and the **operator**.

The solver is deterministic code. It handles the initial checkbox interaction, reaches the challenge iframe, captures the grid each round, and restarts when a round expires. Those operations do not require a model to choose each next click. The unresolved step is visual: deciding what the grid contains—for example, whether a tile shows a bus.

That decision belongs to the operator. The solver supplies the grid; the agent selects the matching tiles and returns its answer; the solver resumes the browser work. Between rounds, the agent waits.

| Component | Responsibility | Model involvement |
| --- | --- | --- |
| Solver | Browser driving, screenshots, round handling | None for deterministic steps |
| Operator | Interpret the grid and select tiles | Visual judgment per round |

The boundary is the important part. The model receives the decision that requires perception, while code retains control of the repetitive interaction sequence.

### Latency becomes a correctness constraint

[18:30](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1110s)

During playback, Gallon remarks that the agent spotted bicycles he had missed in some tiles. He reports that the completed solution is verified by the server. The selected frame shows an intermediate bicycle grid with a tile checked, beside the explanation of expiring rounds.

![A reCAPTCHA bicycle grid with one tile checked, beside text about expiring rounds and the cost of a model call on every step.](https://ai.engineer/api/talk-essay-media/59e164dbbdd8de63ef06f3ea208e143302ebe80abfb478b152defadb3eaacf7c)

A bicycle image challenge beside the explanation that each round expires.

Speed matters because, in Gallon’s account, each image round expires and one challenge can contain several consecutive rounds. Calling a model for every look and click consumes the available window before the interaction sequence finishes. His successful design instead uses deterministic browser code with one quick visual judgment per round. Google’s documented two-minute validity period applies to an issued response token, not to the duration of each image-selection round.

This is where the early CLI argument becomes consequential: fewer model round trips can determine whether the task finishes in time, not merely whether it feels responsive. The relevant architectural property is programmable execution between model decisions. Gallon describes his solution as repeatable and reliable, but supplies no trial counts or measured success rate for that claim.

### Turn exploration into a reusable tool

[19:51](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1191s)

The CAPTCHA demonstrations serve as stress tests for the engineering method. They expose different reasons a browser interaction can fail, then show why an off-the-shelf agent benefits from carefully chosen sensors, explicit verification, and reusable execution code. The durable result of exploration is a recorded solution: a programmable tool that uses the browser’s observations, checks whether actions landed, and escalates only when the simpler technique fails.

The opening account warning also gets a resolution. Gallon says OpenAI rescinded the threat after a conversation, leaving him with access to Codex. Chrome Agent, his daily tool for this work, is installable through the Python ecosystem. He also explicitly encourages building your own: the method does not depend on adopting his particular implementation.

## Resources

- [Chrome Agent](https://github.com/captivus/chrome-agent) — Python CLI for observing and controlling Chrome through CDP, with installation instructions and browser interaction examples.
- [The Dark Arts of Web Automation](https://gallon.me/the-dark-arts-of-web-automation.html) — The speaker’s written companion to the talk, covering the verification loop and three-rung interaction ladder.
- [Arize’s MCP versus CLI skills evaluation](https://arize.com/blog/mcp-vs-cli-skills-for-agents-what-our-eval-found-and-which-you-should-use/) — May 2026 GitHub-task evaluation comparing correctness, latency and cost across MCP, CLI skills and a shell baseline.
- [Code execution with MCP](https://www.anthropic.com/engineering/code-execution-with-mcp) — Explains selective tool loading, processing intermediate results in code and composing MCP operations without a model call for each step.
- [Chrome DevTools Protocol reference](https://chromedevtools.github.io/devtools-protocol/) — Official reference for Chrome’s browser inspection and control protocol.
- [DOM event trust semantics](https://dom.spec.whatwg.org/) — The normative definition of Event.isTrusted and the distinction between browser-dispatched and script-dispatched events.
- [reCAPTCHA server-side verification](https://developers.google.com/recaptcha/docs/verify) — Documents backend response validation, single-use tokens and their two-minute validity period.

## Slides

### A warning while preparing the talk

The speaker says OpenAI threatened to ban his account over work preparing the talk, citing “cyber abuse” with a web browser.

[0:28](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=28s) [Slide says OpenAI threatened to ban the speaker while preparing the talk.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a6a3bf2b4af6613cb4b5264d5c456606ecf2bdbee3f3d0c63a04b9bcf08e977.webp)

OpenAI threatened to ban me for preparing this talk.

…for "cyber abuse." with a web browser.

### The thesis

The speaker argues that CDP-driven clicks and keystrokes follow the same path in Chrome as a person’s. The slide likens the browser to a human, as far as Google, Cloudflare and others can tell in June 2026.

[1:43](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=103s) [The slide likens a CDP browser to a person with a mouse, alongside a Futurama image.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d33907487b5849765f6d58d3466d670ba7ab1996cfc7d673d4076fd21e34dd4.webp)

// THE THESIS

A CDP browser is just a meatbag with a mouse.

* as far as Google, Cloudflare et al. can tell — June 2026

FIG.1 — BENDER · FUTURAMA

### Three parts of the method

The speaker pairs a CLI with CDP through Chrome Agent, then uses a sense–act–verify loop that climbs the ladder only as far as needed.

[2:20](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=140s) [Three cards list CLI, CDP, and a loop on a ladder.](https://assets.ai.engineer/sites/aiecode2025/sha256/081aa7f74355b8041ab98d429c494fda78a96e55259626f75dda655820076596.webp)

// THE METHOD

To pass as human, you need three things.

make a CDP browser operate indistinguishably from a person

01
CLI, not MCP
drive the browser with code — not a wall of tool definitions

02
CDP
the whole browser — using chrome-agent

03
A loop on a ladder
sense · act · verify — climb only as high as forced

→ everything after this is one of these three, shown in the wild.

### CLI and MCP

The speaker favors reusable CLI sequences: both approaches reached roughly 83% task success, while one task took MCP 8 minutes and 71 round trips versus a CLI’s under 1 minute and 7 calls. Anthropic reported up to 75× lower token cost.

[2:37](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=157s) [The slide compares CLI and MCP on reuse, speed, and token cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/63673f03900469eb2782a4797378437919d36d68acf7595d22a5bf535046c871.webp)

// THING 1
CLI, not MCP.

Capability is a wash — CLI ≈ MCP, ~83% accuracy either way [3].
So the question isn't can it — it's what it costs you.

CLI WINS WHERE IT COUNTS →

REUSE [1]
solve once, run forever
code persists — MCP re-asks the model every single run

SPEED [3][2]
71 calls → 7
same task: 8 min & 71 round-trips vs <1 min & 7 calls

TOKEN COST [1]
75× cheaper
150K → 2K tokens · 98.7% less, executing code

[1] Anthropic · code execution with MCP
[2] Anthropic · advanced tool use
[3] Arize · MCP vs CLI eval

### CDP domains

Chrome DevTools Protocol spans 57 domains in eight buckets, from browser control and network traffic to page inspection and debugging. The speaker says a small subset is enough to interact with a browser like a human.

[4:08](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=248s) [A chart groups 57 CDP domains into eight categories.](https://assets.ai.engineer/sites/aiecode2025/sha256/1c030dd6478dcf1e466645fa1b46f51cb73e77cc8bcc514733898f65f5736d7b.webp)

// THING THE 2ND
CDP. The whole browser.
57 domains, grouped into 8 buckets.

Performance & diagnostics 10
Profiler
HeapProfiler
Memory
Tracing
Performance
PerformanceTimeline
Audits
Log
SystemInfo
CrashReportContext

Storage, workers & offline 8
Storage
CacheStorage
IndexedDB
DOMStorage
FileSystem
ServiceWorker
BackgroundService
Preload

Driving the browser 7
Input
Page
Target
Browser
Inspector
HeadlessExperimental
Tethering

Network & traffic 4
Network
Fetch
IO
Security

Web-feature surfaces 9
WebMCP
FedCm
Autofill
Media
WebAudio
Cast
PWA
Extensions
Ads

Read & inspect page state 7
DOM
Accessibility
CSS
DOMSnapshot
Overlay
Animation
LayerTree

JS execution & debugging 6
Runtime
Debugger
DOMDebugger
EventBreakpoints
Console
Schema

Emulation & virtual devices 6
Emulation
DeviceOrientation
DeviceAccess
BluetoothEmulation
SmartCardEmulation
WebAuthn

→ MCP hands you a curated few of these. CDP hands you all 57.
▪ the senses you actually use — next slide

### See. Hear. Operate.

An agent can see a page through its structure, accessibility tree or pixels; hear it through network traffic and logs; and operate it with clicks, keystrokes and navigation.

[5:03](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=303s) [Three columns map browser domains to seeing, hearing and operating a page.](https://assets.ai.engineer/sites/aiecode2025/sha256/543461bc6bf76bf1a7646dd4566b42343cbc614805107682af9b8e59d5e23b3c.webp)

// THE DIGITAL SENSES

See. Hear. Operate.

SEE
visual — what is on the page
DOM · structure
Accessibility · roles
CSS · styles
DOMSnapshot
Page* · screenshot

HEAR
auditory — what it reports
Network · fetches
Runtime* · console
Log
Audits · issues

OPERATE
tactile — act as a user
Input · trusted events
Page* · navigate
Runtime* · evaluate

* multi-sense domain

### Sense → Act → Verify

Sense the page, take one action, then verify its effect through another channel. Repeat until the task succeeds; if the loop won't close, climb the ladder of more human techniques.

[5:37](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=337s) [A loop moves from sensing to acting to verification, then repeats.](https://assets.ai.engineer/sites/aiecode2025/sha256/0dea773c5f45fa4105261b5b2e05bcaaa27b107ea0c7c153192d64a8fcb08246.webp)

// THE 3RD THING

Sense → Act → Verify.
Repeat until it closes.

SENSE
aim
DOM · a11y tree · a screenshot

ACT
do one thing
a click · a keystroke · a nav

VERIFY
confirm
sense again — a different channel

never trust the click's own "success."

LOOP + LADDER
When the loop won't close on a rung — that's the signal to climb.

### The Meatbag Ladder

Start with a programmatic or fake click. If it fails, try a trusted click; only climb to human-like behavior when the page requires it.

[6:43](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=403s) [Three rungs progress from synthetic actions to trusted input and human behavior.](https://assets.ai.engineer/sites/aiecode2025/sha256/11779eb01121acfa91f7b724839cad0d95d004e0d0f62f569b3db566cced85ae.webp)

// THE 3RD THING
The Meatbag Ladder.
Climb only as high as the page forces you.

↑ more human

RUNG 3
HUMAN — the narrow frontier
trusted input + behaviour: mouse path, dwell, jitter, vision
CAPTCHA

RUNG 2
REAL — the workhorse
a trusted click through the native pipeline · Input.dispatchMouseEvent
Demazon

RUNG 1
FAKE — the smart default
don't act human at all — be a program, or fake the click
Outlook

### Explore, then automate

First, climb the interaction ladder by hand until a sequence works. Then record it as code, an agent skill, or both; code is deterministic, while a skill can adapt to UI changes.

[8:03](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=483s) [Two phases show manually finding a working sequence and recording it as code or an agent skill.](https://assets.ai.engineer/sites/aiecode2025/sha256/1dccea3c45be29c038a29e2d21b1ad777ad30edf3c92ca600edf8591bca71915.webp)

// THE TWO PHASES
Explore, then automate.

PHASE 1 · EXPLORE
run the loop by hand
climb rungs until it works — you end holding a known-working solution: this exact sequence, on these rungs.

→

PHASE 2 · AUTOMATE
write it down
CODE — deterministic · fast · ~$0/run · no agent
AGENT SKILL — replays & adapts to UI changes, like a human
when it's extra tricksy → do both.

→ Explore once, automate forever. That's the whole method.

### A word of advice from Pearson Specter Litt

Beside a suited lawyer, the speaker explains that the demos draw on real experience and run on infrastructure and accounts he owns. He calls them the floor of what’s possible.

[8:33](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=513s) [A counsel-themed slide states that the demos use the presenter's own infrastructure and accounts.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8bbbd87526fc2b09289d346855e07377c61288e0972ba42df2c2eea75b32963.webp)

// A WORD FROM COUNSEL

A word of advice from Pearson Specter Litt.

✓ Everything you're about to see is real — from my own work in the wild.
✓ Every demo runs only on infrastructure & accounts I own.
▲ This is the floor of what's possible — not the ceiling.

FIG.2 — ADVICE OF COUNSEL

### Demazon — trusted click

Demazon rejects synthetic clicks as untrusted. The demo uses Chrome’s Input.dispatchMouseEvent to send a trusted click and add the item to the cart.

[10:49](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=649s) [Product page beside code and a log showing rejected synthetic clicks.](https://assets.ai.engineer/sites/aiecode2025/sha256/6bba8c8361e9d78f403406eec29d7b56babd19df8c458918c8ddc68012cc6b19.webp)

RUNG 2 · REAL
Demazon — trusted click
a JS click is a function call in a costume.

# [RUNG 2] real, trusted click · Input.dispatchMouseEvent
box = iframe_coord('#add-to-cart')  # SENSE

el.click()  # REJECTED
el.dispatchEvent(evt)  # REJECTED
# event.isTrusted = false

Input.dispatchMouseEvent(box)  # ✓ ACCEPTED
# ACT — trusted native pipeline

assert cart == 1  # VERIFY

What you see — the browser
KVM SWITCH
TESmart 2-Port KVM Switch — 4K60, Dual Monitor, USB
$269.95
In Stock
Ships from and sold by demazon.com
Add to Cart

What the server sees — the attempt log
#  time      step  trusted  decision
1  14:00:39  atc   false    rejected (synthetic)
2  14:00:46  atc   false    rejected (synthetic)

→ the gate is event.isTrusted. the server log is the X-ray.

Interface text:
rung2_add_to_cart.py
stage-stack.mp4
demazon.com
0
/log (x-ray)

### Demazon — added to cart

The demo rejects two synthetic clicks, then accepts a trusted mouse event and confirms the item is in the cart. The log shows both the rejected attempts and the accepted events.

[11:03](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=663s) [Cart confirmation beside a log showing accepted trusted events.](https://assets.ai.engineer/sites/aiecode2025/sha256/f1cb9e507931dfa4fbcb72aa11606dc0d7ed7f9ee6522723b62feea3423bc336.webp)

RUNG 2 · REAL
Demazon — trusted click
a JS click is a function call in a costume.

# [RUNG 2] real, trusted click · Input.dispatchMouseEvent
box = iframe_coord('#add-to-cart')  # SENSE

el.click()  # REJECTED
el.dispatchEvent(evt)  # REJECTED
# event.isTrusted = false

Input.dispatchMouseEvent(box)  # ✓ ACCEPTED
# ACT — trusted native pipeline

assert cart == 1  # VERIFY

→ the gate is event.isTrusted. the server log is the X-ray.

What you see — the browser
✓ Added to cart
KVM
TESmart 2-Port KVM Switch — 4K60, Dual Monitor, USB
In Stock
$269.95
Subtotal (1 item): $269.95
← Continue shopping

What the server sees — the attempt log
#  time      step    trusted  decision
1  14:00:39  atc     false    rejected (synthetic)
2  14:00:46  atc     false    rejected (synthetic)
3  14:00:51  atc     true     accepted (trusted)
4  14:00:54  commit  true     accepted (trusted)

Interface text:
rung2_add_to_cart.py
stage-stack.mp4
demazon.com
1
/log (x-ray)

### Demazon — protection prompt

Demazon rejects two synthetic clicks, then accepts a trusted mouse event and opens a protection-plan prompt. The log shows each attempt’s trust status and decision.

[11:21](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=681s) [Protection-plan prompt over the product page, with an accepted event in the log.](https://assets.ai.engineer/sites/aiecode2025/sha256/4d6065d9c3d9ca77ccf63d73e55c40575945c92e6efd0303fc12a2a2939dfe1c.webp)

RUNG 2 · REAL
Demazon — trusted click
a JS click is a function call in a costume.

# [RUNG 2] real, trusted click · Input.dispatchMouseEvent
box = iframe_coord('#add-to-cart')  # SENSE

el.click()  # REJECTED
el.dispatchEvent(evt)  # REJECTED
# event.isTrusted = false

Input.dispatchMouseEvent(box)  # ✓ ACCEPTED
# ACT — trusted native pipeline

assert cart == 1  # VERIFY

What you see — the browser
demazon.com
TESmart 2-Port [illegible]
Dual Monitor, US[illegible]
$269.95
In Stock
Ships from and sold by [illegible]
KVM SWITCH
Add to Cart

Add to your order
3-Year Protection Plan  $33.99
Add protection
No thanks

What the server sees — the attempt log
#  time      step  trusted  decision
1  14:00:39  atc   false    rejected (synthetic)
2  14:00:46  atc   false    rejected (synthetic)
3  14:00:51  atc   true     accepted (trusted)

→ the gate is event.isTrusted. the server log is the X-ray.

Interface text:
rung2_add_to_cart.py
stage-stack.mp4

### Cloudflare Turnstile

The demo locates the iframe, calculates the checkbox position, and sends a trusted click through three nested boundaries. The test widget reports success; its always-passing test secret does not prove a real token.

[12:50](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=770s) [Code for interacting with a nested checkbox beside an unchecked Turnstile test widget.](https://assets.ai.engineer/sites/aiecode2025/sha256/c79255466be7d2ff54edb4e496ffcb33416dd8f6fe5cb4e57132bb47eae754bc.webp)

RUNG 3 · HUMAN
Cloudflare Turnstile
a checkbox behind 3 nested boundaries

# [RUNG 3] reach the buried checkbox · click the glass
DOM.getDocument{ pierce } # SENSE
# find the Turnstile <iframe> node
DOM.getBoxModel(iframe) # SENSE
checkbox = (box.x + 30, box.y + h/2)

Input.dispatchMouseEvent(cb) # ACT trusted
# Chrome hit-tests down through all 3

await __cfSolved # ✓ token
read POST /verify/turnstile # VERIFY

Cloudflare Turnstile (test key — always interactive)
EMBEDDABLE WIDGET

Cloudflare’s force-interactive TEST key: the ‘verify you are human’ checkbox shows every time, so it’s a reliable on-camera target. The test secret always-passes, so this proves the interaction (the cross-origin-iframe trusted click), not a real token.

Verify you are human
CLOUDFLARE
Privacy · Help
For testing only. If seen, report to site owner
Check

→ a real click lands on the glass — Chrome does the rest.

Interface text:
solve.py
turnstile.mp4

[13:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=809s) [Code beside a Turnstile test widget showing success and a passed confirmation.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ba413e67c83b82005ea16ded73f23854a917ed44bef0e69fe991f8c6ebbe245.webp)

RUNG 3 · HUMAN
Cloudflare Turnstile
a checkbox behind 3 nested boundaries

# [RUNG 3] reach the buried checkbox · click the glass
DOM.getDocument{ pierce } # SENSE
# find the Turnstile <iframe> node
DOM.getBoxModel(iframe) # SENSE
checkbox = (box.x + 30, box.y + h/2)

Input.dispatchMouseEvent(cb) # ACT trusted
# Chrome hit-tests down through all 3

await __cfSolved # ✓ token
read POST /verify/turnstile # VERIFY

Cloudflare Turnstile (test key — always interactive)
EMBEDDABLE WIDGET

Cloudflare’s force-interactive TEST key: the ‘verify you are human’ checkbox shows every time, so it’s a reliable on-camera target. The test secret always-passes, so this proves the interaction (the cross-origin-iframe trusted click), not a real token.

Success!
CLOUDFLARE
Privacy · Help
For testing only. If seen, report to site owner
Check
✓ Captcha passed successfully!

→ a real click lands on the glass — Chrome does the rest.

Interface text:
solve.py
turnstile.mp4

### MTCaptcha

The agent screenshots the MTCaptcha challenge, reads its characters with vision, then types them one at a time using trusted keystrokes in the widget’s cross-origin iframe.

[14:25](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=865s) [Code beside an MTCaptcha image-text challenge and input field.](https://assets.ai.engineer/sites/aiecode2025/sha256/c5ff8b05bc9c4e390a68bd9d999ecca368f59732444d69181c775201a4bcfce1.webp)

RUNG 3 · HUMAN
MTCaptcha
read it with vision · type it trusted

# [RUNG 3] read with vision · type trusted keys
Target.setAutoAttach{ flatten }  # SENSE
# iframe → its own CDP session
screenshot crop → read chars  # vision

Input.dispatchMouseEvent(field)  # ACT
for ch in answer: dispatchKeyEvent  # ACT
# trusted keystrokes into the OOPIF

await verified == true  # VERIFY

MTCaptcha
VENDOR ACCOUNT
GDPR-focused checkbox + image/text challenge. Verified via MTCaptcha's checktoken API.

Privacy & Terms
[illegible]
Enter text from image
Powered by MTCaptcha™
Check

→ a real challenge, really read and typed.

Interface text:
solve.py
mtcaptcha.mp4

### The final boss of the Internet

After three other CAPTCHA gates, the speaker introduces reCAPTCHA v2: a checkbox followed by an image-selection grid.

[16:34](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=994s) [A reCAPTCHA v2 introduction with a photograph and an image-selection challenge.](https://assets.ai.engineer/sites/aiecode2025/sha256/68433815df5209994997f5114063e762ffb0321e4bab8d94d2f6fb3fc65b9f67.webp)

reCAPTCHA v2

The final boss
of The Internet.

Not just that it falls —
how it falls.

Select all images with
bicycles
Click verify once there are none left

VERIFY

### reCAPTCHA v2: Code + Skill FTW

The solver drives each reCAPTCHA round, screenshots the grid and recovers from expiry. The operator examines the image, selects matching tiles and returns the answer; the slide estimates 15–18 seconds per round.

[17:06](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1026s) [Two columns describe a code-driven solver and an operator that reviews challenge images.](https://assets.ai.engineer/sites/aiecode2025/sha256/208472595aeef2c8f5d36d4ab88b33b08d10d39ef31c8369a2f10e3438d9f69e.webp)

RUNG 3 · HUMAN
reCAPTCHA v2: Code + Skill FTW
① SOLVER → ② OPERATOR

① THE SOLVER
solve_async.py · deterministic · no agent

# [RUNG 3 · FINAL BOSS] code drives the loop · agent sees and thinks
Input.dispatchMouseEvent(checkbox)  # ACT trusted
Target.setAutoAttach{ flatten }  # bframe session

loop each round:  # on a CDP event, never a poll
  await __gridReady  # SENSE · tiles painted
  screenshot bframe @3× → grid.png
  log "AWAITING ANSWER (round N)"
  on __answer(tiles):
    Input.dispatchMouseEvent(each tile)  # ACT
  on __expired: re-click → fresh  # recover
until __tokenWritten → /verify  # VERIFY

# deterministic, fast, free — the driving

② THE OPERATOR
captcha-operator.md · agent skill · effort: low

Monitor( solve-events.log )  # event stream

on "AWAITING ANSWER":
  grid = Read(grid.png)  # one look
  tiles = pick matches  # no deliberation
  __answer(tiles)  # one answer

# persist every round
# until TOKEN ISSUED

# vision stays fast — ≈ 15–18 s / round

### The boss falls

The demo selects bicycle tiles and reports a server-verified solve. To keep up with expiring rounds, deterministic code drives the interaction while AI takes one quick look per round.

[18:30](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1110s) [A reCAPTCHA image-selection challenge beside text explaining the time limit for each round.](https://assets.ai.engineer/sites/aiecode2025/sha256/45bf7d4c37fa571b0c9d024981f87e0f7ade08f256f4a1c9390d352b92e4e1db.webp)

RUNG 3 · HUMAN
✓ SOLVED · SERVER-VERIFIED
The boss falls.
every round on a stopwatch

THE STOPWATCH
Each round expires. A solve is many rounds, back to back.

A model in the loop on every step burns the clock — the gate resets first.

This is why it had to be a CLI, on CDP.
remember — "hold onto speed."

Select all images with
bicycles
Click verify once there are none left
VERIFY

Interface text:
recaptcha-v2-row-1.mp4
A target site of live CAPTCHA widget[illegible]
reCAP[illegible]
The classic[illegible]

[18:36](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1116s) [A CAPTCHA demonstration beside an explanation of the time limit on successive rounds.](https://assets.ai.engineer/sites/aiecode2025/sha256/be78faa824072fdd38f081f9e5ff76e06fc26523700cd4633c21fc9966c2187d.webp)

RUNG 3 · HUMAN
✓ SOLVED · SERVER-VERIFIED
The boss falls.
every round on a stopwatch

THE STOPWATCH
Each round expires. A solve is many rounds, back to back.

A model in the loop on every step burns the clock — the gate resets first.

This is why it had to be a CLI, on CDP.
remember — "hold onto speed."

Select all images with
bicycles
Click verify once there are none left
VERIFY

Interface text:
recaptcha-v2-row-1.mp4
A target site of live CAPTCHA widget[illegible]
reCAP[illegible]
The classic[illegible]

### Not a fluke. Repeatable.

Four additional CAPTCHA demonstrations are labeled as server-verified passes. The speaker says the solution is repeatable and applies to other forms of CAPTCHA.

[19:43](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1183s) [Four CAPTCHA demonstration panels labeled as additional server-verified passes.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e2f16d985405eaf69f63216447ec57216d80490f4e9b3f785c46ffc3ab48020.webp)

Not a fluke. Repeatable.
4 MORE PASSES · ALL SERVER-VERIFIED

✓ pass · row-2
✓ pass · row-3
✓ pass · row-4
✓ pass · row-5

Interface text:
reCAP[illegible]
(check[illegible]
The clas[illegible]
rendere[illegible]
Check[illegible]

### Figure it out once. Do it forever.

Explore a page by hand, then write the solution as code. Use a programmable CLI to drive the browser through CDP, climbing the interaction ladder only as far as the page forces you.

[19:50](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1190s) [Three cards describe using a programmable CLI, CDP and a loop that climbs only as high as necessary.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5217ccf30f91ec2e1601af9e2cf68b8abcb0b740aee1a6c5500130e48003575.webp)

Figure it out once.
Do it forever.

// THE METHOD

01
CLI, not MCP
so you can program it
01

02
CDP
the whole browser — its digital senses
02

03
A loop on the ladder
climb only as high as forced
03

→ Explore by hand, then write the solution as code.

### A meatbag with a mouse

The speaker closes with Chrome Agent, the tool he uses daily for web automation, and an install command. His central idea: teach an agent CDP to follow the same path through the web as a person.

[20:45](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1245s) [Closing slide describing an agent that navigates the web through CDP, with an installation command and the speaker’s social handle.](https://assets.ai.engineer/sites/aiecode2025/sha256/98f8b49fd4c6e4bd88bf33be45276b71310e120bb13689e4bd0c4deee9b459b5.webp)

A meatbag with a mouse.

Teach an agent CDP and it moves through the web on the very same path you do. Everything today came from that one idea.

$ uv tool install chrome-agent

follow me on X
@coreygallon

~/chrome-agent $ happy hacking

## Full transcript

[0:00](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=0s) [upbeat music] All right. The dark arts of web automation.

[0:18](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=18s) That sounds ominous, right? [laughing] It sounds like something I'm gonna teach you that you're going to need a lawyer for. Uh, well, actually, we'll come back to the lawyer in a minute, but a little bit of background.

[0:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=29s) As I was preparing for this talk, OpenAI threatened to ban my account just for the work that I was doing in preparing for the talk. [dramatic music]

[0:42](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=42s) So I checked my inbox a few days ago and got this, and that's a real shocker, right? So what does one do to earn the ban hammer for cyber abuse with a web browser?

[0:59](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=59s) Well, this. I was doing this. So what you see here is every one of these is being solved by an AI agent with no human in the loop.

[1:12](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=72s) And in fifteen minutes or so, you'll understand exactly how this is possible.

[1:18](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=78s) I want my agents to be able to use the web exactly the way that I do. I want them to book the things, to send the emails, to fill in the forms so that I don't have to.

[1:30](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=90s) But the moment that something that isn't a person starts clicking into a page, web pages often fight back. So this talk is about winning that fight, and it starts with one slightly unconventional idea.

[1:45](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=105s) And here it is. This is the premise of the whole talk on one slide. A CDP browser is just like a meat bag with a mouse.

[1:56](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=116s) No joke. Well, at, at least as far as Google and Cloudflare and the rest can tell. No joke. If you have a browser dr-- an agent drive a browser using the Chrome DevTools Protocol,

[2:08](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=128s) your agent's clicks and keystrokes travel the exact same path inside Chrome that yours do,

[2:15](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=135s) and that's the big idea. The rest of the talk is about how to pull it off, and that comes down to three things.

[2:22](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=142s) To-- You need a, a CLI, not an MCP, the Chrome DevTools Protocol, which is where a tool that I wrote called Chrome Agent comes into play, and then a loop run on a ladder.

[2:37](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=157s) So let's start with the one that picks a fight.

[2:40](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=160s) The first thing is to give your agent a CLI and not an MCP server, and by CLI, I mean shell-based tools. Before you in the back knuckle up on this one, there are some specific reasons as to why this is necessary.

[2:56](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=176s) First of all, it's worth noting that capability is a wash. So in a recent study by the guys at Arize AI, both a CLI and an MCP, given the same task, achieve those tasks successfully roughly eighty-three percent of the time.

[3:10](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=190s) However, a CLI beats MCP in reuse, in speed, and in cost. So let's take reuse first. A CLI sequence can be programmed. You write it once, and you run it a thousand times without a model in the loop, whereas MCP hits the model on every single turn.

[3:32](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=212s) The CLI is faster for a similar reason, because there's not a model in the middle of every step. So in that same study that I mentioned before, MCP took seventy-one round trips and eight minutes for the same task that took a CLI only seven turns and in under one minute.

[3:50](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=230s) So hold on to speed because that actually comes back at the end. Lastly, token cost. Anthropic themselves reported that the CLI can be as much as seventy-five times cheaper in terms of token cost.

[4:06](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=246s) So what is it that we're actually running on the command line?

[4:10](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=250s) We drive the browser in the Chrome DevTools Protocol, and that's the second thing that your agent needs to appear human. You already know this protocol, even if you don't know its name.

[4:19](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=259s) That panel that pops up when you hit F12, that drives the browser using exactly this protocol. Your agents can speak it too. They can use the Chrome Agent tool.

[4:30](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=270s) And Chrome Agent also makes it really easy for your agents to write code to replay CDP interactions. The surface area of CDP is enormous, and it changes really frequently.

[4:42](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=282s) So as of now, it's fifty-seven domains, and within those, there are hundreds and hundreds of methods and events. So I've kind of bucketed all of the domains into eight buckets to make it a little easier to keep it in your head.

[4:55](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=295s) But no worries, you don't actually need all fifty-seven domains. In order to interact with a browser in the way that a human does, you need a small subset of these.

[5:04](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=304s) And the best way to think about that small subset is in terms of the digital senses that they give your agent. So you see the page. You may read its structure from the DOM.

[5:17](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=317s) You may read its semantics from the accessibility tree, or just take a screenshot if you need the pixels. You hear the page, so that may be network traffic to and from, uh, or co-- the console or logs.

[5:32](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=332s) And then you operate the page with clicks and with keystrokes and with navigation.

[5:39](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=339s) The third and final thing that you need for your agent to appear human is what I like to call a loop on a ladder. So here's the loop: sense, act, verify, and you repeat that until the page gives in.

[5:55](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=355s) So in sensing, we perceive the page through one or more channels, the DOM, the accessibility tree, a screenshot, as I just mentioned, and then you act. You do one thing.

[6:05](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=365s) You click something, you type something, you take a scr- you, uh, select something, and then you verify. And now this is sensing again, but it's sensing through a different channel than the interaction.

[6:17](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=377s) So, for example, if you've clicked something, don't ask the click if it was successful, check the network or check the screen. So you sense where you are, you make one move, you confirm that it landed, and then you iterate again.

[6:33](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=393s) And when the loop won't close, so you sense, act, verify, and the page still won't do what it is you're trying to do, that's the page fighting back. And so that's telling you to climb the ladder.

[6:45](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=405s) The meat bag ladder, that is. This is a ladder of techniques that are increasingly more human as you climb it. It has three rungs, and you climb only as high as the page forces you.

[6:57](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=417s) Or said differently, you, you climb to the lowest ladder that... uh, the lowest rung, sorry, that actually works. So on rung one, you don't act human at all. If you can just use the API that's exposed within the page, then, then issue a synthetic JavaScript click, then do that.

[7:15](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=435s) It's easy, it's free, it's instant, and it's the right default. You climb to rung two when faking it stops working, so when you need to use a real click, for example, using the CDP Input domain.

[7:31](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=451s) Now, this is agent input that the page cannot tell apart from your own.

[7:37](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=457s) You climb to rung three when you need human input and human behavior. So this is a real mouse path, maybe with a, a little dwell and some jitter, uh, or vision to actually see and interpret things.

[7:52](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=472s) So you start cheap, you climb run- one rung at a time, only when the page makes you. Climb the ladder, loop on each rung, and then write down the path that worked.

[8:04](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=484s) And that's how you arrive at fully automated AI agent-driven browsing. First, you explore. You run the loop by hand. You climb rungs until you... the thing actually works, and then you automate.

[8:18](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=498s) You write the solution down so you never have to figure it out again, and you write it down as code or as an agent skill, or very often, as both.

[8:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=509s) So let me show you how this all comes together but, first of all, with a word from my attorney. So I mentioned that the lawyer would be back, and everything you're about to see is, in fact, real agent browser use in the wild.

[8:43](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=523s) It's taken from real experience. However, upon the advice of counsel, everything that you're seeing is running only on infrastructure that I own and accounts that I operate. Right, now, so who's ready to see the agents get busy with the browser? [audience applauding]

[9:01](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=541s) All right. So here's a simple one, everyday use case we'll start with. Let's send a batch of personalized emails, each one different, from your Outlook web client. We've got some lovely pseudocode on the right side...

[9:16](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=556s) oh, sorry, left side. It's inverted. Left side of the, uh, the, the slide that makes it really simple for you to see how it all kind of works in code.

[9:25](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=565s) Outlook's compose box has nothing for us to defeat, so this is rung one of the meat bag ladder. You don't act human at all. A synthetic click opens the compose window.

[9:35](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=575s) We fill this in programmatically, and another synthetic click sends it.

[9:40](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=580s) And then you let that rip, right? So the reason this is going so smoothly is it's executing a program. So you capture the sequence once and then loop it, whether it's twenty emails or two hundred, and

[9:54](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=594s) from, from just that one command. So you solve it once and reuse it forever.

[10:00](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=600s) The agent is riffing on content to personalize it here as we go, but it's running a program to drive all of the interactions. Now, you may ask, "Why would we drive the web UI at all?"

[10:13](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=613s) Like, why not just use the API? And in corporate environments, the API for an Office 365 tenant requires an app registration, and it also requires admin approval, which, as an employee, you can't often get.

[10:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=629s) And so in this pattern... Well, sorry, whereas the, the web login you have is all you need to be able to do exactly what you see on the screen here now.

[10:36](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=636s) And so in this pattern, the web UI itself kind of becomes a universal API, right? Like a permissionless API. Which is really neat. So that's rung one, but what happens when the page starts fighting back?

[10:51](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=651s) So let's say that you're browsing on your favorite online mega store, and we'll just call them Demazon. [audience laughing]

[10:59](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=659s) So they're a crafty bunch over there at Demazon, and they have no love for your bots at all. Now, if you take that same JavaScript click that just worked in Outlook and you point it at, say, the Add to Cart button, you get nothing.

[11:15](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=675s) Literally nothing. There's... There's no failure, there's no error, just nothing happens. And the page is ignoring it. And the reason for that is the page is checking, "Was this click from a human source?"

[11:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=689s) Chrome stamps every single event with just that answer, whether it's trusted or untrusted. So the JavaScript click that we fired previously is stamped untrusted, and in this case, the page just quietly drops that input.

[11:45](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=705s) But that's no worries. We'll just climb the meat bag ladder. So we move to rung two, and we click using Chrome's Input domain, and that uses the exact same input path that your actual mouse uses, and now it's stamped trusted, and the page can't tell the difference between your mouse and our agent, and bam, the item's dropped

[12:06](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=726s) straight into the cart. So we've got quite the inside view of demazon.com here. So if you look at this lower panel, you can imagine that this is what the page's logs look like.

[12:17](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=737s) Every one of the untrusted clicks fails, but the trusted ones go through, no worries. Now, as a heads-up going forward, when you see this mouse cursor moving here, that is added programmatically just so that you can visually see the mouse inputs that the agent is giving to the browser using Chrome Agent and CDP.

[12:38](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=758s) It's not actually moving my mouse here. All right.

[12:44](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=764s) Rung two is where the real meatbag inputs begin, but that's not enough to replace you in the browser. So we climb to the top of the meatbag ladder, rung three, and this is the narrow frontier where pages are actively hunting for bots.

[12:59](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=779s) There are a variety of techniques that we're gonna deploy here, though, so let's talk through a few of them. I'm sure this guy looks familiar, right? This is Cloudflare Turnstile.

[13:10](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=790s) It looks deceptively simple, but it is the hardest target that we've hit yet, [lip smacks]

[13:16](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=796s) and it's because that little checkbox can't be easily reached through typical web automation programming. Because these cheeky guys have hidden this thing through three isolated boundaries. First, it's encapsulated beneath a closed shadow root, and then the whole widget itself lives in a cross-origin iframe, which in turn also has another shadow root. [lip smacks]

[13:41](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=821s) So to every cheap trick, that, that checkbox is unclickable. There's no element to grab. So what do we do? Well, we just stop trying to grab it, right? [chuckles] We ask the browser where it is that the iframe sits on the screen.

[13:57](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=837s) We do a little bit of math to figure out where the checkbox is, then we fire a trusted click right at that position on the glass, and then Chrome does the rest for us, right?

[14:08](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=848s) A real click lands right in the checkbox, and we're off and running. There's no human in the loop. This is all agent. So that's level one cleared, and the trick here was really just kind of figuring out how to interact with it.

[14:21](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=861s) But all the next levels now make you prove that you can actually see.

[14:26](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=866s) So this is MTCaptcha. You remember this guy? And th-these guys are... And like it, they're, they're still around. Um, your agent's actually gotta read this guy. And so what the agent does is simulate what you would do.

[14:39](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=879s) It takes a screenshot of the challenge, looks at it, and then uses its own vision capability to pick the characters out of the noise. Then it types the answer back using real trusted keystrokes routed into the widget's cross-origin iframe

[14:58](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=898s) one character at a time, and these are the same keyboard inputs that you would send. And then the server agrees. The text is verified, and the token is issued.

[15:08](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=908s) Now, there's one more level before the final boss, and this one is won or lost based on how it is that you move like a meatbag.

[15:17](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=917s) So this is by Lemin. Uh, it's a little jigsaw puzzle where you spot where the piece belongs and then you drag it in to fill the gap, and there's an entire class of CAPTCHAs just like this.

[15:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=929s) Now, this one's tricksy in different ways. There's no shadow root. There's no cross-origin iframe. The piece is sitting right there in the page. The hard part here is the drag itself.

[15:40](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=940s) So when you drop the puzzle piece, these types of CAPTCHAs sample the mouse movement into a trail of points the whole way, and so that includes jitter and changing speeds and all of that.

[15:53](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=953s) So it's not just solving the puzzle, but it's solving it with moves like Jagger. So the agent drags the exact same way that a hand would drag, right? If you kind of watch him move, it eases in gently.

[16:06](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=966s) There's, like, a slight curve. It actually deliberately overshoots the puzzle piece and then eases it right back in, [sniffs]

[16:13](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=973s) just like a meatbag with a mouse. And it's using vision to identify the gap and then human-like motion to cross it. So that's Turnstile, that's MTCaptcha, and that's Lemin.

[16:29](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=989s) Three gates built to keep agents out, and we've just beat each one of them, which leaves only one boss standing.

[16:36](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=996s) And here he is, the final boss of the internet,

[16:41](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1001s) reCAPTCHA v2. It's that little checkbox and then the blurry grid with, like, fuzzy pictures of crosswalks or traffic lights in it. We've all squinted at these ones. But we've got the whole kit now on how to beat this guy, too.

[16:56](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1016s) The digital senses, the loop to deploy them in, the meatbag ladder, everything that you need to take down this cheeky bastard, so let's go. This is the whole machine, and it comes in two halves.

[17:13](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1033s) On one side, we have the solver. This is pure code, no agent, no model. It does everything programmatically. It does the trusted click in the checkbox. It pierces into the, uh, challenge iframe, and in every round, it screenshots the grid, and if for some reason a round expires, it just re-arms itself and goes again.

[17:33](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1053s) All of that, that bit there, is deterministic, it's fast, and it's free. [clears throat]

[17:38](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1058s) But there's one step in the loop that the code can't do, and that's look at that grid of fuzzy tiles and figure out what it is that's in it.

[17:46](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1066s) Is this a bus, for example? That's vision and thinking, and that needs eyes and a brain. And so that's the only thing that we give the agent as a job.

[17:56](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1076s) We call that bit the operator. The solver taps the agent on the shoulder, and the agent takes one look at the grid, picks the tiles with whatever the thing is that we're looking for in them, hands that answer back to the solver, and then just hangs out waiting until the next lap.

[18:13](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1093s) And this is really the entire talk running as one system. Code does the deterministic driving, and the agent does the only bits that require eyes and a brain. All right, who wants to see it go?

[18:30](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1110s) There it is. So while this is playing, by the way, kind of have a look. Like, some of these are really hard. [laughs] Like, the, the agent was able to spot bicycles in some of these tiles that I didn't see.

[18:44](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1124s) But ultimately, it's solved, it's verified by the server, and it's fast. And fast is the whole game here because this big, bad boss is on a clock. Every round expires, and one challenge can be multiple rounds back to back.

[19:03](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1143s) An agent that round-trips a model on every click and on every look burns that clock and loses. The challenge expires well before it ever finishes. The only thing that I've found that defeats this whole mess is exactly what you're looking at here, deterministic code running at machine speed with one quick AI look per round.

[19:25](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1165s) So remember when I told you to hold onto speed? This is exactly why, and this is also why this had to be a CLI running the CDP and not a model sitting in the middle of every single interaction.

[19:38](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1178s) By the way, in case you're wondering, this wasn't a fluke. It's a repeatable, reliable solution now for solving this and, and other forms of CAPTCHAs. But here's what I want you to really walk out of here with.

[19:51](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1191s) The big takeaway is the methodology that enabled this. The CAPTCHAs themselves were just tricksy little tests that demonstrate the methodology. This came down to careful, disciplined engineering, and the engineering enabled the agent to do something that it could not do at all off the shelf.

[20:09](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1209s) And the method is simple. Give your agent a CLI so that you can program it, drive the whole browser through CDP using its digital sensors, run it as a loop on the meat bag ladder, and climb only as high as the page forces you to, and then explore until you solve it and write the solution down.

[20:30](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1230s) That's what makes this durable and useful. You figure it out once, and you do it forever, which brings us all the way back to the good folks at OpenAI.

[20:40](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1240s) After a quiet word, they kindly rescinded the threat, so I've still got access to Codex, which is nice.

[20:48](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1248s) So you too can use Chrome Agent. It's installable in the Python ecosystem. That's the tool that I wrote that I do all this with, and I use it all day, every day.

[20:57](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1257s) Or frankly, build your own, right? Like, I'm not here shilling a product. Uh, we live in the age of unbounded personalized software. Please, however, do follow me on X.

[21:06](https://www.youtube.com/watch?v=26RtyAm9y_Q&t=1266s) I'd love to chat to you and learn how it is that you're automating the web with AI. And if you'd like to chat more about it, I'll pop out there into the, uh, huddle space, and we can have a chat now, but happy hacking. [audience applauding] [outro music]
