The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans

While preparing his browser-automation demonstrations, Corey Gallon received what he describes as an OpenAI account-ban warning for cyber abuse. The work behind the warning was a set of agents solving browser challenges without a human in the loop.

The practical ambition is more familiar than the opening suggests: book things, send emails, and fill in forms. An agent can understand those tasks and still fail when a website rejects its interactions. Gallon’s premise is that driving Chrome through the Chrome DevTools Protocol gives the agent access to the browser’s own input machinery. He describes that as making an agent resemble a person with a mouse; whether an entire website can distinguish automation is a broader question than how a click enters Chrome.

The implementation has three parts: a programmable shell CLI, CDP access through his Chrome Agent tool, and an interaction loop that escalates through increasingly human-like techniques. The CLI captures repeatable work; CDP supplies perception and control; the loop decides whether an action actually accomplished anything.

Gallon starts by choosing shell tools over a model-mediated MCP workflow. His first point is that task capability need not decide the choice: he cites roughly 83% success for both CLI and MCP in an Arize evaluation. That evaluation used Opus 4.6 through the Claude Agent SDK on 25 GitHub tasks, with five repetitions per arm and LLM-judged correctness. Its results concern that task set, rather than browser automation.

A successful sequence should become a program. Once the steps are known, a shell command can replay them without asking a model to choose every next operation. Gallon contrasts that with an interaction pattern in which the model sits between successive tool calls. Reuse then improves both latency and the amount of information that must pass through the model.

Gallon reports seven CLI turns in under one minute versus 71 round trips and eight minutes for MCP on the same task. In the underlying Arize article, the task is milestone average time-to-close analysis, and the 71-call result is the MCP arm’s worst run—not an average. Only three calls in that run were actual MCP calls; the rest were bash parsing. The useful lesson is about how the work was organized: repeated model decisions and intermediate-data handling can dominate the task.

Gallon also attributes an up-to-75× token-cost advantage to CLI use. The matching Anthropic code-execution example reduces token usage from 150,000 to 2,000 by combining selective tool loading with code execution using MCP. It supports the value of moving deterministic work into code, but not a general CLI-versus-MCP price ratio. MCP itself does not require a model round trip for every operation. That distinction preserves the architectural point: batch predictable operations, and spend inference on decisions.

Chrome’s familiar F12 DevTools panel is an entry point to the same protocol an agent can use. Chrome Agent exposes CDP and helps agents write code that replays browser interactions. Gallon describes the protocol at the time of the talk as having 57 domains, with hundreds of methods and events, which he groups into eight explanatory buckets. The surface changes frequently; the important design choice is to use a small working subset rather than make the agent reason over the entire protocol.

That subset gives the agent several different ways to understand a page:

These channels are complementary. The DOM can expose a control’s structure, the accessibility tree can explain its role, and a screenshot can show what is actually visible. Network and console observations add evidence that the page responded to an action.

The basic loop is deliberately small:

1. Sense: inspect the current page through one or more available channels.
2. Act: perform one interaction, such as clicking, typing, or selecting.
3. Verify: observe the result through a different channel.

A click returning successfully is not evidence that a purchase entered the cart or a message was sent. After clicking, inspect the screen or the resulting network activity. Then start the next iteration from the state that actually exists.

When the expected change does not appear, Gallon climbs his three-rung interaction ladder. The rule is to use the lowest rung that works:

Rung one is the inexpensive default. Rung two changes how input reaches the page. Rung three adds capabilities such as interpreting pixels, following a mouse path, pausing, or varying movement. Escalation follows failed verification, rather than starting every task with the most elaborate technique.

Exploration and execution are separate phases. First, work through the loop manually, climbing only until the task succeeds. Then preserve the working path as code, an agent skill, or both. Code records the repeatable interactions; a skill can retain the instructions needed to use them. The next run should benefit from the previous discovery.

Before the demonstrations, Gallon returns to the lawyer from his opening. He says the examples come from real browser-agent experience, but everything shown runs on infrastructure he owns and accounts he operates. That scope matters for the increasingly adversarial browser challenges that follow.

The first task is ordinary office work: send a batch of personalized emails through Outlook’s web client. The compose window accepts rung-one interaction. A synthetic click opens it, code fills the fields, and another synthetic click sends the message. Once that sequence is captured, a program can loop over recipients from one command. Gallon uses twenty or two hundred emails as examples of reusing the same sequence.

The model still has a job: personalizing the content. It does not need to rediscover how to open the composer or fill the subject field for each recipient. Content generation varies; browser mechanics repeat. The frame pairs the recipient loop with an Outlook message list and reading pane, showing those two responsibilities together.

Why use the web interface instead of a mail API? In Gallon’s corporate scenario, app registration and administrator approval put API access beyond an employee’s reach, while an existing web login already lets that employee send mail. The Microsoft Graph permissions reference distinguishes application Mail.Send, which requires admin consent, from delegated Mail.Send, which does not inherently require it; tenant policy can still constrain access. The example is therefore about a particular access barrier, not a universal requirement of Office 365.

Gallon calls the resulting web interface a universal, or permissionless, API. Here, that means avoiding a separate API integration while using the access of an authenticated account. It does not grant permissions beyond that account’s existing access.

The next demonstration moves to a mock retailer, Demazon. The synthetic JavaScript click that worked in Outlook now does nothing to the Add to Cart button. There is no error: the page quietly ignores the event because it is untrusted. This is precisely why the loop needs verification beyond the action’s return value.

The relevant browser property is Event.isTrusted. It distinguishes browser-dispatched events from script-dispatched ones; HTMLElement.click() produces an untrusted click. It is not proof that a human caused the event. A small JavaScript example makes the distinction observable without involving a shopping site:

const button = document.createElement('button');
button.textContent = 'Inspect click trust';

const output = document.createElement('pre');
document.body.append(button, output);

button.addEventListener('click', (event) => {
  output.textContent += `isTrusted=${event.isTrusted}\n`;
});

button.click();

The programmatic call records isTrusted=false. Clicking the displayed button yourself records a browser-dispatched event. The flag describes event provenance, not the absence of automation elsewhere in the session.

Gallon then moves to rung two and uses CDP’s Input domain. In the demonstration, the resulting trusted click is accepted and the item enters the cart. That establishes the difference for this page’s event check, rather than proving that all bot detection has disappeared. The lower panel illustrates what the page’s logs might look like; it is not a retailer’s internal log feed. The visible cursor is also an added visualization of CDP input, not Gallon’s physical mouse moving across the screen.

Rung three begins with Cloudflare Turnstile. In Gallon’s demonstrated setup, the apparently simple checkbox is difficult to reach through ordinary page scripting. He describes three isolation boundaries: a closed shadow root, a cross-origin iframe, and another shadow root. The obstacle is initially locating and reaching the control, rather than interpreting a complicated visual puzzle.

His approach changes from selecting the checkbox as an element to locating it on the rendered page. The demonstration uses the iframe’s screen location and the checkbox’s relative position, then lets Chrome route a trusted coordinate interaction. Gallon reports completion without a human in the loop. The next challenges add another requirement: the agent must interpret what it sees, not merely reach a visible control.

MTCaptcha presents noisy characters. The agent takes a screenshot, uses its own vision capability to read the text, and enters the answer through trusted keyboard input in the cross-origin widget, one character at a time. The demonstration ends with server verification and token issuance. Recognition, input delivery, and confirmation are distinct steps: seeing the answer alone does not finish the task.

The next example, Lemin, is a jigsaw puzzle: identify the missing piece’s destination and drag the piece into the gap. Unlike the preceding isolation problem, Gallon says this example has neither a shadow root nor a cross-origin iframe. The piece is accessible in the page. The difficulty lies in the drag.

Gallon explains that this class of challenge samples a trail of mouse positions, including variation in speed and jitter. The demonstrated movement eases in, follows a slight curve, deliberately overshoots, and settles back into place. Vision supplies the destination; movement supplies the interaction behavior. His “moves like Jagger” joke captures the distinction: solving the geometry and satisfying the movement check are different problems. Turnstile, MTCaptcha, and Lemin have now exercised three different capabilities—reaching an isolated control, reading an image, and operating a control with a trajectory.

The last demonstration is reCAPTCHA v2: a checkbox followed by a blurry image grid asking for objects such as crosswalks or traffic lights. This combines the earlier capabilities into a system with two responsibilities, which Gallon calls the solver and the operator.

The solver is deterministic code. It handles the initial checkbox interaction, reaches the challenge iframe, captures the grid each round, and restarts when a round expires. Those operations do not require a model to choose each next click. The unresolved step is visual: deciding what the grid contains—for example, whether a tile shows a bus.

That decision belongs to the operator. The solver supplies the grid; the agent selects the matching tiles and returns its answer; the solver resumes the browser work. Between rounds, the agent waits.

The boundary is the important part. The model receives the decision that requires perception, while code retains control of the repetitive interaction sequence.

During playback, Gallon remarks that the agent spotted bicycles he had missed in some tiles. He reports that the completed solution is verified by the server. The selected frame shows an intermediate bicycle grid with a tile checked, beside the explanation of expiring rounds.

Speed matters because, in Gallon’s account, each image round expires and one challenge can contain several consecutive rounds. Calling a model for every look and click consumes the available window before the interaction sequence finishes. His successful design instead uses deterministic browser code with one quick visual judgment per round. Google’s documented two-minute validity period applies to an issued response token, not to the duration of each image-selection round.

This is where the early CLI argument becomes consequential: fewer model round trips can determine whether the task finishes in time, not merely whether it feels responsive. The relevant architectural property is programmable execution between model decisions. Gallon describes his solution as repeatable and reliable, but supplies no trial counts or measured success rate for that claim.

The CAPTCHA demonstrations serve as stress tests for the engineering method. They expose different reasons a browser interaction can fail, then show why an off-the-shelf agent benefits from carefully chosen sensors, explicit verification, and reusable execution code. The durable result of exploration is a recorded solution: a programmable tool that uses the browser’s observations, checks whether actions landed, and escalates only when the simpler technique fails.

The opening account warning also gets a resolution. Gallon says OpenAI rescinded the threat after a conversation, leaving him with access to Codex. Chrome Agent, his daily tool for this work, is installable through the Python ecosystem. He also explicitly encourages building your own: the method does not depend on adopting his particular implementation.