Bringing agents onto the world wide web

Why can an agent use a computer, yet still struggle to finish ordinary work on the web? For anyone who tried Operator, the possibility is easy to see. The difficulty is turning that possibility into a system that keeps working. Paul Klein IV, founder of Browserbase, frames this as a capabilities overhang: models can do more than the systems around them reliably enable.

The web was built for people. Pages change, their contents consume large amounts of context, browsers fail to start, and blockers interrupt otherwise straightforward tasks. Klein encountered these problems early in his career, when web automation meant maintaining scripts every day. Agents now help him write more durable automation, but a durable script is still some distance from an autonomous agent that can navigate unfamiliar work. That gap is the problem to solve.

A year earlier, Klein considered models a major bottleneck for long-horizon tasks. His assessment has changed as their ability to use interfaces has improved. He attributes part of that progress to reinforcement-learning environments: training on human trajectories in environments that resemble the real web gives models practice with the work they will encounter. Klein says computer-use RL environments received as much investment in the preceding six months as coding environments received in the preceding year. That is his account of the field’s momentum, rather than a measured investment comparison presented here.

Better models nevertheless leave a deployment question. Klein invokes Dwarkesh’s contention that sufficiently capable models would naturally diffuse into use, then identifies the missing engineering: the agent harness, meaning the scaffolding, tools and systems that let a model interact with the world. Investing in that surrounding system can expose capabilities that a bare model interface leaves unused.

Karpathy’s November 2023 picture of an LLM surrounded by tools provides a useful architecture. A code interpreter lets it compute and execute programs; audio and video inputs, including screenshots, let it observe; a browser gives it access to websites; other LLMs can work as sub-agents. The harness connects these capabilities into an operating system for the task rather than leaving the model to produce text alone.

Coding agents offer a precedent. Klein points to a Factory-versus-Claude Code comparison using the same underlying model: changing the surrounding system can change task performance even when the model stays fixed. He credits Cursor with early harness engineering and describes Browserbase’s browser work as a similar effort to adapt models to a particular domain. The relevant comparison is between complete systems using a common model, not a claim that a harness somehow exceeds an intrinsic model score.

A domain harness is something an engineering team can build without becoming a model lab. A company knows its tasks, interfaces and failure modes; it can use that knowledge to improve the tools and context available to its agent. Whether custom harnesses will ultimately outperform increasingly capable RL-trained systems remains open in the talk. The immediate requirement is more practical: add the harness, then measure its performance against a baseline model configuration.

The opportunity extends well beyond programming. Klein cites Greg Brockman’s habit of asking why he did not use Codex for a task, treating that question as a way to find capabilities that remain unused. Klein reports qualitatively higher task completion in coding than in computer use, which he attributes to better tools and more developed harnesses. An Andreessen slide supplies the next motivation: the many non-coding workflows that could benefit from computer use. His requirements for reaching them are multimodal operation, harness engineering and reliable infrastructure.

Multimodal operation here includes choosing both the model and the method. A complex page may justify a more capable model; a simple page may not. The agent can also combine coding with computer use instead of treating every operation as a click on a screenshot. One route is to inspect network requests, then have a coding agent write a script that replays the relevant requests.

Klein reports that reliable production browser agents often write code alongside using the browser. His comparison is Claude Code producing a script versus Claude in Chrome carrying out the interaction: for repeatable work, the script can represent the task with less context than repeated page inspection. The browser helps discover or navigate the workflow; code can carry the repetitive portion.

Once a workflow repeats, memory and skills become part of the execution strategy. Klein introduces Browse.sh as a catalog of website skills, launched a few weeks before the talk. Those skills let an agent inspect what it can do before it visits a site. WebMCP provides another way to obtain knowledge of available website actions. Both address the same waste: an agent should not have to discover a familiar operation from scratch every time.

The same principle applies when the agent controls a site through a CLI such as Playwright’s: supply the relevant skills and context alongside the tool. Dumping the entire page into the model makes each interaction expensive and can obscure the useful information. The harness chooses both the tools and the information needed to use them. Its context should preserve what determines the next action while excluding page content that does not help produce a repeatable result.

The infrastructure beneath the harness must also be dependable. Klein uses the rush to buy Mac minis for OpenClaw as a revealing example: a machine at home provides macOS, remote access over SSH and a residential network environment that affects encounters with CAPTCHAs. That can support a personal setup, but it does not by itself solve operating thousands of customer agents. His joke about not having seen a SOC 2 compliant Mac mini deployment at scale points to the distance between a working personal machine and a managed production environment.

Consistency matters even before scale. Repeated runs should encounter the same kinds of inputs and outputs, including a stable page size and layout. If one run gets a mobile layout and the next gets a desktop layout, an otherwise unchanged task can expose different controls and navigation. The harness and model then have to absorb variation introduced by their own infrastructure.

For a Playwright-based harness, a fixed browser context makes that viewport requirement explicit. This TypeScript helper uses one desktop size for every page it opens; callers supply the browser and destination:

import type { Browser, BrowserContext, Page } from "playwright";

export async function openAgentPage(
  browser: Browser,
  url: string,
): Promise<{ context: BrowserContext; page: Page }> {
  const context = await browser.newContext({
    viewport: { width: 1280, height: 720 },
    isMobile: false,
    hasTouch: false,
  });

  try {
    const page = await context.newPage();
    await page.goto(url, { waitUntil: "domcontentloaded" });
    return { context, page };
  } catch (error) {
    await context.close();
    throw error;
  }
}

The caller closes context when the task finishes. Fixing the viewport removes one source of layout variation; it does not prevent the website itself from changing.

Improving the agent is only one side of the problem. Improving the web requires persuading website operators that they want agents to visit and work on users’ behalf. Accessibility is an existing foundation: instead of consuming only raw HTML and the DOM, an agent can use the accessibility tree and ARIA labels to identify components and understand which control to activate. A labeled component gives the agent a more useful representation than undifferentiated page markup.

WebMCP takes that idea from identifying controls to exposing structured actions. Klein describes it as publishing MCP servers inside a page; more precisely, Chrome’s WebMCP work exposes page tools through HTML and JavaScript APIs. It began as an early preview, and the subsequently updated Chrome documentation describes an origin trial rather than universal availability. The practical example is submitting a registration form through a website-provided tool: the agent gets a context-efficient action that the website deliberately exposes, without separately installing a server integration.

Files such as llms.txt, skills.md and AGENTS.md provide another place to publish useful instructions alongside a website. Together, accessible controls, structured actions and discoverable instructions reduce how much an agent must infer from the visual interface alone.

Reaching the right page does not answer the next question: how does the agent log in on the user’s behalf? Klein identifies two familiar approaches, each with operational costs.

• Shared passwords: The agent can use the user’s login, but handling those credentials securely is difficult.
• Service accounts: The agent gets limited access, but new work may require repeated permission grants.

Selected actions may also need human approval. For enterprise computer use, access to the required systems becomes a deployment gate even when the model and harness can perform the task.

Klein points to WorkOS auth.md as a way for an agent to discover how to register. The protocol describes registration on behalf of users and scoped, user-linked credentials, rather than necessarily creating an independent identity owned by the agent. The product-design question is therefore concrete: what does signup and login look like when an agent performs it? Making that path explicit gives software a chance to support agent use securely instead of forcing it through an improvised human flow.

Authentication does not settle the broader question of trust. A website built to block bad bots now also encounters agents carrying out legitimate requests. Klein argues that CAPTCHAs are less effective against agents than people assume; the more useful question is whether the site can identify an agent and connect it to the person authorizing its work.

Klein names WebAuthn in this discussion. Browserbase’s published agent-identity work instead names Web Bot Auth; these are distinct technologies, not interchangeable names. His broader proposal is a Verisign-like role for agents: a trusted issuer that can attest to an agent or its vendor. He presents that trust layer as an unresolved opportunity, alongside secure authentication, rather than as an already completed industry system.

Solving all of these problems can become a full-time engineering job. An existing platform is useful when it absorbs that work while preserving the choices an application needs. Klein’s requirements are practical:

• Scale: Support the move from one agent to thousands.
• Model choice: Allow the application to change providers as model capabilities improve.
• Identity and access: Broker trusted-agent access with anti-bot providers instead of leaving every application to negotiate it independently.

The platform’s value is the combination, not simply a remotely hosted browser.

Observability supplies the next layer. Screen recordings show what the agent encountered; logs expose its steps; network activity reveals what the browser exchanged with the website. Those records can feed the next iteration of the agent, making failures and unnecessary work available for review. Klein introduces Autobrowse, published earlier that year, as an example of improving an agent over multiple loops. The goal is to use execution evidence to improve future behavior; improvement on every run is an aspiration, not an automatic consequence of collecting traces.

Browser agents and web data extraction serve more than large AI-native companies. Klein describes being surprised by the range of smaller businesses that can benefit from browser automation. His examples move outside the technology industry: a logistics company in Singapore, a bank in South Africa and a lumber factory in Mexico.

These businesses may run on PHP websites, forms and people clicking buttons every day. Their workflows already have interfaces, but those interfaces expect human operators. Klein’s economic case for computer use is that it can bring AI into this existing software estate. The opportunity is to solve the company’s actual workflow with dependable browser infrastructure underneath it.

Klein introduces Browserbase Agents as a product launched the previous day, packaging the agent and harness together. He says Browserbase’s platform experience includes millions of browser sessions each month, informing its handling of edge cases. This is platform session volume, not a count of successfully completed autonomous tasks.

In the demonstration, a prompt is the entry point to the execution system. The platform supplies the harness, runtime, sandbox, code execution, fetch and search tools, and models. As the agent runs, Klein describes it examining and remembering its steps so that it can reuse what it learns.

The intended integration is a browser specialist acting as a sub-agent within a larger agentic system. That lets the application developer concentrate on the customer’s problem instead of assembling every browser capability. The completed demonstration frame shows a nursing-license search page alongside the run timeline, action list, and text and structured results: the task’s output and the evidence of its execution are visible together.

Optimization then looks backward at that execution and asks how to perform the task better. This closes the feedback loop introduced earlier: observe a run, review the behavior and use what was learned to improve the next attempt. The completed result is evidence of that run; retrospective optimization is the separate process for improving future runs.

Klein ends by acknowledging the frustration behind the room’s sparse attendance: after a year of difficulty getting browser agents into production, some builders have stepped back. His counterweight is firsthand experience with customers for whom the systems are working. He predicts a crowded room a year later as models, techniques and tools improve. The responsibility he leaves with builders is to turn those improvements into better systems—systems that can reliably perform the work people already do on the web.