From MCP to Scale: Pipelines That Build Themselves

How do you scan 10,000 products without paying for an LLM to parse every page? Access is only the first problem. Rafael Levi’s preceding session covered using MCP to reach websites protected by CAPTCHA and bot detection; this demonstration begins with what happens after the agent can reach them.

Ask the model to build the scraper, then let the scraper process the pages. Bright Data’s skills supply instructions for constructing a pipeline, while its MCP tools retrieve the target HTML so the agent can discover selectors. The reusable artifact is executable extraction code, rather than another model response containing manually extracted products.

Writing a scraper creates an ongoing obligation: collect the data, notice when it disappears, and fix the extraction before a client notices. Levi calls this the scraper tax. Maintenance can consume more effort than the original implementation.

Changing selectors and React-driven pages make that obligation harder. An agent with website access can inspect the page, identify the required data, write a scraper, and execute it. To turn those capabilities into maintenance, however, it needs explicit data validation: a successful process exit does not establish that the expected fields were collected.

Levi describes daily collections checked by an LLM client every 30 minutes; when the data is healthy, the client shuts down. A missing field becomes a reason to investigate and repair the scraper. His five-minute repair example illustrates the intended alternative to waking a maintainer at night; the live session does not demonstrate a complete failure-and-repair cycle.

The demonstration uses Claude Code with Bright Data MCP already connected. The Bright Data skills repository supplies scraper-building instructions and best practices. Levi contrasts earlier setups that took weeks with a build he says now takes roughly three minutes.

The initial request defines two inputs: a search keyword and a maximum page count. After correcting the target domain to walmart.com, Levi asks for headphones across three pages. Those inputs matter because the result should be a scraper that can run another search later, not code hardwired to one product category.

The construction sequence separates instructions from page structure:

1. Retrieve the scraper-building skills.
2. Use the connected MCP to inspect the target page’s HTML.
3. Identify selectors and generate the scraper.

The visible scrape_as_markdown call reads the skill material as text. HTML is useful for discovering extraction structure; Markdown is enough when the agent only needs to read instructions.

Levi advertises 5,000 free requests through Bright Data MCP after opening an account. For setup today, the current FAQ specifies 5,000 monthly requests for search and Web Unlocker scraping, with shared account-level credits; that clarification does not establish the offer’s historical scope.

Levi reports that an earlier three-page run saved about one million tokens by generating a scraper and parsing the HTML with a script. That is a reported comparison, without the underlying token accounting shown here. The mechanism is straightforward: the model helps create the extraction logic, but ordinary code performs the repeated parsing.

This can be useful even without a production collection. Instead of searching Google and reading a CNET comparison, a shopper could ask for the best-reviewed headphones across five marketplace pages. The scraper gathers the reviews and product information; the model can then answer the shopping question from the extracted data. Access to protected sites, including those behind Cloudflare, expands the sources available to that workflow.

The Walmart run encounters a demonstration problem: an old scraper from testing is still present. Levi stops the run, asks for its deletion, and invites the audience to choose another marketplace. Amazon is suggested first; he says it is less aggressive than expected and that he can scrape it using data-center IPs. The audience then supplies Very, spelling out very.com.

Levi starts a new session, works through navigation and paste difficulties, and confirms that Very sells headphones. He repeats the headphones task on a site he says he has never used. The switch preserves the original task while removing the previously prepared Walmart scraper from the test.

While the new scraper builds, Levi describes a house-search listener: check a specified area for a private house below a price ceiling, then notify him when a match appears. Levi says his house-search scraper ran every half hour, notified him after a few days, and found the house where he now lives. Scheduling turns a one-time extraction into an ongoing watch for a condition.

His preference for Claude Code over Codex is based on his own experience of getting the work done with fewer detours. In the active session, the agent has found the page structure and search parameters and is building the scraper. Levi estimates that manual selector inspection and scraper construction previously took a day to a day and a half on a good day.

The browser work need not run on the presenter’s laptop. Levi says Bright Data has over 150 million IPs and that a laptop could control 1,000 browsers running on its servers. Those are infrastructure claims, not a concurrency test performed in the session. The immediate result is smaller and concrete: the Very run reports 90 products, with a Unicode problem still to fix in pound-denominated prices.

An audience question prompts a closer look at MCP. Levi describes a 66-tool catalog; the current README lists 69, so a present-day installation should not be expected to reproduce that exact inventory. At the request layer, the agent supplies a URL and the unlocking service handles the access work: required headers and cookies, and CAPTCHA solving with a token when needed. The agent receives page content rather than having to construct those browser-like requests itself.

For retrieval, the useful distinction is what the next stage needs:

Markdown reduces the material passed to the model when page structure is irrelevant. Meanwhile, the Very scraper is still working through its pound-character issue: obtaining the page and correctly representing its data are separate problems.

Levi also describes about 500 prebuilt domain APIs. For an Amazon product, for example, an agent can request structured JSON directly instead of building a custom scraper. Remote browsers cover a further class of work where retrieving a URL is insufficient and interaction is required. Together, these options let the agent choose among page content, an existing structured endpoint, and browser automation.

Levi returns to Walmart with a narrower task: search for headphones and report the first result, initially without Bright Data MCP. The ordinary fetch reaches a robot-or-human security verification screen instead of the requested product results.

He then repeats the task with MCP. The agent uses scrape_as_markdown, requesting page text rather than full HTML. While the response is pending, Levi speculates that a remote browser may be dealing with Walmart’s click-and-hold verification. He describes an in-house AI CAPTCHA solver that moves and clicks controls, but the delay alone does not establish which challenge-handling operation occurred.

The request returns product names and a product list, with Levi describing the first result as a blue headphone.

This comparison demonstrates the access difference for the live Walmart request. It is distinct from the earlier task of generating and reusing a scraper.

With results available, Levi asks the agent to estimate how many tokens it saved compared with parsing the pages directly with an LLM. The intended saving comes from pulling HTML and extracting data with the generated parser, rather than repeatedly sending the full page through the model.

For the Very extraction of 90 products, Levi reads an approximately 62% token saving from the agent’s input/output/total breakdown. He considers that relatively low and wonders whether the site has unusually structured HTML. The source does not establish the model, tokenizer, baseline counts, caching, or whether scraper-building costs are included, so this figure cannot be directly reconciled with the earlier million-token estimate.

The generated code is also the object the agent would maintain. Levi proposes a Claude schedule that runs or checks the scraper every 30 minutes and repairs it when it breaks. He presents the access layer as eliminating CAPTCHA and anti-bot obstacles; the live evidence supports successful access in the demonstrated request, not a guarantee across every protected site.

Asked to show the code, Levi opens the generated scraper. It uses the Web Unlocker API to request page data, followed by a generated parser and output schema. He likens the request to MCP’s scrape_as_markdown; they share an access function, but their output interfaces differ. Web Unlocker can return raw page content or a JSON envelope, while the MCP tool returns Markdown.

The Python code exposes the requested keyword and maximum-page parameters and loops through pages to collect products. That is the important boundary: changing the search should change the function’s inputs, not require the model to rediscover the website and rewrite the parser.

A compact Python orchestration layer for this interface can keep retrieval and parsing separate while accumulating the results:

from collections.abc import Callable
from typing import Any


def collect_products(
    keyword: str,
    max_pages: int,
    fetch_page: Callable[[str, int], str],
    parse_products: Callable[[str], list[dict[str, Any]]],
) -> list[dict[str, Any]]:
    if not keyword.strip():
        raise ValueError("keyword must not be empty")
    if max_pages < 1:
        raise ValueError("max_pages must be positive")

    products: list[dict[str, Any]] = []
    for page in range(1, max_pages + 1):
        html = fetch_page(keyword, page)
        products.extend(parse_products(html))
    return products

Here, fetch_page owns the site’s search URL and access request; parse_products owns its extraction rules. The page loop can remain unchanged when those site-specific details need repair.

Levi locates the API-key and zone configuration, then asks for a second run: laptops across two pages. The agent can trigger the script, or another script can automate the invocation; the visual interface is simply his preferred way to demonstrate it.

Before the rerun finishes, Levi estimates roughly 60 tokens to invoke the script and roughly 100 to obtain the data. Those are preliminary estimates, not the later reported totals. Once the scraper produces JSON, the model can answer questions about the extracted records. The useful economy is passing selected fields instead of whole pages; JSON formatting alone does not guarantee fewer tokens than Markdown.

An audience member asks whether the same system works on authenticated sites. Levi’s answer establishes the service boundary for this demonstration: public data only, with nothing behind login. He warns that creating an account means accepting terms and conditions and advises checking restrictions on scraping or robots. He points to LinkedIn litigation and Twitter’s restrictions after Musk’s acquisition as examples of the conflicts surrounding data access.

Explicit acceptance checkpoints during a search deserve the same attention. Levi describes Bright Data’s scraping workflow as avoiding login and acceptance of terms, then cites litigation victories involving Meta and X. He compares public collection to writing down prices displayed on a counter and selling those observations.

That analogy should not be read as unrestricted permission to collect and reuse anything visible online. The Meta ruling concerned particular contract claims and logged-off scraping; Bright Data’s counsel also reports dismissal of X’s claims, on grounds more specific than a blanket public-data exemption. Technical access, contractual obligations, and permitted reuse remain separate questions.

The laptop rerun stalls. Levi attempts manual execution, encounters variable-setting and batch-related difficulties, and then reports that the run has completed. He asks the agent how many tokens it spent obtaining the results. The interruption makes the operational boundary visible: generating a script does not eliminate configuration and execution problems.

Even for personal tasks, his default request is to build a script that the agent can reuse. For the completed rerun, Levi cites roughly 1,000 tokens to obtain the data and roughly 10,000 if the model also goes through the JSON. These later figures supersede neither the earlier estimates nor their unspecified accounting boundaries; they distinguish execution-related work from reading the returned records. His description of the cost as pennies is not accompanied by a model or pricing calculation.

Levi estimates that this MCP access is most useful for about 20% of domains, particularly heavily protected sites. He names Akamai, DataDome, and Cloudflare, with real estate and large e-commerce sites as valuable targets. His underlying chain is access → scraper construction → maintenance: the agent needs to inspect the actual site before it can build or repair the extraction. The domain percentage is his estimate, not a measured coverage result.

A listener need not collect a million records to be useful. Alongside the apartment example, Levi describes a restaurant-reservation listener that has been waiting for two months and is intended to book automatically when a spot opens. That booking is still pending. The practical extension is from collecting a dataset to watching for a specific event and acting when its condition becomes true.

The final audience question moves beyond scraping: can the agent fill and submit a form? Levi says yes, while retaining the no-login boundary. Flight availability is his example. A search on a service such as Skyscanner may lead to a hashed URL that the agent cannot construct directly, so it must operate the search interface instead.

For geographically restricted results, the agent can request a remote browser with a United States IP, then fill fields and click through the search. Levi says the browser infrastructure supplies prerecorded mouse movements and typing behavior: pointer movement instead of instantaneous jumps, varying typing speed, and potentially mistakes. Those behaviors are intended to make interaction telemetry resemble a person’s activity.

This division of responsibility also affects model choice. Levi says Claude Haiku is sufficient for his browsing agents when the browser infrastructure handles the behavioral masking. The model decides what to do; the browser carries out the interaction, and the generated script handles repeated extraction. That leaves model reasoning concentrated on the parts of the pipeline that need it.