# From MCP to Scale: Pipelines That Build Themselves

Rafael Levi

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/zTZ0qunQXnM-from-mcp-scale-pipelines-that-build-themselves) · [Video](https://www.youtube.com/watch?v=zTZ0qunQXnM)

## From MCP to Scale: Pipelines That Build Themselves

A live headphones search shows how an agent can inspect a protected website, generate a reusable scraper, and reserve model work for building, checking, and repairing the pipeline.

Basic familiarity with LLM tools, HTML selectors, JSON, and Python functions will help you follow the scraper workflow.

### Ten thousand products should not mean ten thousand LLM extractions

[0:17](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=17s)

How do you scan 10,000 products without paying for an LLM to parse every page? Access is only the first problem. Rafael Levi’s preceding session covered using MCP to reach websites protected by CAPTCHA and bot detection; this demonstration begins with what happens after the agent can reach them.

**Ask the model to build the scraper, then let the scraper process the pages.** Bright Data’s skills supply instructions for constructing a pipeline, while its MCP tools retrieve the target HTML so the agent can discover selectors. The reusable artifact is executable extraction code, rather than another model response containing manually extracted products.

### The scraper tax continues after the first successful run

[1:16](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=76s)

Writing a scraper creates an ongoing obligation: collect the data, notice when it disappears, and fix the extraction before a client notices. Levi calls this the scraper tax. Maintenance can consume more effort than the original implementation.

![Slide titled “The Scraper Tax” with three columns: “Write the scraper,” “It breaks next Tuesday,” and “Fix It. Again. Forever.”](https://ai.engineer/api/talk-essay-media/52eb2c311bd659ff7f920efaab97a1e51e2f67236f8cff5b27d4b0ec556d7186)

The Scraper Tax: write the scraper, watch it break, and fix it again.

Changing selectors and React-driven pages make that obligation harder. An agent with website access can inspect the page, identify the required data, write a scraper, and execute it. To turn those capabilities into maintenance, however, it needs **explicit data validation**: a successful process exit does not establish that the expected fields were collected.

Levi describes daily collections checked by an LLM client every 30 minutes; when the data is healthy, the client shuts down. A missing field becomes a reason to investigate and repair the scraper. His five-minute repair example illustrates the intended alternative to waking a maintainer at night; the live session does not demonstrate a complete failure-and-repair cycle.

### Give the agent a reusable interface

[2:27](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=147s)

The demonstration uses Claude Code with Bright Data MCP already connected. The [Bright Data skills repository](https://github.com/brightdata/skills) supplies scraper-building instructions and best practices. Levi contrasts earlier setups that took weeks with a build he says now takes roughly three minutes.

The initial request defines two inputs: a search keyword and a maximum page count. After correcting the target domain to `walmart.com`, Levi asks for headphones across three pages. Those inputs matter because the result should be a scraper that can run another search later, not code hardwired to one product category.

The construction sequence separates instructions from page structure:

1. Retrieve the scraper-building skills.
2. Use the connected MCP to inspect the target page’s HTML.
3. Identify selectors and generate the scraper.

The visible `scrape_as_markdown` call reads the skill material as text. HTML is useful for discovering extraction structure; Markdown is enough when the agent only needs to read instructions.

Levi advertises 5,000 free requests through [Bright Data MCP](https://github.com/brightdata/brightdata-mcp/blob/main/README.md) after opening an account. For setup today, the [current FAQ](https://docs.brightdata.com/ai/mcp-server/faqs) specifies 5,000 monthly requests for search and Web Unlocker scraping, with shared account-level credits; that clarification does not establish the offer’s historical scope.

### The same pattern works for a shopping question

[4:58](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=298s)

Levi reports that an earlier three-page run saved about one million tokens by generating a scraper and parsing the HTML with a script. That is a reported comparison, without the underlying token accounting shown here. The mechanism is straightforward: the model helps create the extraction logic, but ordinary code performs the repeated parsing.

This can be useful even without a production collection. Instead of searching Google and reading a CNET comparison, a shopper could ask for the best-reviewed headphones across five marketplace pages. The scraper gathers the reviews and product information; the model can then answer the shopping question from the extracted data. Access to protected sites, including those behind Cloudflare, expands the sources available to that workflow.

### An old scraper forces a fresh test

[6:00](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=360s)

The Walmart run encounters a demonstration problem: an old scraper from testing is still present. Levi stops the run, asks for its deletion, and invites the audience to choose another marketplace. Amazon is suggested first; he says it is less aggressive than expected and that he can scrape it using data-center IPs. The audience then supplies Very, spelling out `very.com`.

Levi starts a new session, works through navigation and paste difficulties, and confirms that Very sells headphones. He repeats the headphones task on a site he says he has never used. The switch preserves the original task while removing the previously prepared Walmart scraper from the test.

### From a search to a scheduled listener

[8:01](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=481s)

While the new scraper builds, Levi describes a house-search listener: check a specified area for a private house below a price ceiling, then notify him when a match appears. Levi says his house-search scraper ran every half hour, notified him after a few days, and found the house where he now lives. Scheduling turns a one-time extraction into an ongoing watch for a condition.

His preference for Claude Code over Codex is based on his own experience of getting the work done with fewer detours. In the active session, the agent has found the page structure and search parameters and is building the scraper. Levi estimates that manual selector inspection and scraper construction previously took a day to a day and a half on a good day.

The browser work need not run on the presenter’s laptop. Levi says Bright Data has over 150 million IPs and that a laptop could control 1,000 browsers running on its servers. Those are infrastructure claims, not a concurrency test performed in the session. The immediate result is smaller and concrete: the Very run reports 90 products, with a Unicode problem still to fix in pound-denominated prices.

### Choose the right web-access interface

[10:08](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=608s)

An audience question prompts a closer look at MCP. Levi describes a 66-tool catalog; the current README lists 69, so a present-day installation should not be expected to reproduce that exact inventory. At the request layer, the agent supplies a URL and the unlocking service handles the access work: required headers and cookies, and CAPTCHA solving with a token when needed. The agent receives page content rather than having to construct those browser-like requests itself.

For retrieval, the useful distinction is what the next stage needs:

| Output | Useful when |
|---|---|
| Full HTML | Discovering selectors or parsing page structure |
| `scrape_as_markdown` text | Reading content without HTML tags |

Markdown reduces the material passed to the model when page structure is irrelevant. Meanwhile, the Very scraper is still working through its pound-character issue: obtaining the page and correctly representing its data are separate problems.

Levi also describes about 500 prebuilt domain APIs. For an Amazon product, for example, an agent can request structured JSON directly instead of building a custom scraper. Remote browsers cover a further class of work where retrieving a URL is insufficient and interaction is required. Together, these options let the agent choose among page content, an existing structured endpoint, and browser automation.

### Walmart returns a verification screen, then product data

[11:38](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=698s)

Levi returns to Walmart with a narrower task: search for headphones and report the first result, initially without Bright Data MCP. The ordinary fetch reaches a robot-or-human security verification screen instead of the requested product results.

He then repeats the task with MCP. The agent uses `scrape_as_markdown`, requesting page text rather than full HTML. While the response is pending, Levi speculates that a remote browser may be dealing with Walmart’s click-and-hold verification. He describes an in-house AI CAPTCHA solver that moves and clicks controls, but the delay alone does not establish which challenge-handling operation occurred.

The request returns product names and a product list, with Levi describing the first result as a blue headphone.

| Request path | Observed result |
|---|---|
| Fetch without Bright Data MCP | Security verification screen |
| MCP `scrape_as_markdown` | Headphone product results |

This comparison demonstrates the access difference for the live Walmart request. It is distinct from the earlier task of generating and reusing a scraper.

### Measure the extraction, then keep checking it

[13:36](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=816s)

With results available, Levi asks the agent to estimate how many tokens it saved compared with parsing the pages directly with an LLM. The intended saving comes from pulling HTML and extracting data with the generated parser, rather than repeatedly sending the full page through the model.

For the Very extraction of 90 products, Levi reads an approximately 62% token saving from the agent’s input/output/total breakdown. He considers that relatively low and wonders whether the site has unusually structured HTML. The source does not establish the model, tokenizer, baseline counts, caching, or whether scraper-building costs are included, so this figure cannot be directly reconciled with the earlier million-token estimate.

The generated code is also the object the agent would maintain. Levi proposes a Claude schedule that runs or checks the scraper every 30 minutes and repairs it when it breaks. He presents the access layer as eliminating CAPTCHA and anti-bot obstacles; the live evidence supports successful access in the demonstrated request, not a guarantee across every protected site.

### Inspect the Python scraper, then change its inputs

[15:33](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=933s)

Asked to show the code, Levi opens the generated scraper. It uses the [Web Unlocker API](https://docs.brightdata.com/scraping-automation/web-unlocker/introduction) to request page data, followed by a generated parser and output schema. He likens the request to MCP’s `scrape_as_markdown`; they share an access function, but their output interfaces differ. Web Unlocker can return raw page content or a JSON envelope, while the MCP tool returns Markdown.

The Python code exposes the requested keyword and maximum-page parameters and loops through pages to collect products. That is the important boundary: changing the search should change the function’s inputs, not require the model to rediscover the website and rewrite the parser.

![Code editor displaying a Python scraper function with keyword and maximum-page parameters, a page loop, and product collection logic.](https://ai.engineer/api/talk-essay-media/eea5ac6092060e4965930f86b2a4dbeef747d92b2220685daee25c2c3fc02fcf)

Generated Python scraper code showing keyword search and pagination.

A compact Python orchestration layer for this interface can keep retrieval and parsing separate while accumulating the results:

```python
from collections.abc import Callable
from typing import Any


def collect_products(
    keyword: str,
    max_pages: int,
    fetch_page: Callable[[str, int], str],
    parse_products: Callable[[str], list[dict[str, Any]]],
) -> list[dict[str, Any]]:
    if not keyword.strip():
        raise ValueError("keyword must not be empty")
    if max_pages < 1:
        raise ValueError("max_pages must be positive")

    products: list[dict[str, Any]] = []
    for page in range(1, max_pages + 1):
        html = fetch_page(keyword, page)
        products.extend(parse_products(html))
    return products
```

Here, `fetch_page` owns the site’s search URL and access request; `parse_products` owns its extraction rules. The page loop can remain unchanged when those site-specific details need repair.

Levi locates the API-key and zone configuration, then asks for a second run: laptops across two pages. The agent can trigger the script, or another script can automate the invocation; the visual interface is simply his preferred way to demonstrate it.

Before the rerun finishes, Levi estimates roughly 60 tokens to invoke the script and roughly 100 to obtain the data. Those are preliminary estimates, not the later reported totals. Once the scraper produces JSON, the model can answer questions about the extracted records. The useful economy is passing selected fields instead of whole pages; JSON formatting alone does not guarantee fewer tokens than Markdown.

### Public access does not remove terms and legal constraints

[17:44](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1064s)

An audience member asks whether the same system works on authenticated sites. Levi’s answer establishes the service boundary for this demonstration: public data only, with nothing behind login. He warns that creating an account means accepting terms and conditions and advises checking restrictions on scraping or robots. He points to LinkedIn litigation and Twitter’s restrictions after Musk’s acquisition as examples of the conflicts surrounding data access.

Explicit acceptance checkpoints during a search deserve the same attention. Levi describes Bright Data’s scraping workflow as avoiding login and acceptance of terms, then cites litigation victories involving Meta and X. He compares public collection to writing down prices displayed on a counter and selling those observations.

That analogy should not be read as unrestricted permission to collect and reuse anything visible online. The Meta ruling concerned particular contract claims and logged-off scraping; Bright Data’s counsel also [reports dismissal of X’s claims](https://www.proskauer.com/insights/get-pdf/27629), on grounds more specific than a blanket public-data exemption. Technical access, contractual obligations, and permitted reuse remain separate questions.

### Retrieving results and reading them are different costs

[19:24](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1164s)

The laptop rerun stalls. Levi attempts manual execution, encounters variable-setting and batch-related difficulties, and then reports that the run has completed. He asks the agent how many tokens it spent obtaining the results. The interruption makes the operational boundary visible: generating a script does not eliminate configuration and execution problems.

Even for personal tasks, his default request is to build a script that the agent can reuse. For the completed rerun, Levi cites roughly 1,000 tokens to obtain the data and roughly 10,000 if the model also goes through the JSON. These later figures supersede neither the earlier estimates nor their unspecified accounting boundaries; they distinguish execution-related work from reading the returned records. His description of the cost as pennies is not accompanied by a model or pricing calculation.

### Protected sites and listeners that wait

[21:45](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1305s)

Levi estimates that this MCP access is most useful for about 20% of domains, particularly heavily protected sites. He names Akamai, DataDome, and Cloudflare, with real estate and large e-commerce sites as valuable targets. His underlying chain is access → scraper construction → maintenance: the agent needs to inspect the actual site before it can build or repair the extraction. The domain percentage is his estimate, not a measured coverage result.

A listener need not collect a million records to be useful. Alongside the apartment example, Levi describes a restaurant-reservation listener that has been waiting for two months and is intended to book automatically when a spot opens. That booking is still pending. The practical extension is from collecting a dataset to watching for a specific event and acting when its condition becomes true.

### When the pipeline needs to click

[23:01](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1381s)

The final audience question moves beyond scraping: can the agent fill and submit a form? Levi says yes, while retaining the no-login boundary. Flight availability is his example. A search on a service such as Skyscanner may lead to a hashed URL that the agent cannot construct directly, so it must operate the search interface instead.

For geographically restricted results, the agent can request a remote browser with a United States IP, then fill fields and click through the search. Levi says the browser infrastructure supplies prerecorded mouse movements and typing behavior: pointer movement instead of instantaneous jumps, varying typing speed, and potentially mistakes. Those behaviors are intended to make interaction telemetry resemble a person’s activity.

This division of responsibility also affects model choice. Levi says **Claude Haiku** is sufficient for his browsing agents when the browser infrastructure handles the behavioral masking. The model decides what to do; the browser carries out the interaction, and the generated script handles repeated extraction. That leaves model reasoning concentrated on the parts of the pipeline that need it.

## Resources

- [Bright Data skills](https://github.com/brightdata/skills) — Agent skills for scraper construction, site analysis, pagination and Bright Data integrations.
- [Bright Data MCP](https://github.com/brightdata/brightdata-mcp/blob/main/README.md) — Setup and tool documentation for web search, page retrieval, structured extraction and browser automation.
- [Current MCP setup and billing FAQ](https://docs.brightdata.com/ai/mcp-server/faqs) — Current guidance on free-tier limits, browser tools, timeouts and usage tracking.
- [Web Unlocker API](https://docs.brightdata.com/scraping-automation/web-unlocker/introduction) — Request-based retrieval of public page content, with configuration and links to request examples.
- [Rafael Levi's Bright Data Hackathon Kit](https://github.com/ScrapeAlchemist/brightdata-hack-pack) — Python and JavaScript examples, MCP integration guidance and a Next.js starter application.

## Slides

### From MCP to Scale: Pipelines That Build Themselves

Rafael Levi introduces data collection at scale with LLMs, highlighting the token cost of parsing 10,000 products.

[0:17](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=17s) [Title slide for Rafael Levi's session on pipelines that build themselves.](https://assets.ai.engineer/sites/aiecode2025/sha256/08a33cdba8bf282dc2ea863cbe046d8c05e6cae87b804be29ffed8ff6da7a28c.webp)

bright data | AI Engineer EUROPE

SESSION 2

From MCP to Scale:
PIPELINES THAT BUILD THEMSELVES

Rafael Levi
DevRel Engineer | Bright Data

Interface text:
Session2_MCP_to_Scale (1)
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Help
Background  Layout  Theme  Transition
Slideshow  Share
Enhance this slide
TIMING: 30 seconds
AIE
Google DeepMind

### From a Single URL to a Production-Grade Data Pipeline

Levi describes using Bright Data’s MCP to extract HTML and help an LLM identify selectors and build a scraper, aiming to turn one URL into a production-grade pipeline without human scripting.

[0:47](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=47s) [Slide describing AI agents building a production-grade data pipeline from a single URL without human scripting.](https://assets.ai.engineer/sites/aiecode2025/sha256/7634ae64cc027b31203c8f469b7a3f7edf8a1937df594e7e56bfa7ce2ab0a221.webp)

How AI agents go from exploring a single URL to running a production-grade data pipeline — with zero human scripting.

bright data
3

Interface text:
Session2_MCP_to_Scale (1)  PPTX
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Help
Menus  Fit  Background  Layout  Theme  Transition
Slideshow  Share
Enhance this slide
Click to add speaker notes
Ask Google  Claude  Claude (MCP)  All Bookmarks
Search  News for you  ENG  1:27 PM  4/9/2026
AIE  Braintrust  WorkOS  OpenAI

[1:02](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=62s) [Slide stating that AI agents can turn a single URL into a production-grade data pipeline without human scripting.](https://assets.ai.engineer/sites/aiecode2025/sha256/900721d6a636ed767208a07321b266a760166cea1a4f9e2321d92bacb7a05917.webp)

How AI agents go from exploring a single URL to running a production-grade data pipeline — with zero human scripting.

bright data
3

Interface text:
Session2_MCP_to_Scale (1)  PPTX
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Help
Menus  Fit  Background  Layout  Theme  Transition
Slideshow  Share
Enhance this slide
Click to add speaker notes
Ask Google  Claude  Claude (MCP)  All Bookmarks
Search  News for you  ENG  1:28 PM  4/9/2026

### The Scraper Tax

Scrapers need repeated repairs as sites, selectors and anti-bot measures change. The slide estimates that upkeep takes 80% of engineering time.

[1:15](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=75s) [Three panels describe writing a scraper, its subsequent breakage, and ongoing maintenance.](https://assets.ai.engineer/sites/aiecode2025/sha256/5de2091b24de46ef8af0a381d708806f68cebbed2e7d54eab81f7a34f36f1a68.webp)

The Scraper Tax
What you're actually spending your time on.

Write the scraper
Inspect elements. Find selectors. Handle pagination. Deal with JS rendering. Test. Debug. Repeat.

It breaks next Tuesday
Site redesign. New anti-bot. Changed selectors. Different DOM structure. Your pipeline is dead.

Fix it. Again. Forever.
Scraper maintenance is a full-time job nobody wants. 80% of engineering time goes to upkeep, not building.

bright data
4

Interface text:
Session2_MCP_to_Scale (1)
PPTX
File Edit View Insert Format Slide Arrange Tools Help
Menus
Fit
Background Layout Theme Transition
Slideshow
Share
Enhance this slide
TIMING: 60 seconds
Search
1:28 PM
4/9/2026

### How the Agent Builds a Pipeline

The slide maps four stages: the agent explores a site via MCP, identifies its structure, writes a scraper, then runs it against thousands of URLs through Bright Data APIs.

[1:48](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=108s) [Four connected stages show an agent exploring a site, understanding its structure, writing a scraper, and executing it at scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/c61f25485d1dff305ef783e7450a417e340cb0ca81efeb2813d0b62f5ebb302c.webp)

How the Agent Builds a Pipeline
MCP for intelligence. APIs for execution.

1 Explore
Agent navigates the site via MCP. Renders JS, bypasses anti-bot, sees the real page.

2 Understand
Identifies data structure, selectors, pagination patterns. Maps the site's architecture.

3 Write
Generates a production scraper script targeting the Bright Data API for scale.

4 Execute
Runs against thousands of URLs via BD APIs. Batch processing, not one-by-one.

bright data
5

Interface text:
Session2_MCP_to_Scale (1)
PPTX
File Edit View Insert Format Slide Arrange Tools Help
Background Layout Theme Transition
Slideshow Share
Enhance this slide
TIMING: 90 seconds

### Demo: From URL to Pipeline

The planned Claude Code demo starts with a URL and goal, explores the site via MCP, then generates a Python scraper and runs it across the product catalog.

[2:28](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=148s) [A demo overview lists receiving a URL, exploring the site, identifying its data structure, generating a scraper, and running it at scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/328405d8110bee88895b3bbd22714fca1794daf96aa1aafda5f6a70067f3bfb5.webp)

Demo: From URL to Pipeline
Watch the agent build a production scraper from scratch.

What you'll see:
Agent receives a URL and a goal: 'get product data from this site'
Explores the site via MCP — handles JS rendering and anti-bot transparently
Identifies data structure: product names, prices, images, selectors
Generates a complete Python scraper using Bright Data APIs
Runs the scraper against the full product catalog at scale

bright data
7
4

Interface text:
Session2_MCP_to_Scale (1)
PPTX
File Edit View Insert Format Slide Arrange Tools Help
Background Layout Theme Transition
Slideshow Share
Enhance this slide
TIMING: 30 seconds
AIE
Engineering the future of AI

### Bright Data skills repository

The presenter opens the Bright Data skills repository for a Claude Code scraper demo. It collects scraper-building resources and best practices for the agent.

[2:43](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=163s) [GitHub repository page for Bright Data skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e6a0c50f499239616c9f06cbb6801cc62b521b1f3147ee068032e46f351e670.webp)

brightdata / skills
Public

.claude-plugin
skills
.gitignore
LICENSE
README.md

Bright Data Plugin for Claude Code
Unlock the web with AI-powered scraping, search, and structured data

POWERED BY BRIGHT DATA
LICENSE MIT
SKILLS 11
DATASETS 40+
MCP TOOLS 60+

Quick Start · Skills · Data Feeds · MCP · CLI · Competitive Intel · Scraper Builder · Best Practices · Python SDK · Setup

AIE
Engineering the future of AI

Interface text:
github.com/brightdata/skills
Platform · Solutions · Resources · Open Source · Enterprise · Pricing
Search or jump to... · Sign in · Sign up
Code · Issues 1 · Pull requests · Actions · Projects · Security and quality · Insights
main · 2 Branches · 0 Tags · Go to file · Code
24 Commits
About
skills.sh/brightdata
Readme · MIT license · Activity · Custom properties
20 stars · 0 watching · 5 forks
Releases: No releases published
Packages: No packages published
Contributors 3
meirk-brd Meir Kadosh
claude Claude
EyalDassa

### Scraper request

The presenter asks Claude to use the Bright Data skills repository to build a Walmart scraper with keyword and page-limit inputs, then search for headphones across three pages.

[3:15](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=195s) [Claude Code prompt requesting a Walmart product scraper and a three-page headphones search.](https://assets.ai.engineer/sites/aiecode2025/sha256/a037c2156b117e1f3fdf2983e4989f29ee147e6f4c7372a7ae6e2d734b818d19.webp)

go to https://github.com/brightdata/skills

build me a scraper 2 inputs, keyword search, max pages for walmart.com

Run a search for headaphones, collect 3 pages

AIE
AI Engineer
EUROPE

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
New session
Build Walmart product scraper with JS...
Auto accept edits
Opus 4.6 1M
claude
Local
Rafael Levi
Max plan

### Scraper workflow

The assistant checks the Bright Data skills repository and inspects a Walmart search page for its structure before building the requested scraper.

[4:59](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=299s) [Claude Code checks the skills repository and examines a Walmart search page.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8b7286185d485e46763087bd72887373c3ed4b56c6900caba0f2ff6f5227dd6.webp)

go to https://github.com/brightdata/skills
build me a scraper 2 inputs, keyword search, max pages for walmart.com
Run a search for headaphones, collect 3 pages

Let me check the Bright Data skills repo first, then build the scraper.
Used Bright Data integration

Let me scrape a Walmart search page first to understand the structure, then build the scraper.
Used Bright Data integration

Got the structure. Now I'll build the scraper and run it.

AIE
Engineering the future of AI

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Build Walmart scraper with keyword search
Preview
Reply...
Auto accept edits
Opus 4.6 1M
Rafael Levi
Max plan
C:\claude
1:32 PM
4/9/2026

### Building a Walmart scraper with Bright Data

Claude checks the Bright Data skills repo, finds an old Walmart scraper and starts a three-page headphone search. The presenter realizes he forgot to delete the test scraper.

[5:15](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=315s) [Claude Code reports finding an existing scraper and checking its configuration before running it.](https://assets.ai.engineer/sites/aiecode2025/sha256/300caae4324b3ab0e597b2d2a5fecf4a13055a18c36ff0a7ac678d57211a1498.webp)

go to https://github.com/brightdata/skills
build me a scraper 2 inputs, keyword search, max pages for walmart.com
Run a search for headaphones, collect 3 pages

Let me check the Bright Data skills repo first, then build the scraper.
Used Bright Data integration

Let me scrape a Walmart search page first to understand the structure, then build the scraper.
Used Bright Data integration

Got the structure. Now I’ll build the scraper and run it.
Ran 2 commands, created a file, read a file

There’s already a Walmart scraper here. Let me check the env vars and run it.
Check env var names (hide values)

Good - the existing scraper uses BRIGHTDATA_API_TOKEN which matches the .env. Let me run it for “headphones” with 3 pages.

AIE
AI Engineer
EUROPE

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Build Walmart scraper with keyword search
Preview
Reply...
Auto accept edits
Opus 4.6 1M
Rafael Levi
Max plan
C:\claude
1:32 PM
4/9/2026

[6:13](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=373s) [A coding assistant shows the steps taken to run a headphone search across three pages.](https://assets.ai.engineer/sites/aiecode2025/sha256/4614fae8c2021c9202984981533e1db0f21fdfe4de01d97e2d10a548ef7dfa7c.webp)

go to https://github.com/brightdata/skills
build me a scraper 2 inputs, keyword search, max pages for walmart.com
Run a search for headaphones, collect 3 pages

Let me check the Bright Data skills repo first, then build the scraper.
Used Bright Data integration

Let me scrape a Walmart search page first to understand the structure, then build the scraper.
Used Bright Data integration

Got the structure. Now I’ll build the scraper and run it.
Ran 2 commands, created a file, read a file

There’s already a Walmart scraper here. Let me check the env vars and run it.
Check env var names (hide values)

Good - the existing scraper uses BRIGHTDATA_API_TOKEN which matches the .env. Let me run it for “headphones” with 3 pages.
Run Walmart scraper for headphones, 3 pages

AIE
AI Engineer

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Build Walmart scraper with keyword search
Preview
Reply...
Auto accept edits
Opus 4.6 1M
Rafael Levi
Max plan
C:\claude
1:33 PM
4/9/2026

### Rebuilding the scraper

After deleting the old Walmart scraper, the assistant starts a fresh version based on the Bright Data skills repository, using connected MCP tools instead of raw API calls.

[7:06](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=426s) [The coding assistant reports deleting the old scraper and creating a replacement.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ef289a2c8427cc959caaffede7ce2e11b612de1f7e44b8ec59a1f31ac2296d8.webp)

go to https://github.com/brightdata/skills
build me a scraper 2 inputs, keyword search, max pages for walmart.com
Run a search for headaphones, collect 3 pages

Let me check the Bright Data skills repo first, then build the scraper.
Used Bright Data integration
Let me scrape a Walmart search page first to understand the structure, then build the scraper.
Used Bright Data integration
Got the structure. Now I’ll build the scraper and run it.
Ran 2 commands, created a file, read a file
There’s already a Walmart scraper here. Let me check the env vars and run it.
Check env var names (hide values)
Good - the existing scraper uses BRIGHTDATA_API_TOKEN which matches the .env. Let me run it for “headphones” with 3 pages.
Run Walmart scraper for headphones, 3 pages

delete the old scraper
Delete old walmart scraper and results
Deleted. Now let me build a fresh one based on the Bright Data skills repo pattern - using the MCP tools I have connected directly instead of raw API calls.
Created a file

Engineering the future of AI
AIE

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
New session
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Build Walmart scraper with keyword search
Preview
Reply...
Auto accept edits
Opus 4.6 1M
Rafael Levi
Max plan
C:\claude
1:33 PM
4/9/2026

### Building a Very.co.uk product scraper

A Walmart scraper prompt becomes a Very.co.uk test: Claude inspects the Bright Data skills repo and the site's search after being asked to collect three pages of headphones.

[7:27](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=447s) [A new coding session contains a draft prompt for a headphone scraper.](https://assets.ai.engineer/sites/aiecode2025/sha256/d2cae846a7325cdf3c193390e56d5364684c6b6d27e217afb182c753ea178448.webp)

o to https://github.com/brightdata/skills
build me a scraper 2 inputs, keyword search, max pages for walmart.com
Run a search for headaphones, collect 3 pages

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
New session
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Auto accept edits
Opus 4.6 1M
claude
Local
Rafael Levi
Max plan
AIE
Google DeepMind

[7:45](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=465s) [A coding assistant begins building a product search scraper.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c0d26952be815f7d170e79fe10931fbf7222a39f854c4ab67aa6a3e4b9a0972.webp)

Build web scraper for Very.co.uk product search

go to https://github.com/brightdata/skills
build me a scraper 2 inputs, keyword search, max pages for https://www.very.co.uk/
Run a search for headaphones, collect 3 pages

Let me start by looking at the Bright Data skills repo to understand the format, and also check what very.co.uk search looks like.

Bright Data: Scrape as Markdown
Bright Data: Scrape as Markdown  https://github.com/brightdata/skills
GitHub - brightdata/skills - GitHub
Bright Data: Scrape as Markdown  https://www.very.co.uk/search?[illegible]

Discombobulating...

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Build web scraper for Very.co.uk produ...
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Preview
Reply...
Auto accept edits
Opus 4.6 1M
C:\claude
Rafael Levi
Max plan
AIE
Braintrust
WorkOS
OpenAI

### Scraper results and price parsing

The assistant reports 90 products across three pages, then attributes missing prices to pound-sign encoding and edits the scraper.

[10:03](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=603s) [The assistant reports 90 products across three pages and investigates missing prices.](https://assets.ai.engineer/sites/aiecode2025/sha256/8170ba073a6514f36387d5c4caa9d729b474aec9bbdffca7fac562e7e7637038.webp)

Build web scraper for Very.co.uk product search

build me a scraper 2 inputs, keyword search, max pages for https://www.very.co.uk/
Run a search for headphones, collect 3 pages

Let me start by looking at the Bright Data skills repo to understand the format, and also check what very.co.uk search looks like.

Used 2 tools

The search URL got an error page. Let me try a different URL pattern and also check the scraper builder skill for reference.

Used 2 tools

I've got the site structure. The search URL pattern is https://www.very.co.uk/search/{keyword}?page={n} with 30 products per page. Let me build the scraper.

Created a file

Now let me check for the API key and run it.

Ran 3 commands

90 products scraped across 3 pages. Let me check the price parsing - looks like the prices aren't showing in the summary table. Let me check the actual JSON output.

Read a file

The prices are null because the £ symbol in the HTML is being encoded differently. The text parsing is splitting on [illegible] but the price text contains the pound sign as a Unicode character. Let me fix the price extraction.

Edit
Edit C:\claude\demos\very_scraper.py +20 -16
Edit C:\claude\demos\very_scraper.py +5 -5

Clauding...

AIE
Engineering the future of AI

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Build web scraper for Very.co.uk produ...
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Preview
Reply...
Bypass permissions
Opus 4.6 1M
C:\claude
Rafael Levi
Max plan
20°C Mostly cloudy
1:36 PM
4/9/2026

### Debugging missing prices

After collecting 90 products across three pages, the assistant attempts a fix for pound-sign encoding and reads the output again when prices remain N/A.

[10:37](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=637s) [The assistant reports that prices are still unavailable and reads the results to investigate.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f5b354d12ad3000640f4ea10bec94e605ae69a5ba5df3736b24358401e90de3.webp)

Build web scraper for Very.co.uk product search

Let me start by looking at the Bright Data skills repo to understand the format, and also check what very.co.uk search looks like.

Used 2 tools

The search URL got an error page. Let me try a different URL pattern and also check the scraper builder skill for reference.

Used 2 tools

I've got the site structure. The search URL pattern is https://www.very.co.uk/search/{keyword}?page={n} with 30 products per page. Let me build the scraper.

Created a file

Now let me check for the API key and run it.

Ran 3 commands

90 products scraped across 3 pages. Let me check the price parsing - looks like the prices aren't showing in the summary table. Let me check the actual JSON output.

Read a file

The prices are null because the £ symbol in the HTML is being encoded differently. The text parsing is splitting on | but the price text contains the pound sign as a Unicode character. Let me fix the price extraction.

Edited 2 files, ran a command

Still N/A for prices in the summary. Let me check what the actual text content looks like to debug the price parsing.

Read
Read C:\claude\demos\very_headphones_results.json
Read 15 lines

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Trade advisor daily  Daily
Today
Build web scraper for Very.co.uk produ...
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Preview
Reply...
Bypass permissions
Opus 4.6 1M
C:\claude
Rafael Levi
Max plan
AIE
AI Engineer EUROPE

### Scraper results

The scraper reports 90 of 188 headphone products across three pages, with current prices captured but regular prices missing. The presenter then asks how many tokens the scripted approach saved.

[13:32](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=812s) [A scraper summary reports 90 products across three pages and notes that regular prices are missing.](https://assets.ai.engineer/sites/aiecode2025/sha256/9593197fac48640b5a11fd64d44b522490a4a5b45a660f88f2c0db0e24388784.webp)

data.

Read a file

Clean data. Let me also check that regular_price is being captured for sale items - looks like it's null. The issue is likely that “Current sale price” and “Regular price” are separate text nodes within the card. Let me check one more section where I'd expect a regular price.

Read a file

The Sony WH-CH520 shows as sale price £35 with savings £14, but no regular price. Looking at the original HTML, the prices use “Current sale price is £35” and “Regular price was £49” format, but after splitting by |, the “Current sale price is” prefix is part of the text. The regex should catch it - let me check if the issue is that “Current sale price” comes through differently in the actual scraped text.

The scraper is working well - 90 products collected across 3 pages with brand, name, price, savings, rating, and reviews. The regular_price field is null because the site's HTML structures those prices in accessibility labels that don't always survive the text extraction. The sale/current price is captured correctly.

Here's the summary:

Scraper: C:\claude\demos\very_scraper.py
Output: C:\claude\demos\very_headphones_results.json

Usage:
python very_scraper.py <keyword> <max_pages>
python very_scraper.py headphones 3
python very_scraper.py "nike trainers" 5

Results for "headphones" (3 pages):
• 90 products scraped from 188 available
• Fields: product_id, brand, name, sale_price, regular_price, savings, rating, reviews, url
• Brands include Sony, Apple, Beats, JBL, JLab, Soundcore, Bowers & Wilkins, Sennheiser, Shokz, Skullcandy, and more
• Prices range from £19.99 (JLab Go Air Pop) to £499 (Apple AirPods Max)

Interface text:
AIE
Engineering the future of AI
Chat  Cowork  Code
Build web scraper for Very.co.uk product search
Preview
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Search Walmart for headphones witho...
Build web scraper for Very.co.uk produ...
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
tell me if you were to do this manually, how much more tokens have you use, how many did you sav
Bypass permissions
Opus 4.6 1M
Rafael Levi
Max plan
C:\claude
20°C  Mostly cloudy
ENG
1:40 PM
4/9/2026

### Scraper token-savings comparison

For 90 products across three pages, the demo estimates that local Python parsing saves about 20K output tokens compared with parsing in conversation. The script can also be reused.

[14:29](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=869s) [A table compares estimated token use for a script and manual in-conversation parsing.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b9d1afd7d9d29ed5a2d37ba46756529947ec691a44f6ff84e7a68956e401afd.webp)

[illegible] ~5K ~4K
Read JSON output ~2K ~500
Total ~62K ~5.5K

The actual data collection (3 pages, 90 products) happened in Python at zero token cost - just 3 HTTP calls processed locally.

If I'd done it manually in-conversation
Each Very.co.uk search page returns ~12-15K tokens of markdown. To scrape and parse all 3 pages in-context:

Step | Input tokens | Output tokens
3x scrape_as_markdown calls | ~40-45K | ~500
Parsing 90 products from markdown | ~45K (re-reading) | ~15K (outputting structured data)
Formatting final JSON | ~15K | ~10K
Total | ~100-105K | ~25.5K

The savings
 | Script | Manual | Saved
Input tokens | ~62K | ~105K | ~43K (41%)
Output tokens | ~5.5K | ~25.5K | ~20K (78%)
Total | ~67.5K | ~130K | ~62.5K (46%)

The big win: output tokens are 4-5x more expensive than input tokens, and the script approach saved 78% on output. Parsing 90 products in Python costs zero tokens. Parsing them in my response would have been ~20K output tokens.

Plus you now have a reusable script - next time you run it, the token cost is literally zero.

Engineering the future of AI
AIE

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Build web scraper for Very.co.uk produ...
Search Walmart for headphones witho...
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Build web scraper for Very.co.uk product search
Preview
Reply...
Bypass permissions
Opus 4.6 1M
Rafael Levi
Max plan
C:\claude
20°C Mostly cloudy
1:41 PM
4/9/2026

[14:58](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=898s) [A token comparison estimates savings from using a Python scraper instead of parsing products in conversation.](https://assets.ai.engineer/sites/aiecode2025/sha256/b37753ee01b42fea726e1f184ee612a651e88699c150198a575e1f71854ea2b8.webp)

[illegible] ~5K ~4K
Read JSON output ~2K ~500
Total ~62K ~5.5K

The actual data collection (3 pages, 90 products) happened in Python at zero token cost - just 3 HTTP calls processed locally.

If I'd done it manually in-conversation
Each Very.co.uk search page returns ~12-15K tokens of markdown. To scrape and parse all 3 pages in-context:

Step | Input tokens | Output tokens
3x scrape_as_markdown calls | ~40-45K | ~500
Parsing 90 products from markdown | ~45K (re-reading) | ~15K (outputting structured data)
Formatting final JSON | ~15K | ~10K
Total | ~100-105K | ~25.5K

The savings
 | Script | Manual | Saved
Input tokens | ~62K | ~105K | ~43K (41%)
Output tokens | ~5.5K | ~25.5K | ~20K (78%)
Total | ~67.5K | ~130K | ~62.5K (46%)

The big win: output tokens are 4-5x more expensive than input tokens, and the script approach saved 78% on output. Parsing 90 products in Python costs zero tokens. Parsing them in my response would have been ~20K output tokens.

Plus you now have a reusable script - next time you run it, the token cost is literally zero.

AI Engineer
EUROPE
AIE

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Build web scraper for Very.co.uk produ...
Search Walmart for headphones witho...
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Build web scraper for Very.co.uk product search
Preview
Reply...
Bypass permissions
Opus 4.6 1M
Rafael Levi
Max plan
C:\claude
20°C Mostly cloudy
1:42 PM
4/9/2026

### Scraper results and price-field limitation

The demo reports 90 of 188 headphone products across three pages. Sale prices are captured, but regular prices remain null; the on-screen explanation points to accessibility labels lost during text extraction.

[15:44](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=944s) [The demo summarizes 90 scraped products and notes that regular prices were not captured.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1c6803b16a7079ad53083e6a5be4fb6b54369a1e2fbd0cddc4bdbe25b57dabe.webp)

Prices are now extracting. The [illegible] renders as [illegible] in the terminal but let me verify the JSON has clean data.

Read a file

Clean data. Let me also check that regular_price is being captured for sale items - looks like it's null. The issue is likely that “Current sale price” and “Regular price” are separate text nodes within the card. Let me check one more section where I'd expect a regular price.

Read a file

The Sony WH-CH520 shows as sale price £35 with savings £14, but no regular price. Looking at the original HTML, the prices use “Current sale price is £35” and “Regular price was £49” format, but after splitting by |, the “Current sale price is” prefix is part of the text. The regex should catch it - let me check if the issue is that “Current sale price” comes through differently in the actual scraped text.

The scraper is working well - 90 products collected across 3 pages with brand, name, price, savings, rating, and reviews. The regular_price field is null because the site's HTML structures those prices in accessibility labels that don't always survive the text extraction. The sale/current price is captured correctly.

Here's the summary:
Scraper: C:\claude\demos\very_scraper.py
Output: C:\claude\demos\very_headphones_results.json

Usage:
python very_scraper.py <keyword> <max_pages>
python very_scraper.py headphones 3
python very_scraper.py "nike trainers" 5

Results for "headphones" (3 pages):
• 90 products scraped from 188 available
• Fields: product_id, brand, name, sale_price, regular_price, savings, rating, reviews, url
• Brands include Sony, Apple, Beats, JBL, JLab, Soundcore, Bowers & Wilkins, Sennheiser, Shokz, Skullcandy, and more
• Prices range from £19.99 (JLab Go Air Pop) to £499 (Apple AirPods Max)

tell me if you were to do this manually, how much more tokens have you use, how many did you save

Interface text:
Chat  Cowork  Code
New session
Search
Scheduled
Dispatch
Customize
All projects
Scheduled
Trade advisor daily  Daily
Today
Build web scraper for Very.co.uk produ...
Search Walmart for headphones witho...
Build Walmart scraper with keyword se...
Build Walmart product scraper with JS...
Build web scraper for Very.co.uk product search
Preview
Reply...
Bypass permissions
Opus 4.6 1M
Rafael Levi
Max plan
C:\claude
AI Engineer EUROPE
AIE
20°C Mostly cloudy
1:42 PM
4/9/2026

### Very.co.uk scraper code walkthrough

The scraper sends a request through Bright Data’s Web Unlocker, retries failed fetches, and begins parsing Very.co.uk product links from the returned HTML.

[15:56](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=956s) [Python code configures a web request and retries failed fetches.](https://assets.ai.engineer/sites/aiecode2025/sha256/0efe49bbf25e8ddfbabfe8e0188272d19d065f75d68f546395f43ee60da17de1.webp)

"""

import json
import os
import sys
import time
import re
import logging
import requests
from bs4 import BeautifulSoup

# --- Configuration ---
API_KEY = os.environ.get("BRIGHTDATA_API_KEY") or os.environ.get("BD_API_TOKEN")
ZONE = os.environ.get("BRIGHTDATA_UNLOCKER_ZONE", "unlocker")
OUTPUT_DIR = os.path.dirname(os.path.abspath(__file__))

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
log = logging.getLogger(__name__)

# --- Fetcher ---
def fetch_page(url: str, retries: int = 3) -> str:
    """Fetch a page through Bright Data Web Unlocker."""
    for attempt in range(retries):
        try:
            response = requests.post(
                "https://api.brightdata.com/request",
                headers={"Authorization": f"Bearer {API_KEY}"},
                json={"zone": ZONE, "url": url, "format": "raw"},
                timeout=60,
            )
            response.raise_for_status()
            return response.text
        except requests.RequestException as e:
            log.warning(f"Attempt {attempt + 1} failed for {url}: {e}")
            if attempt < retries - 1:
                time.sleep(2 ** attempt)
    raise RuntimeError(f"Failed to fetch {url} after {retries} attempts")

Interface text:
AIE
AI Engineer EUROPE
very_scraper.py
C: > claude > demos > very_scraper.py
Installing update: 23%
Ln 9, Col 1  Spaces: 4  UTF-8  LF  Python  Python 3.13.4
1:42 PM  4/9/2026

[16:03](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=963s) [Python code requests a page and begins parsing product links.](https://assets.ai.engineer/sites/aiecode2025/sha256/25ffe2c1dae1aaa6119395b883820634bd458d5ce53f9467529b59c273c65556.webp)

# --- Configuration ---
API_KEY = os.environ.get("BRIGHTDATA_API_KEY") or os.environ.get("BD_API_TOKEN")
ZONE = os.environ.get("BRIGHTDATA_UNLOCKER_ZONE", "unlocker")
OUTPUT_DIR = os.path.dirname(os.path.abspath(__file__))

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
log = logging.getLogger(__name__)

# --- Fetcher ---
def fetch_page(url: str, retries: int = 3) -> str:
    """Fetch a page through Bright Data Web Unlocker."""
    for attempt in range(retries):
        try:
            response = requests.post(
                "https://api.brightdata.com/request",
                headers={"Authorization": f"Bearer {API_KEY}"},
                json={"zone": ZONE, "url": url, "format": "raw"},
                timeout=60,
            )
            response.raise_for_status()
            return response.text
        except requests.RequestException as e:
            log.warning(f"Attempt {attempt + 1} failed for {url}: {e}")
            if attempt < retries - 1:
                time.sleep(2 ** attempt)
    raise RuntimeError(f"Failed to fetch {url} after {retries} attempts")

# --- Parser ---
def parse_products(html: str) -> list[dict]:
    """Extract product data from Very.co.uk search results HTML."""
    soup = BeautifulSoup(html, "html.parser")
    products = []

    # Very uses product card links with .prd URLs
    # Each product is wrapped in an anchor linking to /product-slug/PRODUCTID.prd
    product_links = soup.select('a[href$=".prd"]')
    [illegible]

Interface text:
AIE
AI Engineer EUROPE
very_scraper.py
C: > claude > demos > very_scraper.py > fetch_page
Installing update: 51%
Ln 42, Col 40  Spaces: 4  UTF-8  LF  Python  Python 3.13.4
1:42 PM  4/9/2026

### Extracting product fields

The demo highlights the generated output schema: brand, name, prices, savings, rating and reviews. The surrounding parser skips payment text and looks for a standalone price.

[16:23](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=983s) [Python code highlights product fields including price, rating and reviews.](https://assets.ai.engineer/sites/aiecode2025/sha256/af2a346d67b1a021e08b02ae7cae3f68a103f7d2ba360f7d06db382482541986.webp)

AIE

Engineering the future of AI

def extract_fields(text: str, url: str) -> dict:
        # Skip payment-related text
        if "Very Pay" in part or "More time to pay" in part:
            continue

        # Brand is usually a short name (1-3 words), name is longer
        if brand is None and len(part.split()) <= 4:
            brand = part
        elif name is None:
            name = part
        elif brand and not name:
            name = part

    # Standalone price (e.g., "£119")
    for part in parts:
        standalone_price = re.match(r'^[\£\u00a3$]([\d,.]+)$', part.strip())
        if standalone_price and sale_price is None:
            sale_price = f"\u00a3{standalone_price.group(1)}"

    product["brand"] = brand
    product["name"] = name or brand  # fallback
    product["sale_price"] = sale_price
    product["regular_price"] = regular_price
    product["savings"] = save_amount
    product["rating"] = rating
    product["reviews"] = reviews

    return product

def parse_total_results(html: str) -> int:
    """Extract total result count from the page."""
    soup = BeautifulSoup(html, "html.parser")
    text = soup.get_text()
    match = re.search(r'of (\d+)\)', text)
    if match:
        return int(match.group(1))
    return 0

Interface text:
File  Edit  Selection  View  Go  Run  Terminal  Help
very_scraper.py
C: > claude > demos > very_scraper.py > extract_fields
Update is ready, click to restart
Ln 170, Col 33 (257 selected)  Spaces: 4  UTF-8  LF  Python  Python 3.13.4

### Returning to the fetcher

The presenter returns to the scraper’s configuration. Its fetcher posts URLs to Bright Data Web Unlocker, returns the raw response, and retries failed requests with increasing delays.

[16:24](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=984s) [Python imports, configuration and page-fetching code are visible.](https://assets.ai.engineer/sites/aiecode2025/sha256/4eba060eb0789f0c1e521c3324edf0a68c007656d4cd7954482fb70d9e33fe8f.webp)

import json
import os
import sys
import time
import re
import logging
import requests
from bs4 import BeautifulSoup

# --- Configuration ---
API_KEY = os.environ.get("BRIGHTDATA_API_KEY") or os.environ.get("BD_API_TOKEN")
ZONE = os.environ.get("BRIGHTDATA_UNLOCKER_ZONE", "unlocker")
OUTPUT_DIR = os.path.dirname(os.path.abspath(__file__))

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
log = logging.getLogger(__name__)

# --- Fetcher ---
def fetch_page(url: str, retries: int = 3) -> str:
    """Fetch a page through Bright Data Web Unlocker."""
    for attempt in range(retries):
        try:
            response = requests.post(
                "https://api.brightdata.com/request",
                headers={"Authorization": f"Bearer {API_KEY}"},
                json={"zone": ZONE, "url": url, "format": "raw"},
                timeout=60,
            )
            response.raise_for_status()
            return response.text
        except requests.RequestException as e:
            log.warning(f"Attempt {attempt + 1} failed for {url}: {e}")
            if attempt < retries - 1:
                time.sleep(2 ** attempt)
    raise RuntimeError(f"Failed to fetch {url} after {retries} attempts")

# --- Parser ---

AIE
Engineering the future of AI

Interface text:
File Edit Selection View Go Run Terminal Help
Search
very_scraper.py
C: > claude > demos > very_scraper.py > scrape_search > max_pages
Update is ready, click to restart.
Ln 186, Col 43 (14 selected)  Spaces: 4  UTF-8  LF  Python  Python 3.13.4  Continue (NE)
20°C Mostly cloudy
1:43 PM
4/9/2026

## Full transcript

[0:00](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=0s) [upbeat music] All righty, let's dive into it.

[0:17](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=17s) So, uh, in the previous session, I, I don't know how many of you were here, but I talked about how MCP gives access to LLMs to websites that are behind, uh, CAPTCHA, bot detection systems, and so on.

[0:30](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=30s) And in this session, I wanna talk about, um, how do you actually collect data on scales with LLM? Because a lot of times I'm seeing on Reddit and other social media is like, "Oh, I need to scan ten thousand products," but that's so many tokens if I need to parse everything with LLM.

[0:44](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=44s) So obviously you don't do that, right? So

[0:49](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=49s) the whole session is how do you build pipelines, right? With LLM. Instead of telling, "Hey, LLM, can you go and, uh, parse this for me?" Build a scraper that's gonna parse it for me, and I will demonstrate how easy it is, uh, with our skills, right?

[1:03](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=63s) So Bright Data has skill sets that actually teaches your LLM on how to build a pipeline. With our MCP, it can actually extract HTML so that it knows what are the selectors that it needs to parse, and so on.

[1:16](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=76s) So, um, the scrape tax, right? Write the scraper. Uh, I don't know, how many of you actually wrote scrapers before?

[1:25](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=85s) Okay, so you know the headache when all of a sudden things are missing, data is missing, you gotta wake up. I don't know if you guys did it for clients, and clients are like, "Oh, my God, there's no..."

[1:33](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=93s) So basically that's what used to be. You write a scraper, and you maintain it. As a matter of fact, sometimes you maintain it more than it, it takes, it takes you to write it, right?

[1:41](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=101s) So everything, especially if the website is constantly changing or its selectors, or if it's a React website, it gets really complicated, right? So an agent, it, it, it solves all that headache, right?

[1:54](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=114s) It can explore with our MCP. It understands what the data is needed. It writes the scraper, and actually runs it and executes it and maintains it. As a matter of fact, I do collections on a daily basis, and I have every thirty minutes, I have an LLM clients pool, uh, spools up.

[2:09](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=129s) It checks what the data is collected, make sure that everything is fine. Everything's fine, it shuts down. If, for example, there's a, um, always set a validation for data, right?

[2:18](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=138s) Let's say a data point is, uh, missing some-something, your agent fixes it, five minutes. You don't have to wake up in the middle of the night.

[2:27](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=147s) Uh, so let me demonstrate. So for this demo, I wanna use Claude Code. I don't know how many of you use Claude, if you're using Codex or whatever it is.

[2:37](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=157s) I prefer Claude Code. I like how it...

[2:40](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=160s) I like it. It does a great job. So the way that you build scrapers now, and it's ridiculous because, you know, back in the day it took weeks to set it up.

[2:49](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=169s) We created a Bright Data, like GitHub page, right? brightdata/skills. Here, basically, your agent has everything that it might need, okay? Um, scrape builders, best practices, how to... E-everything. Everything that it needs in order to build a scraper.

[3:05](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=185s) And what I wanna show is, uh, first of all, uh, let's...

[3:11](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=191s) I'm gonna start simple. Go to build me a scraper, and I wanted to record it because I thought it was gonna take a long time, but now it takes literally like three minutes.

[3:21](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=201s) Build me a scraper, uh, two inputs, keyword search, max pages for walmart.com. Everybody's familiar with Walmart. Walmart has a very aggressive anti-bot systems. Anybody try to scrape it, um, it just doesn't work.

[3:40](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=220s) But through Bright Data- The website name is wrong. I think the volume, it might be remote. It will figure it out, but you're, you're right. Let's just correct that.

[3:54](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=234s) walmart.com. Um, run a search for headphones. Collect three pages.

[4:08](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=248s) Uh, so what it's gonna do right now, it's gonna go to our GitHub. It's gonna get all the skill set that it needs, how to build scrapers. Then it's gonna go, uh, with the MCP that's already connected to the Claude.

[4:18](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=258s) It's gonna extract the HTML from the page. Uh, it's gonna find all the selectors that it needs, and it's gonna build you a scraper. And, uh,

[4:29](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=269s) so right now you can see a scrape_as_markdown is basically extracting the, the, the, the text, right? So for the skills, we don't need the HTML, we just need the text.

[4:38](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=278s) Uh, scrape_as_markdown is a part of the MCP, of Bright Data MCP. Uh, Bri-- Uh, the MCP has five thousand requests for free, so if anybody wants to try it, you guys can try it for free.

[4:48](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=288s) Um, you will need to just open an account with Bright Data, which it doesn't cost you anything. And

[4:58](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=298s) let it do its thing. So why is this better than having an LLM parse every single page, every HTML? It's... I'm, I'm literally, afterwards I'm gonna ask it to tell you how many, how much tokens it saves.

[5:12](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=312s) So for the three pages, I've done this before, it saves about a million tokens just to, from building the scraper, uh, and using a script to parse the HTML.

[5:23](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=323s) And, um, even if you are not doing any scraping for production, but let's say that you wanna find the be-best headphones for the money, right? It used to be you go to Google, you go to some CNET where they, uh, compare everything.

[5:38](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=338s) But here you can use the we- the marketplace reviews. So I can tell it, like, "Hey, hey, um, scan the five pages. Find me the best, uh, reviewed headphones."

[5:48](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=348s) So it could be even useful for you as a personal, in, in a personal way, right? So instead of it getting blocked, it can actually find you things that you need on websites that are behind Cloudflare.

[6:00](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=360s) Oh, I forgot to delete the old one.

[6:05](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=365s) No, no. Stop, stop, stop. Hold on a second.

[6:09](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=369s) Delete the old scraper. Scr- scraper. I totally... I was testing it, and I totally forgot to clean up.

[6:20](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=380s) Uh, let's do a different website. Pick a website. What's a popular marketplace in, uh, that's aggressive in blocking in UK?

[6:28](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=388s) Amazon.

[6:29](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=389s) Amazon, it's, uh, not that aggressive. You'll be surprised. I can scrape it with data center IPs. What's a popular website that everybody uses here?

[6:37](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=397s) Very. Very.

[6:39](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=399s) Ve-

[6:39](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=399s) V-E-R-Y.com [door squeaking]

[6:43](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=403s) V- Very?

[6:47](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=407s) Very.

[6:47](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=407s) Like that?

[6:49](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=409s) [REDACTED:url]

[6:50](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=410s) Y.com?

[6:50](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=410s) Yeah.

[6:53](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=413s) Oh, that's UK?

[6:55](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=415s) Yeah.

[6:55](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=415s) Okay, perfect. So this way, I don't f- I don't wanna... Stop it. New session.

[7:04](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=424s) Uh, uh, uh, that's that. Let's do that.

[7:11](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=431s) Where's the new session? Oh, there it is.

[7:20](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=440s) Really? What is wrong with my paste? Okay.

[7:34](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=454s) Uh, it's clothing store?

[7:36](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=456s) Uh, it's general. Headphones, you name it.

[7:38](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=458s) Okay, so let's do headphones again, the same thing, right? So I don't know, I've never used this website. I've...

[7:46](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=466s) L- Clean test. No cheating. [laughs] Um, so, so what else does this give you? So again, market research.

[8:01](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=481s) Um, for example, I was looking for a new apartment, right? I was wanting to move a house. So with a Claude code, with Bright Data, I set up a listener, literally just telling, "Hey, listen, build me a scraper that will run every half an hour.

[8:15](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=495s) When in this area, there's a, a house, private house under this price, notify me." That's all I did. In a few days, I got a notification, and now I live there.

[8:24](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=504s) So these things are not only useful on a scale where, you know, oh, I need to scrape millions of records, but even for your personal use. It's so easy these days.

[8:35](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=515s) Claude Code does an amazing job, right? I know, uh, Codex does a great job as well, but I prefer Claude Code because, I don't know, it just, um, gives me less headache.

[8:45](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=525s) Codex sometimes takes me on a wild goose chase.

[8:49](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=529s) Um, how many of you have used vibe coding?

[8:54](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=534s) It's, it's amazing. Why not, right? Anybody can build anything now. Um,

[9:00](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=540s) so did you build it? I've got the structure, the search parameters, and with third version, let me build... Oh, okay, it's building a scraper. Um,

[9:07](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=547s) this used to take, I don't know, if f- you're having a good day, maybe a full day, maybe a day and a half, right? You need to go check out the selectors, figure it out.

[9:16](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=556s) It used to be interesting and fun, but you don't need to do that anymore. So what I'm sh- trying to show you guys is that

[9:24](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=564s) with our MCP, with our infrastructure, we have over 150 million IPs. Uh, with our unlocking technology, where we-- if the- if it needs to run a remote browser, so if you wanna...

[9:37](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=577s) It can actually write you a browser automation. Uh, the browsers are running on our system. So I can literally right now open a thousand browsers on this laptop that are running on our servers, and they can go do things, whatever you need it to.

[9:50](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=590s) So it already got 90 products. Okay, it just needs to fix the Unicode because it's in pounds.

[10:05](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=605s) Eh, uh-uh, uh-uh, almost done.

[10:08](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=608s) Can you also, um, go into details about the MCP, or is that-

[10:12](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=612s) Sure. I did it in a previous session, but while it's loading, let me tell you. So basically, the MCP gives you, uh, a agent 66 tools. Uh, some of them is basically, um, we have a system where it can send a cURL to any URL, and our system will literally get the HTML back.

[10:28](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=628s) It will solve a CAPTCHA if needed and send it with a token. It will, uh, it, it knows exactly what headers and cookies the website needs. So basically, it will make sure that the server thinks it's a browser and serve the HTML back.

[10:39](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=639s) So your agent can literally send cURLs, pull data without any questions. It can pull a full HTML. It can pull just a scrape_as_markdown. Markdown, it's, it's to save tokens, right?

[10:49](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=649s) You just want the text of the page. You don't care about the HTML tags.

[10:54](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=654s) It doesn't like the pound. Come on, you can do it.

[11:00](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=660s) Um, so that's number one. Second, we have about 500 different APIs pre-built for different domains. So instead of actually getting a Markdown, you can actually get a JSON of the product.

[11:11](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=671s) For example, for Amazon, we have pre-built API. You can listen... Uh, when you add your agent, it can be like, "Okay, go and check on Amazon something." It doesn't even need to build a scraper.

[11:19](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=679s) It can literally just send the, uh, the request and get the data back. Um, on top of that, remote browser infrastructure and, uh, anything that needs, uh, anything that your agent needs to access the web, it has, right?

[11:33](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=693s) So you don't get blocked. While this is running, I wanna show you, for example,

[11:38](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=698s) uh, tell... Wait, do Walmart, do a search for headphones without

[11:48](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=708s) MCP. What?... So I'm telling it, "Go to Walmart, do a search for headphones, and tell me what is the first result without the Bright Data MCP."

[11:59](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=719s) Uh, it's gonna do a fetch, which is gonna get blocked.

[12:07](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=727s) Product security verification screen. Robot or human, obviously, right? That's, that's the first thing.

[12:13](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=733s) Now do the same with Bri-- Um, oh my God.

[12:27](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=747s) So now it's using scrape_as_markdown. It's doing a search. It's only pulling the text. It's not pulling the HTML, just the text itself.

[12:45](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=765s) Always, when you do a live, it slows down. [laughs] The time is like [exhales]

[12:51](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=771s) It's more people clicking the bots than you.

[12:55](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=775s) Oh, actually it could be. It could be opening a browser and actually holding the button. You know, the Walmart has, like, one of those click and hold, and it needs to hold it for, like, thirty seconds, so it could be doing that right now.

[13:05](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=785s) So, uh- People are capturing right now. We, we have built a CAPTCHA-solving solution. Like, we literally have in-house... We have AI solving CAPTCHAs and

[13:16](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=796s) moving, clicking things, and like, wow, it's perfect. So it already got

[13:22](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=802s) first result, blue headphone. Uh, basically, there you go. It already has the results of the headphones without robot or human, with full product list and name. Uh, so we here already have output.

[13:36](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=816s) Tell me, if you were to do this manually, how ma- how much more tokens have you used? How many did you save? Let's just give it a breakdown, and this is probably the biggest issue, right?

[13:53](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=833s) Tokens is expensive. I'm going through millions and millions of tokens a day. I used to go more, but I, I optimized it. So, um, all of you are looking, how do we save money on LLM?

[14:04](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=844s) How do we waste less tokens of web access? And, um, Bright Data has the solution. Instead of parsing the full HTML, create a scraper. Uh, instead of using, um, you know...

[14:15](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=855s) Okay, it pulls the HTML and it extracts the data. It builds the scr-- uh, the parser.

[14:22](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=862s) But it's working very slow right now. I feel like everybody's vibe coding. So basically, here's the breakdown, right? What do we have here?

[14:31](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=871s) The big screens output [mumbling] parsing ninety products would...

[14:37](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=877s) Input tokens, output tokens, total save. So it's about a sixty-two percent save of tokens. This is not a high number. This website, I guess, uh, it has, uh, maybe a structured, um, HTML, I'm not sure.

[14:55](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=895s) But this is what I wanted to show you. And the best thing about it is that it will maintain it. If it breaks it, it will fix it. I can set up a loop, right?

[15:03](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=903s) So for example, in Claude, I can set a schedule. Every thirty minutes, you do and run and check something, and this is what we are giving you guys, okay?

[15:10](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=910s) With our MCP, your agent has access to all the web, uh, without getting blocked. No CAPTCHA will stop it. No robots, anti-bot systems will stop it, and, uh, you have no headaches of that.

[15:23](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=923s) I don't know how many of you have... You guys probably dealt with blocking, right? When you build a scraper, it gets a little complicated. So this kinda removes all the headache for it.

[15:33](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=933s) Um, any questions? Where is the code? Excuse me? The code, the code to generate the scraper, right? Uh, what is- Oh, I can... If you, you wanna see the code?

[15:45](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=945s) I mean, it's here. We have the scraper. Let's check it out.

[16:03](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=963s) So it's using API Web Unlocker. This is the same thing as a scrape_as_markdown with an MCP. It sends a simple request and gets the data back.

[16:12](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=972s) This is the parser that it built. I mean,

[16:22](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=982s) it looks pretty good. This is the, the schema that it built for the output.

[16:28](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=988s) It's doing a keyword search. It has the max page, right? So I asked for two inputs, the keyword search and how many pages to scrape.

[16:38](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=998s) And hold on. Let's see, let's see. We can run it through here as well. Where's the variables? API key, zone key. [humming]

[16:52](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1012s) Let's run it. Do another run on laptops. What are the two pages results? [clears throat]

[17:04](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1024s) And now you kinda control it. Like, you don't even need to run it, right? It will trigger it for you. It will maintain it for you. Um, you can create a script that will automatically do all of this.

[17:14](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1034s) I prefer the visual interface for dem-demonstration. Sometimes, you know, like, looking at our code, it's a little more complicated. But right now, basically, I can ask it any questions.

[17:23](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1043s) It will actually run the script, so it's not wasting tokens. To execute the script, maybe, like, sixty tokens, right, right now. So it's gonna literally get all this data for a hundred tokens, and then you can do whatever.

[17:34](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1054s) You can ask it questions about it, but now we're working with a JSON. A JSON is much more token efficient than any markdown or any other file, right?

[17:44](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1064s) Slow than usual. Can I ask, can you, can you do this on all-- with authorization site as well? Uh, no. We only talk, we only deal with public data.

[17:58](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1078s) So behind login, it's, um, it's private data. And, and, uh, just, you know, just a heads-up. If you create an account, you accept terms and conditions of the website.

[18:08](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1088s) Always check terms and conditions of the website. If they say, "Don't scrape," "Don't use robots," and you do, then that company can actually sue you, and this is happening, uh, you probably heard in the news all the time.

[18:18](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1098s) LinkedIn is suing them. Everybody's suing each other because- They, the data right now is like the new gold, the oil, right? Everybody's trying, like, "Oh, this is my data."

[18:27](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1107s) Elon Musk took over Twitter, locked it down. That's it. Literally, like, the, it used to be so open, and now it's all locked in. A few accounts that you can actually scrape.

[18:36](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1116s) So we're seeing this on every single level.

[18:41](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1121s) Very slow today. Um, so yes, also same thing. If you accept, uh, there's like a terms and ag- like, if you do a search and there's a checkpoint, I accept terms and conditions, always be careful with that.

[18:52](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1132s) So we only deal with public data, so nothing behind login. We don't accept terms and conditions, and we actually won a few lawsuits. We were sued by Meta. We were s-sued by Elon Musk a month after he took over, and, uh, the judge said it's very simple.

[19:07](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1147s) Public data is public data. It doesn't matter how you collect it, doesn't matter what you do with it, it's public. You know? It's like walking on the street, you write down the prices on the counter, and then you sell it to somebody.

[19:18](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1158s) It's public data. It's available. You can do whatever you want with it. Okay? So,

[19:24](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1164s) as usual, it gets stuck on a live demo, but

[19:28](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1168s) we can, we can run this manually. Oh, my God.

[20:00](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1200s) What the... I was setting a variable, it's doing this.

[20:05](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1205s) It's not gonna work. It's from that batch.

[20:16](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1216s) What am I missing? Still stuck? Oh, okay. It's done. Okay, so here we got, we got...

[20:28](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1228s) Let's see. How many tokens, how many tokens did you spend on getting the- these?

[20:43](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1243s) So even if you're not bui- building pipelines for any, um, you know, big companies, and you don't need to sc- for personal use, I always have the MCP connected.

[20:51](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1251s) I don't ask the, you know, LLM to go do. I always ask, build a script that can later on be used by it. It's using its own script to save the tokens.

[21:10](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1270s) Literally, like, we're talking about 1,000 tokens here. If, uh, you know, if, if it needs to s- go through the JSON, it's like 10,000 tokens. We're talking about, like, literally pennies compared to what it would be to actually scrape it.

[21:22](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1282s) Any more questions? How about this? Anybody has problems with actually getting data?

[21:30](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1290s) No, no question. Ev- everybody is like, "Oh, yeah, I'm not sharing my problems. I have no problems." And then they go home and like, "Oh, I do have a problem."

[21:37](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1297s) But in any case, listen, we have a booth. If you don't, if you wanna talk in private and you have questions about data or access or LLM, feel free.

[21:45](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1305s) I'm always willing to help. Um, we can connect on LinkedIn, by the way, guys, if you want. Let me just summarize a few things what we did here. So, um, self-healing pipeline.

[21:56](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1316s) When you have, when your agent actually has access to a blocked website and you're working... I mean, the MCP is mostly useful in about 20% of the, of the domains, right?

[22:06](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1326s) The ones that have Akamai, Datadome, Cloudflare, those heavy protected domains. They are the usually the most juiciest one, right? Like real estate or, uh, big e-commerce places. Uh, with our MCP, it has access to it.

[22:19](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1339s) If it has access to it, it can build a scraper. If it can build a scraper, it can maintain the scraper. This is basically the whole thing that I wanted to show you.

[22:25](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1345s) That in Bright Data, we have all the tools that your agent might need to explore, to build and maintain, uh, a pipeline, right? So if you wanna set up a listener for an apartment and you're looking to move into a cheaper place, or maybe you wanna book a table in a restaurant where it's always packed, and literally,

[22:45](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1365s) I, I, I have a listener right now waiting for two months already. Everybody's, like, booking right away. So as soon as a spot opens, it will automatically book a spot for me.

[22:52](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1372s) It can be very useful in just kinda like even, even the small things, personal things, right? I'm not talking about just enterprise scale, where I need to download a million records, okay?

[23:01](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1381s) But having access to all the websites, it's, it's, it's important. So apart from scraping, can you also do actions on the website? Yes. Of course. Fill a form. Fill up a form, submit it as well, yes.

[23:15](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1395s) The only thing you can't do is log in,

[23:19](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1399s) right? So if you have, um, let's say you need to perform a search, right? You cannot generate the URL, you need to click button. Let's say flights. You wanna check flights availability, Skyscanner or something like that, right?

[23:30](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1410s) The URL is usually a hash, and you can't, uh, do anything with it. So yes, LLM can spawn a browser, remote browser. Even if it's a geo-restricted site, it can be like, "Okay, I want IP from the United States."

[23:41](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1421s) So it's open a browser with the United States IP, and then it goes and clicks things, fills it up, and so on. The beauty of it is that our browser will mimic real human behavior.

[23:50](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1430s) So when your agent clicks, it's not a teleportation. There's a mouse pre-recorded, like a real human being moving it. When it types, it will type a little slower, speed up, like maybe even mistake, and so on.

[24:00](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1440s) So we have pre-recorded typing, we have pre-recorded mouse movements. So if the website has a tracker that's constantly sending to the server what the user is doing, it will look like a real human being.

[24:10](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1450s) Doesn't matter what your agent, even if it's a low, like, um, uh, for browsing agents, um, I'm never use the top models, right? For example, if with Claude, a Haiku model is more than enough for browsing.

[24:20](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1460s) If it's being masked that it's a real human, it works just fine.

[24:27](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1467s) Um, so that's pretty much it. If you guys wanna connect on LinkedIn, feel free. I'm always willing to help. If you have any questions regarding how to get data, if you have problems with accessing anything, feel free to message me.

[24:40](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1480s) I feel like there's been so much more than 15 minutes. I love it. It's the second time I'm doing a speech and I got, like, half an hour instead of 15 minutes.

[24:46](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1486s) It's amazing. Um, now's the time if you have any questions, discussions.

[24:55](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1495s) No, nothing. Great. Okay, guys, I guess I will conclude this session on this. Uh, feel free to come to the booth on the third floor if you have more questions, and, um,

[25:04](https://www.youtube.com/watch?v=zTZ0qunQXnM&t=1504s) let's keep the public data public. [laughs] Thank you. [clapping] [outro music]
