Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data

A product intelligence agent needs to answer a concrete question: should you buy this product? Yohan Raju, a solutions architect at Bright Data, opens with an exercise that will collect an Amazon listing, look for discussion of the same product on Reddit and combine the results into a recommendation. Before building it, he examines the work required to put usable web evidence in front of a model.

Collection often starts with a small, known set of websites. For ecommerce, Amazon and Walmart offer prices, reviews and broad product coverage. Then the request expands: add Shopee, Shopify stores, Etsy and more. Raju estimates three to five days to develop a scraper, including testing and QA. That is his practical estimate for an end-to-end scraper, rather than a measured constant for every website.

At a hundred or a thousand sites, new development competes with maintenance. Each scraper depends on knowledge of where the site exposes its fields, through page selectors or an API. If Amazon and two other sites change tomorrow, someone must identify the changes, repair extraction and determine whether customers have already received incomplete data or missed an agreed delivery time. The growing cost comes from supporting the existing collection while adding new coverage.

The selector example isolates a small change with a downstream consequence. A scraper extracts a title and a price. If the website changes the selector identifying the price, the scraper can stop returning that field even though the product page still exists. Product analysis downstream then loses an input. Repairing the selector restores future collection; recovering records missed during the outage may require backfilling.

Raju proposes agents as a way to automate scraper discovery, construction and repair. That is a proposed reduction in engineering work; this part of the recording does not demonstrate an agent detecting a selector change, fixing it and recovering missing data. The concrete failure remains useful: successful page access does not guarantee successful field extraction.

Two separate mechanisms make collection site-specific:

• Different layouts. Amazon and Walmart both have prices, titles and descriptions, but their layouts differ. Knowing the desired fields does not supply the extraction rules for each site.
• Access controls. Anti-bot systems and CAPTCHAs can prevent the collector from reaching usable content. Raju describes sites where a data center IP works and others where a residential IP and a browser are needed. Browser behavior matters too: rapid clicks, rapid scrolling or blocked images can trigger detection.

A traditional scraping job begins with something that decides what to collect and when. A scheduler or cron job might request daily or weekly updates; a configuration can supply particular URLs. Those inputs launch collection through a proxy and access layer. Raju distinguishes static data center or ISP addresses from rotating residential addresses, then places Bright Data's Web Unlocker and Browser API in the same access layer.

The access tools handle decisions such as whether a request was blocked, whether to retry and which IP to use. Once content arrives, parsing and validation have a different job: determine whether the required fields actually arrived. Raju's example is checking whether price is returned 90% of the time, alongside requirements for reviews and ratings. The 90% figure illustrates a completeness check; it is not a reported service success rate. Reaching the page and obtaining a sufficiently complete record are separate conditions.

Post-processing then deduplicates records, enriches data and processes batches for delivery to cloud storage or internal systems. Raju describes this as moving through bronze and silver stages toward a database. A duplicate or incomplete record can still be harmful after collection succeeds, so downstream cleaning belongs in the architecture. In the pipeline he describes, an access failure or selector break can force a rerun and sometimes a backfill, putting delivery commitments at risk.

The freshness problem becomes clearer through a travel question. “When's the best time to travel to San Francisco?” can invite a general seasonal answer. Add “this year,” and current events may affect the recommendation. The second question creates a reason to collect new information before answering, rather than relying entirely on information already available to the system.

Raju calls these the sync and async paths, grouping pre-trained information and RAG together on the latter side. The useful distinction here is when the evidence is obtained: during the request or beforehand. The talk's grouping does not distinguish model training from retrieval over an existing collection, so it should not be read as a precise definition of RAG.

Fresh collection also has a cost. Repeated requests for today's New York City weather could reuse a cached answer instead of paying for another web lookup and more model work each time. That trades repeated collection for reuse over a chosen freshness window. The example assumes the cached information remains adequate for the question; it does not establish a universal refresh interval for weather.

Even when the system chooses to search, the desired source may be inaccessible. Raju points to sites that restrict named AI crawlers through robots.txt and to the broader access problems his collection tools address. Choosing to fetch current evidence creates a dependency on the collection infrastructure actually returning it.

Bright Data's MCP server gives an agent access to collection tools through a common connection. Raju describes integrations with Claude Code and LangChain and a library of more than 60 APIs at the time of the recording. The capabilities include search results, page extraction as Markdown or HTML, and site-specific collection for services such as Amazon, Reddit, LinkedIn and Instagram.

The social-listening example shows why discovery and extraction are separate operations. A user supplies a name and asks for their social profiles and follower counts. Claude Code first uses search through MCP to find pages on Instagram, TikTok and LinkedIn. It then sends those discovered pages to the collection pipeline, obtains follower counts and returns them with context. Search supplies the destinations; the subsequent API requests supply the requested profile data.

Browser automation adds actions such as clicking and scrolling when fetching a page alone is insufficient. Raju suggests ordering food or searching travel sites for flights as possible applications. These are examples of the kinds of workflows browser tools could support, rather than completed demonstrations in this recording.

What changes when the same request has a different collection connection? In the side-by-side Glassdoor demonstration, both Claude sessions receive a request to use a browser to retrieve the contents of a Glassdoor URL. Raju identifies the left session as connected to Bright Data's MCP server: it returns results. The right session, without that connection, is blocked. The comparison makes the access dependency concrete while leaving the requested task unchanged.

This example demonstrates an access difference for the shown request. It does not establish a general success rate across Glassdoor pages or other sites. Its practical lesson is that a model can have a clear instruction and still lack the infrastructure needed to obtain the evidence.

Job data introduces a different reason for continuous collection. A resume-matching system needs to know which companies are posting, how many positions are available, their salaries and their locations. Collecting a dataset once cannot keep those fields current. Structured, consistent output lets later stages consume repeated updates without rebuilding their interpretation of the records each time.

Historical records answer another class of question. Months or years of job data can show how advertised salaries and demand have changed, while fresh records describe today's offerings. Raju uses a possible increase in software engineer salaries to illustrate the need to examine trends. This is a proposed data requirement for matching and market analysis, not a demonstrated matching algorithm or an established explanation for salary changes.

Reliability matters because the analysis depends on the data continuing to arrive. An interruption in collection can interrupt the larger pipeline, even if its model and downstream code have not changed. Here, freshness, historical coverage and stable record structure serve different purposes and must be planned together.

The hands-on exercise uses a Google notebook with two credentials: a Bright Data API token and a Gemini API key. Its sequence is explicit: call Bright Data's API for Amazon data, collect that response, call the Reddit API, capture its raw response and perform AI analysis over the collected evidence. This exercise uses API calls directly; the preceding MCP discussion explains another interface through which an agent could choose collection tools.

The selected product is a Stanley Quencher tumbler. The two sources contribute different kinds of evidence:

• Amazon product data. Price, images, URLs and product details describe the offer and the item.
• Reddit discussion. A keyword search for the specific tumbler retrieves posts about whether people consider it worth buying. This adds opinions beyond the listing's product fields.

After the keys load, the Amazon request returns. Raju identifies price, rating, other sellers and product features in the response. These are the fields available for the later analysis, rather than facts the model must supply from memory. He then triggers the Reddit search for Stanley Quencher reviews.

While Reddit collection runs, Raju returns to the choice of interface. The MCP tool library spans ecommerce, social media, financial intelligence, research and business intelligence. Direct scraper calls give the application an explicit sequence, as in this notebook; exposing tools through MCP allows an agent to choose which collection capability to invoke.

During the wait, an audience question asks what gives Bright Data an access advantage over other bots. Raju's answer returns to infrastructure: a collector might work for a few requests or a few days, then encounter additional detection. He uses Glassdoor again, describing a site recognizing repeated access from a data center IP and introducing more anti-bot checks.

The claimed advantage is maintaining access as conditions change: proxy infrastructure, built-in anti-bot handling and changes in how requests reach a site. That also explains the build-versus-buy decision. An in-house scraper carries an ongoing cost of diagnosing site changes and adapting collection, which Raju says can take more time than maintaining extraction itself. The answer describes the work the service takes on, without quantifying its reliability or cost advantage.

The Reddit response eventually returns a few posts about the tumbler. Raju describes comments suggesting that some people find it embarrassing while others are obsessed with it. The collected discussion therefore contains mixed social reactions, even before the model turns it into a buying recommendation.

How do those opinions become a decision alongside the product data? The final answer recommends buying the tumbler, citing a decent discount, overwhelmingly positive reviews with some mixed opinions, and high demand with low stock on Amazon. The observable result is a recommendation supported by several stated reasons. The demand and stock statements are claims in that generated answer; the recording does not expose enough of their underlying fields or inference to establish them independently, and a few retrieved Reddit posts do not establish representative customer sentiment.

The exercise ends with a useful division of work. Collection tools obtain current records from the sources; the application decides which sources to combine, what question to ask and how to turn the evidence into an action. The same pattern can extend beyond ecommerce, but access to more data leaves those decisions with the builder. In this notebook, the completed path is specific: an Amazon listing, Reddit discussion about the same tumbler and an explained purchase recommendation.