Your Agent's Biggest Lie: "I Searched the Web"

Talk transcript

1. 0:00

   [upbeat music] Okay, so let's just work with it like this, a small room.

2. 0:19

   Hi, everybody. Welcome. My name is Rafael, um, representing Bright Data. Uh, Bright Data is basically, uh, web access platform to help agents or anybody collect public data on scale.

3. 0:31

   And, um, I'm here to talk about the LLMs misleading people and all the time convincing them, "Hey, I did a search. Hey, I did this," uh, while it didn't.

4. 0:40

   Why? Because LLMs are programmed to please people, please users, so they making up things, and this is the biggest issue right now I'm seeing with LLMs, and I'm building applications all the time.

5. 0:50

   I would rather LLM tell me, "No, I can't," but it never does. It always tries to make things up. So, um, currently, the web is actually fighting the robots and automations, and it's been something that's gone on for years.

6. 1:02

   Everybody knows CAPTCHAs. The first CAPTCHAs showed up year 10, like, you know, a decade ago, and it just keeps growing. And now you have AI blocking AI, and there's a whole war going on, and the web access is actually not as simple as it looks.

7. 1:17

   So they're getting CAPTCHAs, and, uh, they don't actually report CAPTCHAs. So it tries to go and find the data a different way. Sometimes it goes into, uh, training data, and this is where the worst thing is, when it uses training data and tells you that this is the current situation.

8. 1:34

   Training data is from twenty twenty-four, we're in twenty twenty-six, and it doesn't add up, right? So these are some of the new things that I was just literally checking out, right?

9. 1:44

   So Cloudflare blocks AI crawling for about twenty percent of the web, right? So twenty percent of the web is literally not accessible by AI, by default fetch that's built into it.

10. 1:53

    And Cloudflare also right now released an AI Labyrinth to actually trap bots and mislead them and provide to them fake data, so then your results are getting even worse.

11. 2:04

    Okay? So the invisible failure route, right? There's no error, no warning, just the wrong answer, right? So the agent sends the request, it gets a CAPTCHA. Uh, even an empty page, it doesn't tell you, "Hey, I got an empty page," right?

12. 2:19

    It will try to make up something, and this is where the most of hallucinations come from, the need to please and lack of data. So it literally makes things up.

13. 2:27

    I've seen it literally make numbers up, provide fake citations. You click on the citation, it's a four O four, the page doesn't exist, and I'm sure all of you have, uh, seen that happening recently.

14. 2:38

    I mean, literally, like sixty percent of citations on ChatGPT is not working. How many of you have tried to pro- purchase a product, "Hey, find me this product online, I wanna buy it, and give me a link to the product."

15. 2:48

    You click on the link to the product and there's no product. Like, so what is this product that you're talking about? The URL doesn't exist. The product doesn't exist.

16. 2:56

    So where do I buy this product for the fifty bucks? It doesn't exist.

17. 3:01

    Um, so what I wanna show you, and I'm gonna show you in code, how many of you are familiar with coding, like, um, VS Code? No.

18. 3:09

    Okay, perfect. So nobody's gonna get lost if I'm gonna switch to that. So I'm gonna do a demo basically with MCP and without the Bright Data MCP, and we're gonna compare what is going on there.

19. 3:22

    So first I'm gonna-- I, I, I wanna show you that I have exactly identical prompts for both of the, uh, scripts, right? So without MCP and with MCP. So the, the...

20. 3:33

    I, I give it five tasks. Uh, property, rightmove.co. Let's go check out some of the properties. LinkedIn, Instagram, Amazon, and TikTok. These are the five basic, um,

21. 3:44

    uh, sites that I wanted to access. They are very heavy on anti-bot systems, and I wanna show you the difference. So first I'm gonna run it without an MCP, and I'm gonna literally to, to, let the AI talk f- for itself.

22. 3:56

    And GPT 5's a bit slow, so it's gonna take some time. But basically, what we're trying to do is we're trying to access this URL. It's, uh, some local properties that I did literally half an hour ago.

23. 4:08

    Um, LinkedIn company in Israel. Let's check it out. Instagram account, uh, some Amazon product, and some TikTok.

24. 4:17

    This is not limited to just w- these five websites. It's just something that I picked that is-- I know for sure will not work without MCP. So as you can see, without MCP, I don't have live web data access.

25. 4:31

    Uh, it doesn't have any browsing tools, right? So this is with tools not available, just by default, out of the box, um,

26. 4:38

    um, GPT 5. I mean, it's a strong bo-- uh, it's a strong, uh, LLM, and zero success, five failed. Same exact thing, exactly the same prompt,

27. 4:52

    I'm running with our MCP. Uh, our MCP has sixty-six tools. While it's running, I just wanna go over some-- the tools that it has. Search engine. Search engine is basically the, uh, LLM is able to do Google search, Bing search, DuckDuckGo search.

28. 5:09

    Real searches, not just like, you know, search the, the web that it's di- doing in the background. Um, it has...

29. 5:21

    It also has scrape as a markdown. That's a very strong one. Basically, it can send a curl to any URL and get just the markdown without the HTML tags.

30. 5:30

    So you're now wasting tokens on parsing the HTML.

31. 5:34

    Uh, search engine batch, if you wanna do like, you know, like a hundred keywords, you can literally se- it can literally send a hundred keywords and get the hundred keyword back results like, so the scaling is also huge.

32. 5:44

    Discover. It also has pre-built APIs for many websites, as you can see. And of course, it has a scraping browser infrastructure. So it's a remote browser that your LLM can open and navigate.

33. 5:55

    The remote browser solves CAPTCHA by itself. It has a unique fingerprint. So it can open a hundred browsers, navigate the same website without getting blocked. So the whole idea is that with our MCP, not only can it do single sessions, it can do multiple sessions in parallel.

34. 6:11

    And as you can see, let's see, we have success for the Rightmove, we have success for the LinkedIn Mistral, uh, Instagram also worked. Uh, Amazon product, it has the information for the product itself.

35. 6:26

    Uh, and then what I did is on the second part, I asked the LLM to compare the results from no MCP with MCP, so that you don't take my word for it, let's see what the ChatGPT will actually say,

36. 6:39

    if it didn't get stuck. It looks like it's stuck for some reason.

37. 6:49

    May I ask a question?

38. 6:49

    Of course. Please, I, I love questions.

39. 6:52

    For social profiles, do you have, like, uh, accounts that you run?

40. 6:57

    No. We only work with public data.

41. 7:00

    Public data.

42. 7:00

    Only publicly available data. Um, collecting data behind login is not really legal. Why? Because when you s- sign up and create an account, you accept terms and conditions. When you accept terms and conditions, you need to really check it if it says, "Can you scrape?"

43. 7:14

    Can you... Do you allow... Do they allow robots to access the website? And that's why m- maybe some of you heard there's a lot of lawsuits going on, LinkedIn suing these people.

44. 7:23

    Everybody's suing each other.

45. 7:24

    Amazon.

46. 7:25

    Amazon suing, yes. But I think even, yeah, for example, LinkedIn and Instagram, I think they don't even show you really any public data, even if you're on the- There's plenty of public data for, for LinkedIn, of course.

47. 7:37

    If you take, for example, I will take this URL. I might get blocked. I don't know how is the local IP, but, um, and I open an incognito window, right?

48. 7:47

    I mean, usually it just shows me, like, really- There you go. A few. So this is a public data that can be collected.

49. 7:56

    So if you put a person instead of a company name-

50. 8:00

    Company, it's the same thing. It's the same thing. The only thing is they're very critical, right? So if you are using, let's say, a Wi-Fi, uh, IP of a big event like this, it probably will block you because it's a data center IP, it's low-quality IP.

51. 8:12

    From home, you could probably access maybe 5, 10 profiles, but eventually it will also ask you to log in. Uh, so we only deal with public data. So, uh, um, when you guys are using us, you are safe in the sense of nobody's gonna come knocking on your doors to sue you. [laughs]

52. 8:28

    If that makes sense.

53. 8:29

    Can we, can we provide credentials to our own social accounts?

54. 8:31

    No. We... Again, we don't deal with data behind login, uh, we consider it to be illegal, so, um, we don't deal with accepting terms and conditions and data behind login.

55. 8:41

    Only publicly available.

56. 8:43

    So I imagine you would cache some of these so that you don't have to constantly go and request it. Do you have an idea of how-

57. 8:49

    We have a whole data set. So if you guys don't want to do live data, and you don't care if it's a few months old, we have data sets literally that you can just filter by, let's say that you're looking...

58. 8:59

    If we're talking about LinkedIn people, you're looking for AI engineers in a certain area, you can filter it and just get the data set right away.

59. 9:06

    Okay.

60. 9:06

    So, and, and your agent will actually have access to that. So your agent can filter the data set and get you the data if we want to, right? So it has access to all the tools.

61. 9:16

    So here's basically head-to-head comparison by the LLM itself. Without MCP, failed, no live web access. With, it's listed, failed, successful, failed, successful, so that's basically it. Uh, anti-bot bypass, CAPTCHA solving.

62. 9:30

    Right, so again, our system automatically solves CAPTCHAs, so if your bot navigates to a website with a browser and it has a CAPTCHA, our s- our browser has built-in CAPTCHA-solving solution, so it will automatically solve the CAPTCHA, and your bot can continue browsing without getting blocked.

63. 9:44

    How much time I got? I don't know. Um, so just to kind of summarize it, right?

64. 9:51

    This is the biggest hallucinations that you guys see. The agent gets blocked, it needs to please you, and it makes things up. And, um,

65. 10:01

    fake content also, right? So now, uh, if you wanna Google what is Cl- uh, Cloudflare AI Labyrinth, it's basically a system, once it detects a bot, it's not... doesn't block it, it literally feeds it fake data.

66. 10:14

    So bigger hallucinations, right? And, um, the easiest fix for this is just to make sure that your agent doesn't get blocked, and it's as easy as to implement our MCP.

67. 10:26

    Um, our MCP has a free tier of 5,000 requests. So if you guys wanna try it out, um, this is... You can connect with me on LinkedIn if you want to.

68. 10:35

    Or is this the... Hold on a second. Is this the sign-up for the MCP? I'm lost in these QR codes.

69. 10:42

    One second. Internet. Yes, please.

70. 10:51

    Uh, how does it detect if, if there's Cloudflare Labyrinth or not?

71. 10:55

    Um, so the way we approach it is that we make your agent look like a human being, literally like, uh, there's mouse movement pre-recorded, there's typing. When it types, it's like it's, uh, mimics a real human behavior.

72. 11:08

    So, uh, the Cloudflare literally just doesn't even ask, are you a robot or not, right? So this is our, our approach. Instead of trying to s- understand how they detect, we make the agent look as human as possible so that it doesn't trigger the actual, uh, blockage.

73. 11:25

    Misleading data is one of the toughest things that you can actually encounter. A lot of websites right now, and Asia is doing that, right? Hotels, they're literally providing different prices.

74. 11:34

    You go check out on your phone, you get one price. You can check from your computer, you get a different price. You connect through a proxy, you get a third price.

75. 11:41

    Which one is correct? It's really hard to tell. When it comes into the domain of misleading, uh, the best bet is to make sure that you, uh, y- your agent looks like a human, and hope for the best.

76. 11:52

    That, that's basically what the approach is right now. Um, AI Labyrinth was literally released a few... like, a month ago. Uh, I don't have much of statistics on what is, like, you know, exactly how it's working.

77. 12:06

    All I know is that- It didn't really affect us. We don't see any kind of change in data, and we are collecting petabytes of data on the daily. Like, we have so many customers always scraping.

78. 12:15

    We're caching data, so we're always comparing the results. We don't see any degradation in the results, uh, so I think we're doing a good job in that case. Uh, this is a QR code that you guys can sign up for.

79. 12:26

    Uh, we have a GitHub page, uh, as well, uh, githubbrightdata.com,

80. 12:34

    or githubbrightdata. And another thing that I would recommend for you guys to check out is the Skills page, right? So what we did is we created skills, and I'm gonna have another session in a couple of hours if you're interested in seeing what the Skills does, is basically you can take any agent, tell it to go here,

81. 12:54

    and w- this page will teach it on how to build a scraper or how to build a pipeline that will collect you the data. It has all the information it needs, all the APIs, and, uh, if you will come back to my next session, which is at 1:00, I think, something like that, I'm gonna literally demonstrate how

82. 13:15

    it builds the pipeline. Literally, in front of you, I'm gonna tell it, "Hey, listen, let's build a Walmart collector for ABC," and it's gonna build it, and it's gonna scrape it, and instead of parsing each individual in HTML, it's gonna build a parser, and it saves about 99% of the tokens.

83. 13:28

    'Cause I see a lot of people as like, "Hey, I need to parse 10,000 pages, but it's so token heavy." Don't parse with the LLM. LLM builds the parser, and then the script runs it, but that's the next session.

84. 13:40

    Any questions?

85. 13:41

    Um, question on the performance. Uh, I saw you MCP exposes 69 tools, which means if I s- if I need, let's say, search capability for my information, do I need to load all the 69 tools?

86. 13:52

    No. Of course, filter it. Yeah, I just showed 69 tools because just to show it. Uh, if I need just a scrape markdown, a search, I would just literally load two tools.

87. 14:01

    Okay.

88. 14:01

    Uh, otherwise, you're flooding context with irrelevant data. Of course not.

89. 14:07

    The, uh, experiment at the beginning. Does that use the web search tool?

90. 14:12

    I'm sorry. I didn't hear the beginning. Again?

91. 14:14

    Uh, the, the experiment at the beginning, uh, like your MCP

92. 14:18

    Yes.

93. 14:18

    ... used the web, web search tool through the OpenAI API, or is that just like a call to adaptive data? I'm sorry, like, how does that compare? I've not had experience with doing this before, but

94. 14:31

    Um, so we didn't do any searches. Um, I literally told it, "Hey, go to this URL. See if you can load it," right? So the-

95. 14:38

    Sorry.

96. 14:38

    I didn't use the search, uh, in this demo.

97. 14:41

    Sure.

98. 14:42

    Uh, but of course, again, even with our MCP, it can actually do a Google search.

99. 14:47

    Yeah.

100. 14:48

     And that's the... One of the biggest benefits is because we're used to Google results. So by default, when you're asking LLM, you expect it to do a Google search, but it doesn't.

101. 14:58

     So the results with, uh, MCP much, much better, and I, I recommend, sign up, try it out. It's free. See the results, compare what you guys get.

102. 15:07

     Is that 5,000 requests per day or?

103. 15:09

     Per month. Per month, which is, you know, like, for an MVP, for a little experiment, that's more than enough. Uh, we also have pay-as-you-go, so if you do need a little more, it's, it's nothing like, you know, it's...

104. 15:20

     Okay. [laughs] I mean, just for a prototype, just for banging around.

105. 15:23

     Yeah, yeah, it's for prototype, it's perfect. I always... I do a lot of hackathons, and I'm always recommending, "Hey, listen, set up an MCP and tell your agent to go build whatever you need."

106. 15:31

     It does a much better job than without. [outro music]