Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data

0:13 All right, let's begin. So, nice to meet everyone. My name is Johan Raju. I'm a solutions architect here at Bright Data. I've been with the company for about six years, and I've been working in this

0:24 space of data collection and web scraping. We initially started off as a proxy infrastructure company uh with the largest residential IP network in the world and then we moved into the data

0:35 collection platform. Now we're the world's largest web data collection platform collecting pabytes of data per month only second to Google. So just a little information for you guys on you

0:47 know where we started and and how how we moved over to today. So I'm going to be covering two topics. First is I'm going to be speaking about building AI agents with real-time web data. Some tools that

0:59 people are generally using when it comes to you know AI agents versus depending on your rag model with pre-index data and what are the advantages what we see in the industry and you know what

1:10 changes uh are to come and what it takes to uh run these agents. And then finally, I'm going to have my hands-on exercise that you guys can follow along with me. Uh, where I have a product

1:21 intelligence agent that will essentially use a product from Amazon, get sentiment on Reddit, and then provide you analysis with, you know, should I buy that product? Yes or no.

1:34 So working in the scraping space for a long period of time, one thing I've noticed is, you know, when you think about building scraping or collecting data,

1:47 what does it really take? Right now, the early customers in the scraping space that I've worked with, you know, when when you first start, it's always about, hey, I want these are my target sites

1:57 and this is what I want to scrape, right? You know, people usually start off with low volume. you know, that's what comprises a general traditional scraping archite architecture. Now,

2:07 let's say I'm an e-commerce uh user and I want to collect e-commerce data. You know, my biggest targets might be Amazon and Walmart because, you know, I can find a lot of products on there and I

2:16 can get price intelligence and data on reviews and, you know, all the general information for a vast majority of products. Now, when does scraping become challenging? And this is a change that

2:29 I've actually seen with customers that I work with today um occurring in the industry when it's not just about Amazon and Walmart anymore. Now you want Shopppee. Now you want Shopify stores.

2:41 Now you want um Etsy and you know a bunch of other sites. Now if we take a step back and I mentioned you know you know your target sites in order to build scrapers around your target sites it

2:53 takes generally about you know 3 to 5 days to develop a scraper if you're a developer and that's not only developing it if it's QAing it running it in development testing it out and you know

3:04 the whole endto-end pipeline now if we think about you know 10 sites okay might take a bit of time now you think okay I want to add a 100 sites a thousand sites this is where things start getting

3:15 complicated. Now before AI even came into the picture, we would developers would have to manually understand the site, you know, understand where the selectors are located. Can I target

3:26 these data points in the API and actually build crawlers around that knowledge and I've worked with many people, you know, that's that have been scraping for 10 plus years that have

3:37 been managing and maintaining these crawlers. But when it comes to scaling up, then you're thinking, okay, like now I'm adding more sites to what I already am scraping, but I also need to manage

3:47 the scrapers that I already have. So let's say Amazon makes a change tomorrow and two other sites make a change tomorrow, but you're managing a thousand sites. This requires a lot more

3:57 engineering resources and a lot more time for you to actually figure out, okay, what change was made? How can I count counteract it? How can I fix it? And is this actually impacting my

4:07 customers, my SLA? and my data quality. Now before we had this bottleneck of having to scale up your resources and when at those times when you would have to manage the thousand sites you'd need

4:20 you know two times more engineers, four times more engineers etc. But now in this AI space, none of that is needed because we have things like agents and agents can actually go out and discover

4:30 and build sites, build scrapers for these sites a lot easier than, you know, team of developers and engineers. Now, this is one classic example of an e-commerce site of targeting selectors

4:44 on the page. So, you can see here we're just in this example, it's just targeting price and title. Now, what happens if the price has changed? the selector for the prices changed, you

4:54 know, tomorrow. That will require some downtime on your end because, you know, if you're the one scraping the site because, you know, now you're not getting price and that can affect the

5:03 rest of your pipeline that you have to maybe analyze products, you know, etc. Now, usually that require like a team to go in, fix it, then maybe back fill on the data that you know you've lost. But

5:15 now with AI agents that can be automated and done automatically. So when you also build scrapers, one thing to always know is every website is different. Now sites in the

5:30 example I use like Amazon and Walmart, they have two different layouts even though they both might have things like price, title, description. So each site requires a unique scraper to build build

5:42 for and the even more important than that is each site wants to protect their data. Now sites usually like to ensure that no one else is uh you know has enough like the same product

5:57 intelligence they have or you know depending on what type of data that they're you know trying to keep secure they will always implement different types of anti-bot and different types of

6:06 capture to throw you know scrapers and bots off from collecting that data. Now I've seen all different types of blocks. I've seen, you know, the most basic sites that you can get through

6:17 with data center. And then I've seen sites that, you know, are protected by Cloudflare, Okami, you know, you name it, that you actually need to have a residential IP address. You need to have

6:26 a real browser. And you need to seem like a real user. You know, if you click too fast on the page, if you scroll too fast, that could all trigger anti-bot if you're blocking images, things like

6:35 that. Now, like I mentioned before, engineering has been a bottleneck until the AI space came here when we we had the AI agents ready like you know able

6:45 to build crawlers and you'd have to actually have a deep understanding of the site you know um a lot of sites like I mentioned because the layout is very different it means that they'll try to

6:56 make sure the data is as hard as possible to collect for you and you would have to have a deep level of domain knowledge expertise to get to that data. data.

7:07 Now this is a diagram of how the traditional scraping architecture look like. So just to briefly go over this we have on the left hand side the you know the a the actor essentially what's

7:20 triggering this crawl. So some people like using like schedulers you know um cron jobs you name it where they want the data fed to them daily weekly etc. And that kicks off the job directly over

7:32 to the next layer. Some people like doing, you know, specific config, specific URLs. Same uh same idea right there. It goes and that goes right over to the proxy layer. Now, the proxy is

7:44 essentially how you mask yourself from being detected. There's different types of proxies that you can use. you know, some static IP addresses like data center ISP, some rotating residential IP

7:57 addresses that come from like laptops, tablets, TVs that are more authentic to sites because they can say, "Okay, this is registered to an actual device, we'll let them in." And then some tools that

8:07 have anti-bot capabilities built in like what bright data offers web unlocker or browser API which is in the browser automation side of of of it where the the actual product itself manages okay

8:20 is my request getting blocked am I going to retry am I using the right IP that's all in a part of the layer when you think about the anti-bot side of things now from there then the next layer is

8:32 you know parsing the data and actually validating if the data is, you know, authentic. That's a big part of, you know, after you actually get to the site and collect the data, am I getting price

8:43 back 90% of the time and do I want to validate that I'm getting it back 90% of the time? Do I need review data? Do I need ratings? You know, things like that. You have to think about, okay,

8:54 like how do I ensure that I'm not going to be setting up when the data goes through my pipeline that it's going to be checked accurately by validation rules. Then we go into the

9:05 post-processing side of like the ingestion layer. So you know dduping records enriching your data pro processing your batches that is once you you know some people like delivering it

9:16 to cloud stoages some people like delivering it to their internal systems once it gets there you know what are you doing with the data how do you ensure that you're not providing the same

9:23 record twice or you know unclean data. So you know moving it from in the medallion architecture bronze silver and then finally moving it over to your database and if anything fails in that

9:37 process then you need to start the whole uh process over and this is where I was mentioning you know if something breaks in the antibbots layer or if it breaks and you know selector has changed you

9:48 need to start the whole process over sometimes backfilling you know maybe you might miss SLAs's and things like that And so here I'm going to be talking about now specifically AI. Now how many

10:01 of you guys today are using things like you know claw GPT any of the uh actual like AI brains? Okay, cool. How many of you guys have entered a query where it's it's you see

10:13 it says thinking or looking for an answer or searching the web? Okay, cool. Nice. So this is where you see and I've split it up into two where the left hand side is where you see okay

10:27 when the AI is actually thinking or searching the web that's where it decides okay the query that you gave me I actually need to go out and find the answer to this I can't use my

10:38 pre-trained data so some prompts like I'll give a basic example when's the best time to travel to San Francisco so if I say that sometimes the the uh AI brain will usually go over to the rag

10:52 side, the async where it'll be used to pre-train data that you know, hey, maybe the best time to travel is during the summer, and it'll actually provide that back to you. Now, if I change my query

11:02 and say, when's the best time to travel this year to San Francisco, then the AI the AI has to actually go out into the web and see, okay, like do I need to look at what's going on? You know, is my

11:15 user asking me for events that are going on? you know, how do I like I need to make a decision this year real time to find out when I can recommend for them to travel. So, these are two different

11:26 paths that um each a the each AI can actually make and provide you an output with. Some some things I noticed is like responses sometimes can be cached. Like you know if someone's asking what's the

11:40 weather today in in New York City like a few times a day that might be a good response to cash because you only need one person to really ask that and you can you know what the weather is going

11:51 to be for the day and you can just provide it, you know, so you don't have to burn through tokens or you know um uh like money to actually go out in the web and and actually find that data.

12:05 Now like I mentioned there's like two paths to uh gather this intelligence. The sync path and then the async path. Both are advantageous for different cases.

12:16 The sync path is you know if you really actually want to go out on the web and actually find what you're looking for and async is if you know it's okay to rely on pre-trained data. And both

12:28 require different sort of setups. And we also do offer a tool which I'm going to cover in the next slide um where you can actually use leverage Bright Data's tools to actually go out into the web

12:39 and collect data. Now one thing I actually noticed is if you go into any website and if any of you guys are familiar with like robots.txt, a lot of sites usually block things like

12:50 Claudebot and GPTbot. So if you ask these engines these questions that are actually going out to the web and trying to find the data, they might not be able to get those data from those specific

13:01 sites because they're actually blocking those you know GPT or claude from actually accessing the data. Now that's where we sort of come in and because what our advantageous is advantage is

13:12 we're able to actually unlock the web. You know whether it's if you have a scraper or if you have an agent or you're using um you know something like cloud or GPT we have tools that will

13:24 allow it to actually go out into the web without actually getting blocked and scraping data. And we've been masters at this for for a long time now. I'm going to be talking about the the web model

13:37 context protocol. That's what MCP means. So it usually integrates with something like claude code, lang chain, you know, name it like you could think of it as like a brain that it needs to speak to

13:50 and that brain essentially connects to our MCP endpoint which is essentially 60 plus APIs that we control in our library. And some examples you can see here are you know search engine. So, you

14:03 know, Google results, scraping pages as markdown files, and scraping pages as HTML. Now, if I want to target specific sites like, you know, Amazon, Reddit, LinkedIn, Instagram, that's all included

14:17 as well. I actually, as an example, I I worked with someone two weeks ago that, you know, was trying to build a social media listener with the MCP, and one use case they said was, "Hey, um, this is my

14:31 name. I want to go out into the web and collect me all of my social media profiles and return me how many followers I have. So what this did was he was using it on cloud code. It made a

14:42 call to our MCP using the search engine first found the results his pages on Instagram, Tik Tok, LinkedIn, brought that back, made the API request to our data pipeline and then brought back the

14:57 results on how many followers you have and gave them the context back. So that's the power of you know what this can do and how much of this the web that the MCP can unlock. And also this works

15:07 for browser automations. Um so you know creating things that you want to use a browser for like clicking scrolling you know maybe door dash like automating an order or um automating you know on a

15:20 travel site trying to find you know what's the best times to buy a fight a flight book a flight. Um, so things like that it's very advantageous for. And I'm going to show you a little video on how

15:33 the MCP works. Let me bring. So on the left hand side you can see this is clawed with bright data's MCP setup and on the right side this is

15:44 without bright data's MCP setup on claude. So you can see it's the same prompt being asked and it's use your browser to get the contents of this page which is a

15:55 glass door URL and on the left side you can see with bright data as MCP it actually went over to glass door and got the results but on the right hand side it actually got

16:07 blocked from going into glass door. So this is one perfect example of you know sites like Glaster I've worked with on a on a data collection level that's like a very protected site and you know this is

16:20 one great example on why the MCP is advantageous you know pairing it up with something like claw or you know GPT or any sort of brains that you know you'd want to run it on.

16:35 Okay. So now I'm going to be speaking about an example use case. So working with AI companies, these are examples of you know customers that I speak with that use bright data in any

16:50 form of way. So you know training models um fine-tune enriching their uh enriching their models or fine-tuning their models um providing AI agents real-time access like like we mentioned

17:03 or enriching rags. So in this example, I'm going to be speaking about specifically jobs, right? So let's say someone came to me and said, "Hey, listen, I I need job data and I want to

17:14 train a model with jobs data like maybe resume matching, you know, as an example." So one thing we need to know is when you're

17:24 thinking about jobs data, it's not like other, you know, collecting other forms of data, you know, where it's I can pull in the data and then I can just forget about it or train a model. You need

17:33 continuous updates about jobs. You know, which companies are posting what, you know, how many jobs are available, what's the salary, you know, location, etc. So, this is why it's important that

17:43 the data is actually structured and has a consistent output. The actual performance is very key as well because you know if you're building a model to analyze like job patterns or

17:54 you know anything in the job market you need that data flowing to you consistently and you need also need this to be very reliable because if something breaks or

18:04 you know if there's downtime then that can actually affect the actual in the entire pipeline that you're you want to run and the timeline for this you know it's

18:14 not only about getting data starting from today about jobs throughout. Some people need jobs data previously, you know, like the past few months or past few years. So that's why it's very

18:25 important that you know, you're covering all of these because if you want to build a big picture on, you know, in this example for jobs on, you know, how do I find train my model on jobs data so

18:36 I can find the best matches for this position. You need to understand how the trends changed. you know maybe salary has increased for software engineers over time and you need to understand why

18:46 what the demand is what's the current you know offering and you know you can you build your model around that so this is like the example use case like an AI agent for resume matching

18:58 so it's exactly what we discussed here um so immediately you need real-time access to actually go through and um you know scrape the data and be able to train this model along with it.

19:14 Okay. So, I'm going to be covering a hands-on exercise here. So, I have a Google notebook here. And if any of you guys are following along, there's two things that you require to

19:29 actually run this script. One is a bright data API token and one is a Gemini API key. And I have the list of instructions in the Google Drive on how to set up both. Um, and essentially I

19:43 have everything set up here for you to run. So I'll speak a bit about what we're doing here. So first we're calling Bright Data's API endpoint to collect Amazon data.

19:53 We're collecting the response from Amazon. Then calling the API for Reddit itself, capturing the raw Reddit data response and then performing AI analysis over

20:04 that. Now before I run this, I want to show you the exact data that we're getting from each of these APIs. So Amazon, we're getting product level data. So I

20:17 picked a Tumblr cup as an example and essentially we have every single data that about the product. So you know price, images, URLs, etc. And I I'll show you this once

20:33 it runs as well. My screen's a little small, so it might not show the entire thing, but and we also have a Reddit API where what I'm going to do in this uh script that I built is I'll actually run

20:47 a keyword search for that specific Tumblr cup. And I'm going to be outputting posts that, you know, talk about what what what's exactly, you know, is should I

21:01 buy this or not? Is this worth it? Is this not worth it? What are people saying? So, let's run this live. All right. Right. So, we got our keys loaded here.

21:41 All right. So, we got our Amazon data that came back. You can see it has price, rating, you know, other sellers that are selling it,

21:52 all of the actual data from the product itself, features, you know, you name it. Now, we'll go ahead and trigger a Reddit search for the Stanley Quenter um Tumblr. Just

22:08 having, you know, reviews about the actual product itself. And while this is running, I will show you a bit about what I spoke about the model context protocol, the MCP server.

22:41 So we have different layers. I know I talked about things like you know browser automation and search engine but we also do cover things like you know e-commerce, social media, financial

22:53 intelligence, research, business intelligence. So these are can all be plugged into an MCP endpoint you know app store travels and we are still

23:06 adding to it. Currently we have about like 60 plus tools available to use for AI agents. Um, some people prefer actually using scrapers themselves, but we do this will provide allow your AI

23:18 agent to choose which tools to use. So, it'll be a lot more streamlined and easier While this is running, I can take any questions if anyone has.

24:05 Yeah. So the question was how come you guys can scrape resume level data but you know bots can't do it right. So what what gives bright data the advantage and this is why we're able to do it whereas

24:18 like you know things like bots aren't able to is because we have the world's largest infrastructure like proxy infrastructure and where you're talking about you know a few requests maybe like

24:27 you know maybe a few days worth of scraping pages like you know glass door as I use as an example might work right but let's say after a while glass door catches on like hey you know he's using

24:39 a data center IP he's accessing the page you nothing's actually being done on the page and then they start adding all these layers of anti-bot into the picture. That's when companies like us

24:50 bright data would come into the play because you need to actually not only own like a scraper or a bot, but you need to think about how do I bypass you know what the site is protected with the

25:00 capture and how do I do it consistently without triggering the block. So, we've been doing it for years and we have tools that bake into this anti-bot logic where we're changing our approach to a

25:13 site every time at scale, but sometimes um comp like people that are starting off scraping might be doing it in-house and managing it. You have to also think that's also engineering time and

25:23 resource time because trying to figure out what change did that site make also takes time, even probably more time than managing the scraper. Okay, cool. Looks like it came back with

25:38 data here. So, I can show you. So, it h it c captured a few Reddit posts about the actual Tumblr itself. So, some comments about, you know, what people were saying about

25:52 the actual Tumblr. It looks like people were saying it was embarrassing and people were obsessed, but and then it gives the actual response here. So you could see

26:04 actually buy it. It's on a decent discount. You know, there's overwhelmingly positive reviews, but a few minor mixed opinions. There's high demand, but low stock on Amazon because

26:15 it returned that from the data. And you know, a few other analysis points about the actual Tumblr and the amount of reviews it has. It actually dissected all the data points that we

26:26 returned from our API and gave you a decision on yes, should I buy it? Should I not buy it? Now this is just one example of you know using data from starting from Amazon running it on

26:36 Reddit and then making a decision but you can think of it it think of this on like a bigger picture you know you can it doesn't have to be e-commerce it can be any sort of use case you know as long

26:45 as we have like we have the tools that can allow you to go to the page and access the data. Now it's up to you what logic and how you want to build that pipeline and build it at scale as well

26:56 because what one thing we do best is actually provide the data at scale. >> Cool. I think we're uh able to wrap up and if you guys um want to c if you have questions after I'll be available or you

27:08 could stop by at the Bright Data booth. But thank you for your time guys. >> [music]