How to Build Financial Agents That Work Across the Web - Jay Sahnan & Arik Bird, Browserbase

Hello. Hello, everyone. Um, we're here to talk about how to build financial agents that work across the entire web. I hope you've been having a great day so far in New York. Um, I am Jay.

And I'm Arik. And I think the place to start is what even is a browser agent, right? So if we think about it... And first, if we wanna go back to the previous slide, quick show of hands, who is building agents today? Show of hands. Okay, we're in the right place.

And who works on like finance operations? I heard about some conference going on today. Uh, anyone in finance? Okay, a couple, more engineers. Um, if you also wanna ask us any questions, uh, throughout, you can scan this, um, QR code, and you'll be able to ask away.

Cool. Let's back to the show. So what is a browser agent? And a browser agent, if you think about it, if an agent needs to access the web, interact with the web, read information from the web, take action on the web, it needs a browser. Whether it is booking a plane ticket, booking a flight, planning a date night for you and your wife, getting some pricing information on a competitor, it typically needs a browser to be able to access those hardest to reach sites that don't have an API or don't have an MCP. Um, and so in a

nutshell, anything that you need to do on the web, your agent should have the tool called the browser.

Show of hands, who knows these three companies: Ramp, BILL, Stripe? They familiar with the crowd? Cool. When you think about what these three companies have in common, well, first, they're all financial institutions. They're dealing with highly sensitive information. But when you think about it for like Ramp, BILL, and Stripe, a lot of this information lives on the hardest to reach parts of the web, whether that is behind a portal where you need to log in, whether it's on a website that is heavily protected by anti-bot. It is extremely challenging to get

access to this information, and the answer is using browser agents. So if we wanna start off, we'll go with Ramp first. So Ramp, I'm sure everyone in this room is familiar with Ramp. We actually are big users and fans of Ramp. Uh, they make it incredibly easy to, for a company to manage spend. When they were growing and scaling, they started having to interact with hardest to reach portals and more portals than, than ever before. And so the problem or the gap was: How can

we fetch these receipts efficiently at scale? They didn't know about the browser agent, so instead they're like, "Let's just put together a team of humans, and they will log onto these portals, and they'll go fetch these receipts." Well, that can s- that can be good for a series A, series B company, but when you start growing and scaling to the, to the masses that, uh, Ramp is at, you need a more efficient way. You can't continue to hire humans. You're just gonna be bottlenecked by the limitation of scale. So in comes browser

agents. And now, again, we are big users of Ramp. I booked my flight to this conference using my Ramp card. Immediately after booking my flight on that Ramp card, Ramp spins up a browser agent. That browser agent goes to Delta, it logs in, it captures that receipt, and it matches it to the transaction. Um, so once they do that, that's, that's all good and dandy, but when you're doing it at scale, you're gonna run into things like anti-bot. So how do you reduce that friction? How do you

eliminate getting blocked?

And to understand this, I really want to go back to: How does the browser work on your laptop? Think about Chromium or Safari or Deer. Chromium allows you to access the web. It's an executable binary. And right now from this slide, we can see that we are booking a flight to New York. And in the background, your browser has an entire fingerprint. It knows

your browser version, uh, there's OS, there's fonts, there's screen size, and even time zone. The website also sees a familiar visitor. Maybe you visited the site before. Maybe you've got some cookies. You've logged in, or you've booked too many trips on the same account. There's a familiar pattern as well. Each request will go through a net-- You've got a network reputation and all the navigation. You are clearly a human. And now the problem comes when you try and put that

same ver-version of Chromium into a data center, and that's when we scale beyond the laptop. We need to replicate it for companies like Ramp. They are not gonna have thousands of laptops open running a different human version of Chromium. They're not going to have one person with a cron job running this either. They need scaled infra that lets them get to sites reliably. The problems start to appear, though, when you put that same Chromium in the data center.

Um, let's say we do this. Let's say we've put that in. The problem now is when we launch a cooled up browser, that headless browser in, in an EC2 instance somewhere, every single time we launch it, it's gonna have a different fingerprint. It's gonna have fresh cookies. You may need to log in again. And proxies. It's gonna clear indicator that you are coming from a data center instead of a residential IP. And sure, you

could trick the website by using a proxy service and changing your proxy. And what that means is you might not be presented as many CAPTCHAs, or you can fake your location from that data center. But that still doesn't change the underlying problems with Chromium. You're using that same Chromium on a laptop.

And if we just step one step down and drill a little bit deeper and think about what makes Chromium so good is if you look at the browser process, right? Inside you have a page engine, this is a very simplified version, and where it renders the content, and it has all the HTML, JavaScript. Um, then you have your network, and if you ever, like, inspected before, you go on the Network tab, and you can see all the requests coming in and out. And you have your device and graphics. The engine is so

great that they can make a change every two weeks, which makes it terrible for us as a company trying to automate the web.

So we forked it. And w-- I could go on forever on how many changes we made to the Chromium binary. Um, but we forked it and came out with a verified browser that we run in the data center. And what we did with three things that I'm gonna share now is we first removed the automation flag. An obvious one, right? When you're driving a headless browser, um, through an automation, you will pass this web driver equals to true, and we just swap that to false everywhere. Then we align on

the identity. Instead of now being called headless Chrome, we are calling it Chrome. Now, these may seem like simple fixes, be-because how complicated Chromium is, this goes deep into the code, and it's in so many different places that we need to find. And every two weeks there's a new change, that's why we have our own version. We also keep the process alive. What I mean by that is a lot of scraping jobs or automation jobs or workflows that people are doing as developers locally or even in their hosted infra will start and

stop. They kill the browser, and they kill, uh, Chromium instantly as soon as the task is done. Instead, we leave Chromium running and close the windows out.

So why don't we put it to the test? We've got a snippet here, and I'm gonna ask Arik, can we launch a browser?

Okay. And bear in mind the Wi-Fi has been terrible today, so it's launch is gonna take a second to render. And we are now looking at an iframe of a live view of a hosted browser on Browserbase. That's great. We can go to Ramp. Okay? Nothing special. Let's stop that, and let's go to the next slide. Why don't we launch a hundred browsers? Do the exact same thing, but we'll launch them concurrently.

Okay. And to the next slide. And now what's gonna happen in the background is I'm calling the same thing with a bunch of different websites, and you will be able to, Arik, scroll through the, um, some of the sites. And we're not automating at the moment, we're just going and visiting these sites. There's a hundred browsers. Now I'm gonna try something for the next minute. Um, I set this up very quickly, and the Wi-Fi is a little-- sucks a little bit. And it's all through an ngrok portal on my laptop. So if it

doesn't work, uh, we'll move on. Um, I want everyone to scan this QR code, and you'll be given a random website out of one of these hundred. Um, someone tell me if they can see a browser. I'm trying to prove to you that these are real browsers in our cloud, and you have access. Someone shout out when they-- when it's loaded.

Hopefully, the demo gods are with us.

Yeah, you got a browser? Awesome. Yeah? Got another one? Great. Cool. Sweet. Nice. You can actually interact with this as well. Um, through the iframe, you can click, control, move, and do things. And that takes me on to the next thing. I'm gonna, uh, stop that now. Thanks, Arik. Um, and let's take me on to the next thing is control. How do we control these browsers?

And just to sum this up, Ramp now can deploy thousands of browser agents. They have saved north of thirty thousand manual hours in data entry that is done-- that used to be done by humans, that is now done by agents, and their scale has gone up five X. So just by switching from humans to agents, they were able to accelerate their business and save a bunch of those hours, which now they can dedicate to other initiatives across Ramp. When we talk about BILL, so Ramp, scale, and access. Now it's about control. Everyone's familiar with

BILL. Um, BILL, you know, processes a ton of payments across thousands of different vendors. When you think about using agents to do that, the biggest challenge is, well, every portal... Next slide, please. Every portal is different. The checkout flow is different, the process to fill out the payment is different, the UI changes. So in reality, the way that BILL thinks about this is it's impossible to automate with agents. That's why we need a team of humans to go click through portals and do this on

behalf of our customers. Um, and again, the answer to this is an agent that can use a browser. Um, and so for BILL, the notion was: How can we build a single agent that can interact and act upon thousands of different portals? One agent, thousands of different portals.

And that brings us to Stagehand. Stagehand is our open source SDK that lets you control a browser. Think about Playwright, but built way on top of it. Your agent will sit and use Stagehand to control the browser. I'm gonna show you a little bit how. You can go to the next slide.

I want to start with understanding again what is wrong with Playwright. Playwright's great. I first learned how to code by using Selenium and Beautiful Soup, um, and writing Playwright scripts, um, by hand. This was before AI. Um, now I can't remember the last time I coded. Um, and so with Playwright, what the problem is, is as soon as that website changes, the button moves. The text box changes, like, from first name to just name. In the, in the

back, like, when you're looking for that XPath in Playwright, or you're trying to click on an element or a selector, your script's also gonna break And now that's fine if you're doing one-off tasks and you're running everything locally. But when you try to scale like a company like Ramp or BILL, you need reliability every single time. The reason we bought stage-- uh, like built Stagehand, uh, was because we wanted to give the ability to agents to navigate the web and control them. The next slide. So how does it actually know? What's different? Well,

if you-- Pl-pl-- Sorry. What's different? So Stagehand actually works a little bit like Playwright in some sense. You can use deterministic rules and go page.click or everything you would see in a Playwright script. But when it gets a little bit more complex, you can actually send natural language prompts to go and do actions on the site. Pay, um, page.click or page.act, go fill out this form, or enter my name, or enter password,

um, find the red button. Stagehand will then be able to take the DOM and accessibility tree and everything it sees on the website, pass it to an LLM, pass it to an agent, and understand where to click and go and click it. Now, you may be thinking it might get expensive, but that's where we have our excellent engineers behind Stagehand.

And to demonstrate this quickly, Arik, if you could hit that Compare button. If you were to use an agent with Playwright, you'd be sending everything on the left. That's one hundred and twenty-two K tokens, and that's a raw HTML. And you ask your agent, "Where do I click? Where do I go to?" What Stagehand does before it hits that agent is it shrinks the DOM to a really cool accessibility tree where we're only sending like four thousand tokens for this same example, and it'll give direct selectors. So Stagehand will go click that,

save a lot of money on, um, the amount of tokens that you're sending, but also reduce the context and get quicker results. And if we go next, another reason why we use, um, Stagehand and with Browserbase is because there's fewer net-- there's fewer round trips. Let's say we are currently controlling a browser in a data center, and we've got to talk back. We send the HTML back and forth every single time. What we do at Browserbase is we put Stagehand next to the browser. And that-- what

that means is Stagehand is a Chrome extension on a hosted browser, so those actions become a whole lot quicker, and there's no round-trip latency from those actions and those, like, instructions from the agent of where to click and where to go next. And so when you bring these two things together, you need scale from the browsers. Stagehand gives you the control, and we're gonna show you an example that use both.

This is our last example. So with Ramp, we saw that browser agents solved the issue of scale and access to heavily protected sites. With BILL, it was adaptive automation. How can browser agents enable you to interact with thousands of, of different portals with ha- without having to manage thousands of different Playwright scripts? Now with Stripe, everyone's familiar with Stripe in this room. Stripe onboards thousands, tens of thousands of merchants on a monthly, quarterly, quarterly, yearly basis.

And again, they knew that the only way to access the information to verify that these businesses were real and they were the right business to work with is through a browser. And this was before the age of browser agents. So they had a team of humans going on and going to Secretary of State portals and using the browser to get that information. So a quick switch is being able to use the browser agent to go and automate this KYC verification.

Um, let's dig a di- bit more into Stagehand here. And so we have, instead of using a direct agent like BILL, you could use act, extract and observe. Those are three key primitives that Stagehand, uh, we give-- we expose to you as a developer. Um, one is observe, find the Learn More button, and this will take an action. Next slide. act, click the Login button. Next slide. And then the extract. Now, this is my favorite one because you can extract anything

into a Zod schema, which is great for engineers because we're taking really messy, unstructured data on the web and converting it to really easy-to-use data and readable and send it to an agent. Next slide. Now, I'm gonna show a couple more demos. We're gonna start with one company website, and we're gonna go to Chipotle and find out what it sells using Stagehand. Um, and next slide then. Okay, so this browser will now spin up on a hosted browser in Browserbase, and it's gonna use Stagehand,

um, extract to extract it.

And there we go. We've got Chipotle Mexican Grill. Um, it's selling some lime chips. Cool. Uh, if we press stop.

Now, this is where it's gonna get fun. Let's go for the next one. And now we're gonna research four different sources for Chipotle. We're gonna go to their website, investor page, SEC filing, and New York Stock Exchange listing. Okay, so we're running, and Arik, you can also scroll once they load.

Okay, great. So we've got the SEC filing down at the bottom, and we've got New York Stock Exchange, and up top I think we've got the website, um, that will appear in a second. But there we go. They've got investor relations as well. So these are actually happening quicker than they are on the screen. This is already being happened in, in the back. It's just a live view that's streaming back to us. And obviously, when you're running so many browsers at scale, you're not always gonna be checking the live view. This is more for development. Um, okay, cool. Let's stop that. And let's go to the next one. Now we're gonna do twenty

companies for the same three sources. And we're gonna hit Run. And bear in mind, this is the same code that's running on every single website. Then we go to the next slide. And we can see Chipotle will come up at the top. I'm getting hungry now. Um, J.P. Morgan. Um, there we go. All right, keep on scrolling. Let's see what else is there. Goldman Sachs.

Going and doing the same tasks across every single site without interruption, unblocked, all using Stagehand as well. And it's really cool 'cause it's a verified browser that I mentioned at the very start, which means that you don't get hit with as many CAPTCHAs, and if you do, we solve them. And the Stagehand allows us to navigate without anyone looking at this. Okay. And we can stop there. And I think we can see some results on the right-hand side. Um, next slide. Okay, cool. We've got some of the results here.

Um, SEC filing data. Keep on scrolling.

I honestly have no idea what the SEC stuff means. Uh, I'm not a finance guy, but hopefully it's useful. Cool. Um, let's go to the next slide. And so that SEC template, I-- was something I built a while ago. If you want access to that particular, just the SEC one, you can do. Um, but we also have, if you wanna get started with anything Browserbase or anything I showed, we have-- If you go to our website, we have a copy prompt. Literally just send it to your agent, and that'll do a lot of the work as well. Uh, next slide. Okay. Thank you so much.

That's our talk.