[
  {
    "startMs": 12920,
    "text": "All right. Hello. Thank you everybody for being here. Uh, I know it's the last workshop of the day. I appreciate all of you turning up here. Uh, I am Shubh. Uh, I'm here to talk about hill climbing skills, uh, specifically how can you improve your agents without really going into the models, without really going in the weights. Uh, I have a Slido, uh, up here as well. I'll just, uh, flash the QR code in case any of you wanna, uh, put in any questions,"
  },
  {
    "startMs": 42880,
    "text": "and we can address them towards the end as well. Um, a little bit about me. I'm Shubh. I'm, uh, on the sales engineering team at Browserbase. Uh, I essentially work with our customers deploying browser agents in production. Uh, I'm not sure, maybe we'll do a show of hands. Uh, how many of you know what browser agents are? Anybody? All right, a few. Amazing. Uh, have any of you built browser agents before?"
  },
  {
    "startMs": 73120,
    "text": "Have any of you used these tools? Show of hands. Okay, a few of them. Amazing. Nice. Uh, maybe I'll, I'll talk a bit about what browser agents are before diving into the whole concept of hill climbing for skills, uh, so that it becomes clear what we're doing. Towards the end of the talk, we'll also look into how you can do this in your domain, in your, uh, field as well. All right, uh, so quick"
  },
  {
    "startMs": 102620,
    "text": "primer on what browser agents are. If you have an agent that needs to drive a browser, you basically call that a browser agent, right? Browser becomes the tool to your agent. And how do you control it? You control it with some sort of CLI, some sort of MCP, some sort of method where you can control the browser and say, \"Hey, go and click on this red button. Go and click on the refresh button. Go and, uh, take a screenshot or take a s-snapshot of the"
  },
  {
    "startMs": 132300,
    "text": "accessibility tree of this website,\" right? So these are the s- low-level tools that your agent can have access to, to control the browser. Uh, and Browserbase is essentially a platform for you to build and run these browser agents. We provide the infrastructure and the SDKs, the skills, the CLI that go with building these browser agents. Uh, and, and we're gonna be using all of these tools to see what hill climbing does look like. All right."
  },
  {
    "startMs": 162300,
    "text": "Uh, I want to talk a bit about the biggest problem with agents today, essentially reliability, right? Uh, people are frustrated with browser agents getting stuck or getting into an anti-bot or a CAPTCHA issue, uh, or maybe just forgetting or being lazy and not completing the task. Uh, and we see reliability essentially as an engineering problem. I think the recent models, the Opus and the Fable, when it, when it"
  },
  {
    "startMs": 192160,
    "text": "came out for those three days, it was really, really smart,  right? Uh, but we see that a lot of the work that has to be done for reliability is the harness around it. It is the tools that you give it. It is the skills that you give here. Um, and, and that's what we're gonna be sort of looking at, how do we improve the reliability of these browser agents? I wanted to give you all a few examples of, like, where these agents suck today, right? If I zoom in a bit, you'll see this browser agent is trying to,"
  },
  {
    "startMs": 222180,
    "text": "like, go on Google Flights and click on the, uh... It, it it's trying to drag the time filter, and it's unable to figure out, how do I drag that? Dragging is a tricky operation for these agents to do, and it's getting super confused. It doesn't know what the HTML element for that is, what the JavaScript for that is. It's not really able to do this, and it gets stuck, and it's not able to book my flight tickets. Uh, another example I wanted to show, uh, is the browser agent OpenTable. Let's say I have"
  },
  {
    "startMs": 251860,
    "text": "my OpenClaw or my Hermes agent, and I need to book a date night, and I tell, \"Hey, go to OpenTable and, and find me some,\" uh, and, and look at the anti-bot that it runs into, right? Uh, and it's just, like, access denied. I don't know what happens here. Uh, but, like, can my browser agent get around these issues? Can they be more reliable, and can they book my date night restaurants is the biggest  question I wanted to ask here. Uh, and this is where we sort of look at different ideas to improve the"
  },
  {
    "startMs": 281720,
    "text": "reliability of these agents. And one of them, uh... Oh, actually, let's, let's dig into the root cause here, right? Like, why this is happening. Uh, we have, like, these Opus models, these Phi-5 that's super nice general purpose. You might have used Codex, ComputerUse, very nice. But when you want to sort of do, uh, long-running, uh, tricky websites like the OpenTables or the Google Flights, it sort of becomes very difficult to run these in"
  },
  {
    "startMs": 311380,
    "text": "production, and these site-specific tricks or these JavaScript that you need to run on your browser, the agent doesn't know about them, right? And the most important thing is it's every run starts from zero. You're not really having any sort of learning baked into your runs. I think Dwarkesh talked about this a couple of days ago. It's like, why is progress on ComputerUse still so slow today? Uh, although ComputerUse seems like a verifiable problem, why is it still not solved? Essentially, there's"
  },
  {
    "startMs": 341320,
    "text": "no learning happening. Every agent run is a start-from-scratch run, and it's not learning from its production traces. Uh, and these browser agents, I call them geniuses that never learn. Um, and some people might say, \"Oh, why don't you fine-tune on some ta- some, like, OpenTable or restaurant websites?\" Uh, but it, it's not really scalable, right? Uh, if you're thinking of building agents, you can't really be fine-tuning for every new task that comes up. You want some f- You want a, a, a modality"
  },
  {
    "startMs": 371404,
    "text": "where it can learn from its production traces, uh, and get better over time. This is where we sort of started thinking about, and this was, like, six mo- four to six months ago where Karpathy came up with this idea of auto-research. This is essentially a loop, right? Uh, and this loop sort of takes a verifiable problem statement and goes at it, tries it again and again, looks at its own traces, and tries to improve itself. And we wondered:"
  },
  {
    "startMs": 401284,
    "text": "how can we apply this to browser agents? How can we apply this, uh, to these complicated workflows, these complicated websites, which are not deterministic, they change every day, they have different CAPTCHAs every time? Uh, how can we learn from these, right? Uh, this is where I wanna sort of switch into a demo and sort of show this live. Uh, and I'm gonna start with, uh,"
  },
  {
    "startMs": 428644,
    "text": "a prompt over here. So I want to maybe do something like, uh, go to Google Flights and, uh, find me a business class flight from SF to New York for this Saturday. I want to have one carry-on bag, uh, and this flight should be in the morning. I hope my... Okay. This is correct."
  },
  {
    "startMs": 455464,
    "text": "Just paste this and run this, right? So starting a browser-based session here. Uh, so what you see in the center is a remote browser has just spun up and has navigated to Google Flights. If I expand my agent here, uh, it's doing all of these things. This is the reasoning traces that you can see live, and it's trying to sort of apply the business class filter. So it, it'll do its job and, and, and we'll come back to this, uh, but I wanted to talk about what's really happening here,"
  },
  {
    "startMs": 485704,
    "text": "uh, uh, and, a-a-and the architecture behind this auto-research loop."
  },
  {
    "startMs": 495824,
    "text": "So we, we fired the Google Flights task. Uh, and, and, and the architecture here is this auto browse loop that we call this. We took auto-research, and we took browsing. We call it auto browse, and this loop essentially is two Claude Agent SDK loop, uh, agents. Uh, you can call one as the doer agent and one as the teacher agent. This doer agent, its job is to actually go and do the browsing. It has access to the browse CLI that we talked about. It has access to the browser, uh,"
  },
  {
    "startMs": 526004,
    "text": "and it goes and sort of clicks on buttons, looks at the screenshot, tries to get me information. And the teacher, the teacher's job is to look at the traces of the doer and learn what went wrong and what went right, and they keep working together until they find a very good solution that converges, right? Uh, we don't tell it how many loops you need to go for. We don't tell it what's the success factor-- uh, success, uh, rate that we want. We"
  },
  {
    "startMs": 555884,
    "text": "don't tell it, \"Hey, we need to do this within X number of tokens or X number of seconds.\" We just give it a broad prompt, and we let the bitter lesson, we embrace the bitter lesson, and we're like, \"Okay, let the doer and teacher go at it.\" Uh, a-a-and the key thing that happens is there is a strategy.md file, uh, that is essentially the artifact across multiple runs. After every iteration, the teacher thinks about what happened, looks at the screenshot, looks at the"
  },
  {
    "startMs": 585724,
    "text": "logs, and then thinks, \"Oh, you did not click on this button. Uh, you should actually click on this button instead.\" And it puts that in the strategy.md file, and that artifact carries over to the next iteration, and it improves from there. Maybe we'll go back to our demo and see what happened. All right, so iteration number one, uh, there was a failure here, uh, and the learning from this iteration was skip the entire form calendar widget because you can just do"
  },
  {
    "startMs": 615804,
    "text": "a query parameter, and you don't really need to click around, right? The agent was smart enough in this instance to think, \"Oh, I'm wasting so many tokens, and I'm wasting so much time clicking around where I could just use Google Flights query parameter.\" Now, this is obvious in hindsight, but maybe as a human, I wouldn't have thought of this, and I wouldn't have written this complicated query parameter myself, right? Uh, so this is the learning from the first iteration. Now it's going and doing things in iteration number two. It'll keep"
  },
  {
    "startMs": 645644,
    "text": "going on till it feels like it has maxed out all the tricks, it has maxed out all JavaScript hacks that it could do, uh, and, and, and then, uh, think about, \"Okay, uh, this is the skill that I can use from here.\""
  },
  {
    "startMs": 663604,
    "text": "I think the key, the key thing, uh, that I wanted to also talk about is at the end of this converged loop, the output that I want is a SKILL.md file. This SKILL.md file would be what I give to my future agents, uh, and it would have a collection of all the hacks or all the tricks or all the learnings from previous runs. We'll, we'll look at what that skill might look like. Uh, it could look like, hey, a pre-filled"
  },
  {
    "startMs": 693224,
    "text": "URL is mandatory. Don't go around and waste your clicks and tokens clicking on these buttons. There is a reopen bug in the calendar. Don't do this. Uh, if you wanna move the drag, uh, if you wanna drag your timestamp thing, uh, use JavaScript injection. Don't try to do it with the mouse, right? These are the kind of tricks that are now persisted for every future run and gives me so many more savings, uh, from a"
  },
  {
    "startMs": 722704,
    "text": "performance, reliability, and cost perspective."
  },
  {
    "startMs": 727724,
    "text": "Now, uh, we'll go back to our, uh, run and see if it has passed. Okay, so iteration number two has passed, and if you can see, uh, the savings in, in cost are, like, four X. Uh, this chart is being created on the fly. Uh, so it started from $12, and the second one was $3 and- Keep in mind, my agents here are Opus 48, both for the doer and, uh, for the, uh, teacher."
  },
  {
    "startMs": 757912,
    "text": "Now, iteration three just came in. Uh, the number of turns decreased. The latency is also better, uh, but th-there was a failure in iteration number three. So there's something that it sort of over-corrected on, and iteration number four should probably be learning from that, and we'll see if it does that. Uh, while it's doing its own thing, I wanted to show some previous runs that I did, uh, just in preparation for this, uh, demo. Uh, so for example, I went to, uh, Auto Browse, and I was like, \"Hey, go to"
  },
  {
    "startMs": 787952,
    "text": "the AI Engineers website and tell me which talks and workshops are talking about hill climbing,\" right? Uh, and it did that across multiple iterations, and you can see the replay of each of these. These are replays that are coming from Browserbase. Super helpful, important for, uh, observability perspective. And as you can see, uh, it sort of, uh, got better in term- i-in terms of cost, seven point four x. Uh, even though it was able to pass in all the iterations, I still got improvements in cost."
  },
  {
    "startMs": 818212,
    "text": "Now, I wanted to actually see what are the learnings from this skill, right? Uh, how do you sort of, uh, persist all of these learnings? It says, \"Go straight to the sched-schedule page. If there are open errors, don't try-- retry blindly. Run this. Uh, type the keyword into the session search input. Wait briefly.\" So, like, have these waits, because sometimes websites take a lot of time to load, so you, you can have these kind of tricks. Uh, \"And then read the results.\" Uh, you know, like"
  },
  {
    "startMs": 848172,
    "text": "there's, there's a lot of learning that is happening in these skills. Maybe I wanted to show some interesting ones. I wanted to show this OpenTable one, because I started with this blocker of, like, access denied when we started this demo. Uh, and if we go to the, uh, screenshot, this was the access denied thing that we got, right? In, in iteration number one. Let's see how it learned from there. Uh, it realized that the, uh, block is environmental because the website doesn't"
  },
  {
    "startMs": 877952,
    "text": "want this agent to do something. Uh, in iteration number two, it figured that if I do a query parameter and if I put a latitude, longitude, metro ID, I automatically get that, uh, UI that works. Do not open slash R, that's blocked, and do not reload slash S. These are the places where the blocks kick in. If you don't do that, and you follow this technique instead, your agent will work, right? Uh, and as you can see,"
  },
  {
    "startMs": 907972,
    "text": "uh, it sort of went from six point two two dollars to zero point five one dollars. The time taken went from hundred and eighty-nine seconds to twenty-seven seconds, uh, and the skill it came up with was here. Uh, build a search URL, do the date math like this. Uh, this is the known metro pins for San Francisco, uh, and, and, and follow these steps, right? Never call slash R because that will return access denied. Now, these are the kind of learnings where a human could never"
  },
  {
    "startMs": 938012,
    "text": "have done, but you just sort of embrace the bitter lesson and ag- let agents do the job, uh, and they will find these, uh, tricks for you."
  },
  {
    "startMs": 948272,
    "text": "I want to, uh, also show, uh, a few examples of, like, the kind of improvements we see. Uh, the reason why this problem is so close to my heart is because I, from a sales perspective, I am deploying these browser agents into production. Uh, the sale is only done when the browser agent is successful, so reliability is like the most important thing for me, and this is what I-- this is the key tool that I take to my customers and I'm like, \"Just fire your agents and let"
  },
  {
    "startMs": 978292,
    "text": "them do this auto research thing, and your agents will be successful in production.\" Uh, and these are the kind of, uh, success metrics that people are looking for. Up to sixteen x cheaper, twelve x cheaper, three x cheaper, uh, on different tasks. Of course, every task is different. The bottlenecks on each task are different, so, uh, you might see variance there. But, uh, the idea is you can hill climb on these, uh, techniques."
  },
  {
    "startMs": 1006612,
    "text": "I wanted to go back to our live demo and see if there's any, uh, improvement. All right, so we have a graduated skill. Uh, and the skill applied end-to-end, no further learning. The results are showing correctly. One-way business class, San Francisco to New York, July the fourth, one carry-on bag included. Uh, and there is this eight AM, six forty-nine dollars, right? So it was able to get all of this, and I can sort of watch the entire replay to see if that is accurate. Uh, it was able to sort"
  },
  {
    "startMs": 1036592,
    "text": "of figure out this entire skill. The fast path is encode the whole query in the URL, str-strongly preferred. The-- Again, like, I know I'm, I'm, I'm showing these things, but when I first started looking at this, the results here, this was like my moment of like, this is the Mythos moment for browser agents. You know? Like just reverse engineering the hell out of things. Uh, I, I look at this with some of my customers. The customers are mind-blown. Uh, earlier, they would be writing"
  },
  {
    "startMs": 1065932,
    "text": "scripts for each of their websites. They would be like, \"Hey, I need to, uh, spend so many engineering hours on like just reversing these websites. How can I scale this to a hundred, a thousand websites?\" And now our customers have actually built these entire pipelines of, uh, these skills that get generated on the fly, pull requests that get created automatically. If there is a failure in production, an agent kicks in and starts healing the skill as well. There's a lot"
  },
  {
    "startMs": 1095952,
    "text": "of, uh, engineering that we can do around this to make leverage-- to leverage this technique. Uh, I wanted to also show what we've done with this, right? We actually went ahead and created this product called Browse.sh. Uh, we were like- If we can do this for one website, we can do this for five, why not do this for like five hundred websites and give it for free for anybody to do this? Uh, and like, if you remember in the nineteen nine-- or whenever, like in the ninety... Before"
  },
  {
    "startMs": 1125956,
    "text": "internet, you had the Yellow Pages. Uh, you had details about accessing people, businesses, et cetera. This is Yellow Pages for browser agents, right? It has details about how to do any task on any website on the web. Uh, I wanted to actually show this to you all. Uh,"
  },
  {
    "startMs": 1147896,
    "text": "you can actually install Browse CLI and use all of these skills completely for free. We've eaten up the token costs ourselves, and we're making this available for you all. Uh, so if I go to, let's say, search products on Amazon, uh, I can see the entire skill is over here. When should you use the skill? How should you look at like pricing and availability and whatever, right? Uh, this is how you use this. Lot of details about Amazon, the screenshots."
  },
  {
    "startMs": 1177876,
    "text": "Uh, and I just give this browse skills CLI to my OpenClaw, and it just knows when to pull the alltrails.com skill, when to pull the weather.gov skill, and, and this makes my life so much easier. Uh, we essentially are running the same auto research loop behind the scenes, uh, and making this, uh, available for you all. In, in fact, if you would want to, you can go here and you can be like, \"Hey, go and make a skill for"
  },
  {
    "startMs": 1207776,
    "text": "me to, uh, look up this, uh, events in San Francisco for this week.\" Uh, do this, generate a skill, and this will sort of do this for you. This will send you an email, and you can then use your skill, uh, in, in whatever agent you want to. Right? Uh, so we've done this for browser agents. Uh, and I wanted to talk about like, how can you potentially do this for any other domain, right? Uh, I think coding agents is"
  },
  {
    "startMs": 1237636,
    "text": "like one place where I have seen this frustration, where the agent is relearning the repo conventions every time. It's not applying the linting rules correctly, and I have to tell it every time. And maybe you can have your CLAUDE.md and AGENTS.md and improve that. But the, the fundamental premise is like, the agent should not be relearning your tasks from zero, right? Uh, I've seen this in re-research agents. We run a bunch of like deep research agents"
  },
  {
    "startMs": 1267656,
    "text": "for our internal workflows. Uh, and every time it struggles and it tries to relearn, \"Oh, I've pulled this information from Wikipedia, uh, but I did not pull it from this other website.\" Uh, and it's like, why not? Uh, like, we always do this other website. You should learn from your previous traces, uh, and you should pull information. Uh, internal ops. You have all these dashboards, all these databases, all these internal tools, but your agents don't know which ones work and which ones are preferred."
  },
  {
    "startMs": 1298176,
    "text": "Ideally, you should be applying this auto research technique. Uh, a-a-and the two conditions that are required is that A, you should have a replayable environment of some sort so that you can run these loops. And B, you should have some sort of verifiable outcome or some sort of quantif-quantifiable metric for you to run these loops. Yes, I see a question there."
  },
  {
    "startMs": 1323216,
    "text": "Um, one of my challenges when I try to do this kind of work is where to draw the line of hallucinations of content from the site versus just kind of like... Yeah, mostly just kind of like avoiding hallucinations, because I realized that it's just really hard to trust any sort of like prompt or agent which just takes copied text from any site. So in some, in some way, I always try to find some compromise"
  },
  {
    "startMs": 1353036,
    "text": "versus like . Yeah. Like a, like what-- like what kind of compromise you-"
  },
  {
    "startMs": 1359076,
    "text": "Yeah."
  },
  {
    "startMs": 1359236,
    "text": "-would install to make sure it says this guy is blue."
  },
  {
    "startMs": 1361276,
    "text": "Yeah."
  },
  {
    "startMs": 1361276,
    "text": "It says blue but not changed."
  },
  {
    "startMs": 1363276,
    "text": "Yeah, exactly. Uh, so I think the question is like, how do you know that the auto research loop is accurate, whether it is converging, and when it says it's converging, it's actually converging. When it says it's good, it's actually good. When it says it's bad, it's actually bad. Like, how do I have confidence, uh, on what the agent is saying and what the loop is doing? I think that's where I want to sort of talk about the verifiable outcome here. Uh, for Karpathy, it was quite easy because he was looking"
  },
  {
    "startMs": 1393336,
    "text": "at the logs, log loss of the experiment when he was training neural networks, right? So it's very easy to see whether your log loss has gone down or not. Uh, but in many real-world scenarios, that is not the case. For browser agents, as I explained, uh, whether you book a flight ticket or not, that might be straightforward, but did you find the best flight options for me might be slightly fuzzy, uh, and might not be the best, uh-"
  },
  {
    "startMs": 1421856,
    "text": "My question is more about like... Less information more about in a website, there are URLs and numbers, especially with numbers and URLs, it's very easy to use probabilistically one letter can click."
  },
  {
    "startMs": 1432516,
    "text": "Mm-hmm."
  },
  {
    "startMs": 1433096,
    "text": "And that's one thing that like makes it wrong, but then the whole URL being copied, like it gets compromised. So it's kind of like, can I still find them or no? Like is there an unsolvable-"
  },
  {
    "startMs": 1445176,
    "text": "Yeah. If, if I understand this correctly, you're saying like one mistake in the loop can sort of throw off the entire auto research run. Uh, and how do you prevent against that? I, I, I have actually seen where some, uh, iterations actually fail. Uh, for example, if we sh-- uh, look at the previous slide."
  },
  {
    "startMs": 1469676,
    "text": "Uh, there's like failures happening after there have been successes, right? Uh, so it's not always that once you succeed, all the future iterations will always be successful. But I think it's the design of your loop where you can make it resilient to, uh- Failures, uh, where it doesn't over-index on those, uh, and still sort of corrects itself, uh, from any mistake that happened in previous runs. Uh, I think we're trying to sort of prevent like, uh,"
  },
  {
    "startMs": 1499696,
    "text": "what do they call it? The exploding sort of l-loss sort of scenarios. Uh, I think you could do that. Uh, in fact, Karpathy talks about this like this is one way of doing auto research. You can actually auto research on different ways of doing auto research itself. Uh, it's like, is the teacher-doer loop the best? Should there be a multi-agent architecture to do this? Or sh- Is there any other approach that might actually work better for research tasks in neural networks versus browser agents, right? You could have different"
  },
  {
    "startMs": 1529436,
    "text": "designs of your loops to prevent situations like those. Uh, and, and I think going back to my point, as long as you have a verifiable outcome or some quantifiable metric, I think you should be able to design, uh, a good, uh, loop here. Uh, replayable environment, I think sometimes b- is under-discussed, uh, but extremely important. I think it was talking about Dwarkesh. He said, having a verifiable outcome is important, but also you should have, uh, extremely"
  },
  {
    "startMs": 1559436,
    "text": "grindable environments or extremely grindable, uh, use cases as well, right? If your agent cannot, uh, grind through failures, it will be difficult for it to, uh, go for multiple iterations and fix itself. Uh, so do you have... Like coding agents is a good example because you can start from a Docker image, or you can start from, uh, a particular EC2, and, and you can replay from different parts of the, uh, stack."
  },
  {
    "startMs": 1589856,
    "text": "Uh, but maybe, uh, knowledge work is probably not that easy to replay. Uh, maybe there's like a lot of different human-in-the-loop actions that make it very hard to replay from a particular point in time. So investing in your environment also can be super, uh, rewarding in these cases."
  },
  {
    "startMs": 1612676,
    "text": "Amazing. Uh, I just wanted to share the, uh, URLs of, uh, the Auto Browse loop that I, uh, demoed today. So if you scan the QR, you'll be able to pull the, uh, GitHub for that. Uh, and also the second QR here, uh, is the skill itself. You can just do /autobrowse on your Claude Code or Codex and be like, \"Hey, /autobrowse. Use Opus 4-8 on both of these, uh, teacher and doer, and go create a skill for me to do this.\""
  },
  {
    "startMs": 1642556,
    "text": "Uh,"
  },
  {
    "startMs": 1646056,
    "text": "I'm looking at one of these questions. Uh, and also, if any of you have ideas of something we want to sort of put the agent against, feel free to type it in the Slido, and we can actually try it live. Uh, but let's look at this question. Is there a way to optimize for convergence? Uh, I think this is the same question that we were just talking about. Uh, high cost is interesting because you might think that, uh, doing longer iterations might become costlier"
  },
  {
    "startMs": 1676096,
    "text": "over time. Uh, I have run a bunch of evals on this where I can throw a sonnet at this problem but run it for longer iterations versus throwing Opus for lesser iterations. I think you can get the same job done by throwing sonnet. L-larger number of iterations, but still cheaper overall. Uh, also this cost is amortized, right? Like one Auto Browse loop that we probably run is like maybe thirty dollars, forty dollars. If it's super complicated, a hundred dollars at most."
  },
  {
    "startMs": 1706596,
    "text": "Uh, so this is amortized. As long as those costs make sense for you, uh, to deploy this in production, uh, just go for it. Uh, but there's a lot of interesting things, uh, that you could do, uh, with evaling the kind of loops that you're generating. Uh, yes, I will share the slides as well. Uh, I can do this on the GitHub itself. Uh, can you sh-share how you thought about, uh, auto-research into Auto Browse? Uh, I mean, the"
  },
  {
    "startMs": 1736236,
    "text": "process was fairly straightforward. Uh, took the GitHub repo that Karpathy put out, uh, and then there's all this context about my customers, uh, that I have, uh, where I was like, \"Hey, these customers face these kinds of reliability problems.\" Uh, the way I solve it for them is I go into their office, I sit with them, very forward deployed, uh, and I try to sort of just brute force and reverse engineer this. This is what I do, so I am the teacher agent. Uh, can"
  },
  {
    "startMs": 1766196,
    "text": "you automate me out of the process and create a loop, borrowing from, uh, Karpathy, and, and, and, and build the same thing? And of course, there's a lot of iterations on top of that. Uh, there's a lot of engineering as well, uh, to simplify this. Uh, but yes, uh, that was the process of converting this."
  },
  {
    "startMs": 1788156,
    "text": "Can you use this in a proxy? Can you be logged into a site? Yes. Uh, again, this goes back to the environment that you design. Uh, Browserbase, the infrastructure that all of this is running on, uh, gives you the ability to log into websites, uh, save the cookies, uh, reuse those cookies in future runs so you don't have to log in every time. Uh, makes it super easy for you to, uh, replicate real-world workflows. Uh, and yes, you can use proxies as well if you have, uh, particular IP addresses,"
  },
  {
    "startMs": 1818096,
    "text": "residential IP addresses. You can bring those in as well. Uh, a lot of success is definitely how good your infrastructure and environment is, especially in these adversarial scenarios, uh, like CAPTCHAs and anti-bots. Uh, so making the infrastructure as good as possible is definitely of value."
  },
  {
    "startMs": 1842256,
    "text": "Amazing. Any, any asks for something that we might wanna run live? Yes, question."
  },
  {
    "startMs": 1848936,
    "text": "One question"
  },
  {
    "startMs": 1856296,
    "text": "Great, great question. Actually, I wanted to sort of show the code as well. Uh, so if you, uh, look at the GitHub, you'll be able to find this evaluate.mjs file. Uh, essentially it lists out the entire loop, uh, what's happening here. Uh, but to answer your question, the teacher and the doer are, uh, working one after the other for every loop. So the teacher is finding learnings in every"
  },
  {
    "startMs": 1885916,
    "text": "iteration. Uh, maybe if I show the demo, uh, over here. Uh, let's go to the one we just did. Iteration one had these learnings,  and then iteration two had some other learnings, right? You should include the explicit ISO date in this particular format. Iteration three would have some other learnings. The date resolution still opens the bare page and clicks... Wasting three turns,  uh, reading today's date, uh, or having the executor compute today. So it's figuring out something"
  },
  {
    "startMs": 1915836,
    "text": "about the date, time. So in every iteration, the teacher is learning something new and putting this back into the strategy.md. Maybe I'll make this real by looking at the traces for one of these runs. Uh, so over here, these are all my runs. Uh, so for example, this was the OpenTable reservation example. Uh, and, uh, this was the strategy.md that the teacher sort of appended to every"
  },
  {
    "startMs": 1945796,
    "text": "time. So, uh, step zero, you should do this, so on and so forth. Uh, this was the task. This was the, uh, prompt that I gave it. Find me reservations for the Bix ti- Bix Restaurant. Uh, and these were the kind of traces that, uh, it was sort of learning from. It would take screenshots. It would take these sort of traces. At every step is doing browse open. This is what the output is. Then turn number two, browse snapshot. Snapshot essentially takes the accessibility tree, and this is the snapshot. So it's"
  },
  {
    "startMs": 1975736,
    "text": "reading this entire thing and at every step thinking of like what went right and what went wrong. Uh, yeah, I mean, this is like machine code. Uh,  I won't show this entire thing. Uh, but yeah, this is the summary that it also writes, uh, at every iteration. Uh, so yeah, there's, there's a lot, lot of like engineering tweaks that you can do to suit this to your domain as well. Uh, and, uh, like the easiest way to, uh, read this is"
  },
  {
    "startMs": 2005736,
    "text": "we put it out here on skills.sh as well, uh, in auto browse. Uh, and you can sort of do something like this."
  },
  {
    "startMs": 2016416,
    "text": "Um, is this for a, a single task? Uh, because generally you want to have a agent, uh, that handles lots of tasks in a particular domain. So really you are not training one, uh, training one model for one task, right?"
  },
  {
    "startMs": 2034936,
    "text": "Great question."
  },
  {
    "startMs": 2035936,
    "text": "Like, one model for a whole set of tasks and then have it aggregate, uh, score for-"
  },
  {
    "startMs": 2043456,
    "text": "S- such a good question. I think that goes back to like how do you increase the reliability of your, uh, agents? Uh, reliability is an engineering problem. Uh, you can design your harness to sort of go for one skill for each task, or you can be like, hey, one skill for every task that you will ever see. Uh, I think it entirely depends. How you do the harness engineering entirely depends on the use case that you have."
  },
  {
    "startMs": 2073656,
    "text": "Uh, and you run evals with all these different strategies, uh, and see which one works for you. For us, this was... Yeah."
  },
  {
    "startMs": 2080916,
    "text": "Sometimes that doesn't work because if you design the pipeline, the assumption you have is that these tasks are independent. But, but really when the user asks a question he might end up with a-"
  },
  {
    "startMs": 2095836,
    "text": "Yes"
  },
  {
    "startMs": 2096076,
    "text": "... with tasks. And so he might be talking about multiple tasks."
  },
  {
    "startMs": 2100596,
    "text": "Yeah. Yeah, and this is where, again, the harness comes into the picture. If the harness has the intelligence to bring in skills from OpenTable and from Resy and Google Res- Google Reservations, maybe it can bring all of those in to figure out what is the best way to book me a date night. That's your solution, right? Like, the harness should be able to, uh, figure out which tools to use to get the job done. That's like"
  },
  {
    "startMs": 2130536,
    "text": "how I would go about it. Uh, figure out what are the most common use cases that your customers have. For those use cases, what's the best skill files, prompts, tools? Maybe you wanna do like specific tools for specific use cases, right? Maybe you wanna do generic tools for all use cases. Like that's the same argument here, is like, do you do s- one skill for each or one skill for all? Uh, and eval across all of these strategies and, and let the harness figure it out."
  },
  {
    "startMs": 2158676,
    "text": "I have a question about that info with GEPA."
  },
  {
    "startMs": 2161556,
    "text": "Yes. Yes. Yeah. Uh, I mean, it's a very similar concept as this, right? Uh, so DSPy."
  },
  {
    "startMs": 2169396,
    "text": "Yeah."
  },
  {
    "startMs": 2169876,
    "text": "Yeah. I, I think those guys were onto something really cool. Like they were around like three, four years ago where they started building DSPy and GEPA, all of that. I think, uh, we have been doing a lot of research, uh, along these lines as well. Browserbase works with, uh, Microsoft. We actually released a paper, uh, about how to build the right sort of verifiers, uh, for your agents. Uh, I think running evals is straightforward until you start questioning, okay, how should I eval"
  },
  {
    "startMs": 2199896,
    "text": "my eval, right? Like w- uh, do I need to put all these humans to verify all these runs, or is there a good verifier that I can build here and all of that, uh, uses the, uh, learnings from, uh, these papers as well. I can share all of that in the GitHub as well."
  },
  {
    "startMs": 2220468,
    "text": "I'll also see if there's any new questions here."
  },
  {
    "startMs": 2225688,
    "text": "Where and how are these skills stored, retrieved at scale? Uh, right now, for this repo, uh, all of these get stored in my .claude over here. So each of these is a skill file. This was my recipe searcher. Uh, I wanted to build some-- I wanted to cook some dinner yesterday, so I was like, \"Hey, make me some recipes.\" Uh, so all of these are currently in my local, uh, skills folder. Uh, but if you want to ship this, uh, in production, of course, you can,"
  },
  {
    "startMs": 2256048,
    "text": "uh, bake this into your agent loop. Uh, or if you wanna s- I think for browse.sh, the public, uh, publicly available, uh, skills that we have, I think we use Vercel blob storage to store all these skills, uh, and make them on demand. Hope that helps. Uh, websites change frequently. What's your stra-strategy for monitoring that and keeping the website skill updated? Yeah, this is such a great question. Uh, and this is what I was alluding to earlier is like, how"
  },
  {
    "startMs": 2286028,
    "text": "do you keep these sc-skills up to date because the website's gonna change? The, m- the current working theory that we have right now is you don't want to waste tokens running auto research every day. You wanna probably trigger auto research if it's failing at any particular point in production, right? So we build these loops or, like, systems where as soon as we run into any production failure, we run an auto research loop to see what"
  },
  {
    "startMs": 2316068,
    "text": "went wrong, to heal it, create a pull request that goes directly to a, a human engineer, and they fix it. Automatically, the skill gets updated. Uh, so yes, there is that notion of, like, production failure when this might happen. Uh, but it's a trade-off. You can run an auto research, or you can run a sample run every two hours, every three hours to see if the skill is working, uh, perfectly. Uh, so there's different ways of having the, uh,"
  },
  {
    "startMs": 2346988,
    "text": "cron jobs, uh, check for the accuracy of these skills. This is actually one thing that I'm actively working on with a customer right now to implement. And there's all these questions, right? People don't want a hundred percent accuracy. People might want, like, the next cheaper cost. So  like, uh, that's the, uh, question that comes up. What's your teacher's criteria here in the example separate from reaching the goal? It seems to do some optimizations for turns. What other criteria do you use? Uh, so again, this"
  },
  {
    "startMs": 2376908,
    "text": "is the design of the auto research loop. Uh, there's one way of telling your auto research that, \"Hey, I wanna focus on just success. As soon as you get me success, I'm fine. Just give that to me.\" Uh, that's typically not what everybody wants to do. Uh, people want to minimize on, uh, latency, uh, because maybe this is a real-time run where you're focused on how quickly can you get the job done. Uh, so if that is the optimizing metric, and you focus on getting it f- as fast as"
  },
  {
    "startMs": 2406808,
    "text": "possible. What I've seen sometimes these auto research loops are so smart, they realize that browsers are not needed at all, and they just reverse engineer the API, and they file the API, and that's the fastest that you'll ever get to, right? Uh, so sometimes, uh, I've seen situations where, uh, it finds a solution f- that works in five hundred milliseconds. Uh, sometimes it is the number of turns, right? You want to minimize your context loop, uh, and you want to minimize the number of,"
  },
  {
    "startMs": 2437128,
    "text": "uh, turns it takes to complete the task. Uh, in my instance, I think what I have done is it's your decision to pass or converge whenever you want to pause it. I'm not really concerned about the costs. Uh, I care most about reliability, so keep doing it as long as you feel it's relevant, and then it's the teacher's agent's job to, uh, call it a day when it wants to. But yeah, you can, uh, try all different"
  },
  {
    "startMs": 2466968,
    "text": "strategies. Uh, if a website supports WebMCP, will this use it? WebMCP is so cool. Uh, it's essentially this concept-- I don't know if everybody knows this, but it's this concept of, like, making your website programmable. It's, it's come out in this new Google Chrome release recently, where you tell your, uh, website that if anybody wants to book a reservation, they can do XYZ on the website. It's kind of like an API, but it's on the browser."
  },
  {
    "startMs": 2497208,
    "text": "Uh, right now, I don't think our auto research loop supports this. Uh, but I don't think it would be, uh, too hard to do that. Uh, in fact, we recently shipped support for WebMCP with our SDKs that we launched. Uh, we also made some tools for website owners to ship WebMCPs at scale. Uh, but yeah, this is an interesting one. We've not really seen a lot of adoption with WebMCP as yet."
  },
  {
    "startMs": 2526728,
    "text": "Uh, ground truth for pass. That's interesting. Uh, I think as long as the job was done, as long as... For example, in Google Flights, uh, if it found the cheapest flights, in OpenTable, if it found a reservation, uh, so it, it evaluates itself. It's typically a binary. Uh, and yeah. Again, I, I, I fully embrace the bitter lesson. I try not to put many guardrails in, uh, these"
  },
  {
    "startMs": 2556248,
    "text": "evaluations. Uh, have you considered a verifier as a service for browsing tasks, including something that allows us to mock certain side effects like purchases? Uh, I might have to ask, uh, s- a follow-up on that. What do you mean by verifier as a service, Ravish, if you're here? Yes."
  },
  {
    "startMs": 2609298,
    "text": "Uh,"
  },
  {
    "startMs": 2612098,
    "text": "so, uh, do you mean that..."
  },
  {
    "startMs": 2616278,
    "text": "Yeah, the, uh, if, if I understand this correctly, like you have like different environments that you can use for RL training, and you wanna like mock what happens if I am replicating Amazon, if I click on purchase, what happens after? Uh,"
  },
  {
    "startMs": 2634558,
    "text": "no, because, uh, we try to rely more on real world websites. Uh, browser-based, we have the privilege of ha- ha- being able to access most of the web with our, uh, access to different websites, uh, and CAPTCHA solvers, et cetera. Uh, so we try to mimic the real world as much as possible. But we've been seeing a lot of these RL environments where people are mimicking websites to train, uh, and build browser agents that are, uh, uh, really good at like"
  },
  {
    "startMs": 2664478,
    "text": "browsing Salesforce, really good at like browsing SAP. Uh, we try to do that on real world portals as much as possible."
  },
  {
    "startMs": 2674478,
    "text": "Are hCAPTCHAs easy for the skill to move past or is it token intensive? Uh, what is the point of hCAPTCHAs  for website owners? hCAPTCHA, if I'm correc- uh, understanding this, is like a particular kind of CAPTCHA. It's kinda like Google reCAPTCHA, et cetera. Uh, yeah, I would probably talk about this separately. Uh, I don't think we can talk about like what it c- what CAPTCHAs we can solve or not. Uh, but it's not a token-intensive job, uh, per se, because it's an infra-- like it's a thing that's done at"
  },
  {
    "startMs": 2704418,
    "text": "the infrastructure layer, right? Uh, y- y- we don't really need to use an AI to solve a CAPTCHA. Uh, it's solved at the Chrome binary layer, uh, and we do that out of the box."
  },
  {
    "startMs": 2719538,
    "text": "How does the teacher evaluate the doer and when is the skill marked as graduated? Yes. So, uh, it evaluates the doer by essentially reading these traces. Uh, for every iteration, there is this folder that gets created. Uh, there are these, uh, JSON files. Uh, it reads these. Uh, it has a Claude Agent SDK loop. Uh, and it essentially figures out what the learnings from that loop are. Uh, actually, I think I have a slide that talks about this"
  },
  {
    "startMs": 2749498,
    "text": "at the end."
  },
  {
    "startMs": 2753398,
    "text": "Yeah. Uh, this is kinda like the prompt for this, uh, where it says that you're a browser automation agent, you can use these tools, uh, and the strategy.md gets injected at every iteration. Uh, and this is how the, uh, uh, doer-- uh, the teacher evaluates the doer's job. Uh, so yeah, it reads the traces, it has the system prompt, it has the previous strategy, and it comes up with a better strategy."
  },
  {
    "startMs": 2787518,
    "text": "Amazing. Uh, I hope we've covered all the questions. Uh,"
  },
  {
    "startMs": 2796418,
    "text": "any other questions live that we might wanna talk about? Yes."
  },
  {
    "startMs": 2804838,
    "text": "Yeah, it can work with the Stagehand SDK, it can work in your Claude code. Uh, it's essentially a markdown, right? Uh, I took, uh, a skill and I told my Claude code, \"Can you find events in SF for this Friday, July the 3rd?\" I went on and did this. So i- it need not use browser-based, it need not use Stagehand. It is essentially a skill. It is a portable artifact you can use in any agent or any SDK."
  },
  {
    "startMs": 2850478,
    "text": "Yeah. Yeah. Great question. I think if I understand this correctly, how do you prevent against prompt injection? Uh, because websites could trick my auto research loop in learning something bad that can do some malicious behavior in, in a production setting. Uh, there's a few ways we do that. One is at the infrastructure layer itself. Uh, we have a bunch of, uh, protections around our browsers"
  },
  {
    "startMs": 2880118,
    "text": "itself. Uh, we have this browser-based safe mode where you can sort of control which URLs it can go to. You can control what kind of actions it can take on different websites. So at the infrastructure level, at the network firewall, we control things. Now you might say that it is possible that amazon.com gets hacked someday. How do you prevent against like malicious actions on those portals as well? Uh, I think we also try to, uh, run our skills that are generated"
  },
  {
    "startMs": 2910018,
    "text": "against some security, uh, reviewer agent of sorts, uh, before we actually publish this on Browse.sh. Uh, so that's like one way of doing it. Uh, we encourage our customers to have, uh, a good level of, uh, prompt injection guardrail, uh, surface area, uh, if they're sort of giving this out to cus- consumers. Uh, for example, some customers of ours, uh, might want to automate something on Amazon, and what if somebody puts uh,"
  },
  {
    "startMs": 2940558,
    "text": "a listing on Amazon which has some hidden instructions, and how do you prevent against those kind of things? Uh, and that's where, uh, we work with our customers to build the right kind of- Uh, guardrails before the, uh, instruction even reaches the harness. Um, so yeah, I think I would put like three layers here. One is as the, as the query is even entering our surface area, we control that. At the skill layer, we make sure that our skill is not, uh, tricked,"
  },
  {
    "startMs": 2970954,
    "text": "and at the infrastructure layer, we have network restrictions and action restrictions. You can even do something like what Codex or PlotCode does. Um, like always allow or allow once or allow any action. Uh, you can-- Uh, this is what like Browserbase safe mode offers out of the box. I know browser is a crazy threat vector when you think about it, like it has access to anything on the wild web. Uh, and take that super seriously from a prompt injection POV."
  },
  {
    "startMs": 3003014,
    "text": "Amazing. I'll wait for ten more seconds if there's any other questions. Uh, this was fun. I really enjoyed this. I love the kind of questions as well. Uh, and yeah, I will put the QR code back on screen in case any of you want to grab this. Yes."
  },
  {
    "startMs": 3022994,
    "text": "Um, how do you maintain like the quality of your skills? Like in my experience when I review like auto-generate skills, like quickly just like explode and become like outdated. Like how do you-"
  },
  {
    "startMs": 3033794,
    "text": "Yeah."
  },
  {
    "startMs": 3033874,
    "text": "-prevent that from happening and how it's like not only happening from like your own development, um, environment changing, but also from the browser, from the website changing as well? How do you ensure that makes sense?"
  },
  {
    "startMs": 3044834,
    "text": "Yeah. Yeah, exactly. Yeah, I think we talked about it a while back. What you could do is like I used, uh, Anthropic managed agents quite a bit. Uh, so I would run a cron job on managed agents where I would say, \"Use this skill every day at nine AM to do this job, and if this is a success, all good. If this is a failure, send me a notification here.\" That will trigger a healing loop. Uh, so that's how I sort of can bake in, uh, checks and balances around websites changing, anti-bots"
  },
  {
    "startMs": 3074894,
    "text": "changing, et cetera."
  },
  {
    "startMs": 3081974,
    "text": "Amazing. I'm gonna hang around here. Uh, also, uh, I'll be at the Browserbase booths all days of the conference as well. So, uh, in case you're curious and wanna chat more, uh, would love to do that. Thank you."
  }
]