The Current State of Browser Agents

Talk transcript

1. 0:01

   Hi, guys. Um, I'm Jerry. This is Wyatt. Uh, welcome to our talk. We're talking about the current state of browser agents. Um, Wyatt and, Wyatt and I are founders of Halluminate.

2. 0:12

   Uh, we build realistic environments to help train computer and browser-use AI, but that's not as important for today's talk. Today, we are gonna talk about three topics, which is first, uh, we're g- we're gonna discuss what are browser agents.

3. 0:27

   Second, we're gonna talk about how good they are currently, and these findings are gonna be powered by a benchmark that we actually just published last week. And lastly, we're gonna talk about what it means for you, the listener and AI engineer, trying to build with browser agents today.

4. 0:42

   So first, very simply, what is a browser agent? Um, well, our definition of a browser agent is quite simple. It is basically any AI that can control a web browser and execute tasks on behalf of the user.

5. 0:54

   On the left side, you'll see a quick GIF of a browser agent purchasing a power tool by essentially going to a manufacturer and actually ordering it and going through the checkout process by itself.

6. 1:06

   It's really cool tech, and it's only really become feasible in the last year with the advancing large language models and infrastructure around it.

7. 1:16

   Now, um, explaining a little bit about what's going on underneath the hood, most browser agents, not all, but most browser agents are powered by the same loop that you see here of three key steps.

8. 1:29

   Uh, the first is observing, also known as the observation space. Browser agents essentially take a look at the given context of the browser they're currently operating in and figure out what to do next, right?

9. 1:41

   They do this by either taking a screenshot, this is the VLM approach, or there are some browser agents that also extract, uh, the, the HTML and the DOM. This is more of like a text-based approach.

10. 1:52

    Um, after figuring out the context, the next thing that they do is that they reason through and figure out what the next steps are required to execute the task from the user.

11. 2:02

    For example, if I ask a browser agent to purchase a power tool, maybe it says, "Okay, I need to go and click the search bar." Which takes us to the last step, which is browser, browser agents can take an action, right?

12. 2:12

    This is also called the action space, where browser agents can either click, scroll, maybe fill in some text, um, which then takes us back to the first step of the loop, which is a new state of the browser, and then they need to figure out what to do next, right?

13. 2:24

    So this is called the browser agent loop, um, and this helps you build some intuition on what's going on underneath all of that action.

14. 2:34

    And there are a few major use cases, uh, that browser agents have started to penetrate in the last few months. Um, these are the most common ones. I won't go into too much detail here, but, like, just the voiceover, uh, the ones that we've seen most commonly, web scraping involves essentially launching a fleet of browser agents to

15. 2:50

    extract information. For example, a lot of sales teams use this to find data about prospects. Um, software QA is using browser agen- browser agents to essentially click around and test software that's about to be released.

16. 3:01

    Form filling or a variation of this, which is job application filling, is really, really popular today. A lot of automated job prospecting tools are powered by browser agents today.

17. 3:09

    Um, and generative RPA is, like, a really catch-all broad category. This means a lot of things, but a lot of companies are exploring using browser agents to essentially automate traditional RPA workflows that break very often, and you see this with companies like UiPath.

18. 3:22

    Um, so yeah, that's kind of what browser agents are. I'm now gonna pass it off to Wyatt to tell us a little bit about how good they are.

19. 3:28

    Yeah. Thanks, Jerry. Now that we have some context on browser agents, we can move on to the next question, which is how good are they? So to answer that, we need to think about what it means to evaluate a browser agent.

20. 3:42

    At a very high level, it is simple. It is you give a task to the agent, did it do the task? Uh, in practice, this is a lot more complicated.

21. 3:52

    So the first thing is, how do you come up with tasks for your agent to do? How do you come up with that dataset? We need tasks to be realistic.

22. 4:02

    Um, they have to be feasible. Typically, there are some-- there's a domain-specific element to them. Um, and they need to be scalable. You have to create a lot of tasks.

23. 4:12

    We also need to perform the evaluation. Evaluations can be done, uh, automated, uh, if you have, like, a val, uh, validation function. They can be done manually with human annotators.

24. 4:25

    There's LLM-as-a-judge approaches. Uh, there's a lot that goes into that. And then finally, where is the actual browser agent running? On what infrastructure? Um, as you'll see later, infrastructure is actually a key part of how well browser agents perform, uh, so big thing to consider when evaluating.

25. 4:45

    Broadly speaking, there are two types of tasks, read tasks and write tasks. Read tasks are typically just information gathering and collection, whereas write tasks involve actually interacting with and, like, changing state on a site.

26. 5:00

    So read tasks are more of like a web scraping, whereas a write task is really taking action, what you would want a web agent to be able to do.

27. 5:07

    Um, write tasks are more complicated and more challenging, and both in, in creating them as well as actually for the agent to perform them.

28. 5:19

    Now, this brings us to WebBench, which is the benchmark dataset and evaluation that we performed, uh, last week. So we created over five thousand tasks. Uh, open sourced about half of them.

29. 5:31

    These are both read and write tasks, uh, on close to five hundred different websites in a bunch of different categories. You can see in the chart there. Uh, as far as we know, this is the largest scale dataset for web use that we've seen.

30. 5:44

    Um, so we're really excited to put this out there for the community.

31. 5:49

    Um, now let's just take a look here at the performance of a few industry-leading web agents on just read tasks. And you'll see right away this is actually really good.

32. 5:59

    So we have a, a baseline of OpenAI Operator with human-in-the-loop supervision, and that sits at about eighty percent. And our leading web agent is actually hitting right around there, 80%, uh, with, you know, several getting close to that, um, which is great news for information retrieval and data extraction.

33. 6:20

    Agents are pretty good at surfing the web and finding information and getting it, returning it back to you. Uh, as we'll talk later too, the failures here, that kind of last twenty, twenty-five percent, are typically infrastructure and internet related and not necessarily agent related.

34. 6:37

    So here is an example of a read task.

35. 6:41

    So i-it's just extracting information, but you'll notice it's not necessarily a simple task. The UX, UI is pretty complicated. There's multiple searching steps and filtering steps. There's a Cloudflare pop-up, uh, and there's a few different scrolling interactions that the agent needs to partake.

36. 7:01

    Now let's talk a little bit more about write tasks. Uh, overall performance on write tasks is much, much worse. Drops about 50% or more. You can see that, uh, operator baseline there only dropped about 10%.

37. 7:14

    So operator with a human in the loop, just a small dip in performance, whereas fully autonomous agents across the board had a much, much bigger dip. Uh, the question here is what is causing this?

38. 7:25

    Why are web agents so much better at read tasks, and why do they struggle so much with write tasks?

39. 7:33

    There's a few reasons for this. Uh, one of the main reasons is that write tasks typically are a longer trajectory, so it just takes more steps to perform write tasks than it does to perform read tasks.

40. 7:46

    And the more steps a task takes, the more likely an agent is to make a mistake at one of those steps and overall fail on the task. Um, this isn't the only reason, though.

41. 7:58

    Write tasks also typically involve interacting with more complicated or difficult parts of the site, and the user interface there is, is a little bit more challenging for agents. Whereas reading, you're typically searching, maybe filtering, maybe scrolling.

42. 8:12

    Read... Uh, sorry, write tasks, you're gonna be doing a lot of data input. You'll be doing data extraction, a lot more complicated forms.

43. 8:21

    Just all around it's, it's a more difficult interaction.

44. 8:25

    Writing also involves typically login or authentication, so a lot of websites you can go on without logging in, and you can find information. But if you wanna update or add information or delete information, you're gonna have to log into an account.

45. 8:37

    Logging in is a very challenging activity for web agents, both from the interactive, uh, side of things, the user experience side for the agent, as well as managing credentials for the agent, so having a username and password and an email for the agent, uh, and managing that is a challenge.

46. 8:53

    And then finally, anti-bot protections are a lot stricter, typically on sites that have a lot of write tasks, as well as performing write tasks can even trigger, uh, anti-bot things like, you know, you might not hit a CAPTCHA until you have to input information, and then you'll have a CAPTCHA before you can input that information.

47. 9:16

    So here is an example of a write task submitting a recipe on Allrecipes. Uh, the trajectory you'll see is much longer. It also involves not one but two steps for logging in,

48. 9:28

    and the UI here is pretty complicated. It's a dynamic UI. The agent is actually adding new, uh, forms to fill out, and ultimately, I believe this task was a failure.

49. 9:39

    Um, the agent was not able to successfully submit the recipe.

50. 9:46

    So overall, this is combined read and write. Um, our best agent got about, you know, two-thirds of the tasks successful, whereas the average comes at a little bit over 50%.

51. 9:58

    And again, that OpenAI Operator with human-in-the-loop supervision sitting close to 80%, seventy-five, 80%. So I think there's kind of two ways to look at this graph. One is pessimistically, wow, these agents aren't that good.

52. 10:12

    Um, but I actually think this graph is more impressive than it is negative. The fact that web agents today are able to achieve results like this on a pretty challenging, comprehensive benchmark shows how well they are actually performing, even though there, there's been only about a, you know, a few years of development on web agents.

53. 10:32

    So I think there is a long way to go, and obviously at some point you'd hope this graph gets closer to 100%, um, but overall, these numbers are pretty impressive.

54. 10:44

    Now let's talk a little bit more about some of the failure patterns that we saw. So here are a couple examples of situations where the agent was responsible for the failure.

55. 10:56

    On the left side, you can see there's an example of a pop-up that the agent isn't able to interact with. It's not able to close, and so the agent isn't able to complete its task because that pop-up is blocking it from interacting with the site in the way it needs to.

56. 11:11

    Now, on the right side, you can see there's just a timeout issue. The agent took too long, uh, and ultimately failed because it timed out. So we call these agent failures because a more intelligent, more capable, uh, agent should be able to do these tasks, 'cause there's nothing blocking the agent from performing these tasks other than the

57. 11:31

    agent's own abilities. This is contrasted with what we call infrastructure failures. So infrastructure failures are more, uh, related to the framework or infrastructure that the agent is running on rather than the agent itself.

58. 11:47

    So this isn't an issue of the agent not being intelligent enough or not having the capabilities, um, rather it's, it's something underneath that where the agent is actually prevented from doing what it needs to do because of the infrastructure it's running on or the framework that it's implemented in.

59. 12:02

    You can see on the left side, the agent searches for the site. It's blocked, uh, like it's flagged as a bot, and it's blocked from even entering the site.

60. 12:09

    On the right side, the agent is logging in. It re- is re- site requires, uh, email verification. The agent And this framework has no ability to access that, so the agent is unable to log in.

61. 12:24

    Looking at the performance of our agents that we benchmarked overall, you can see here we have success,

62. 12:33

    agent and infra failure, and then agent failure. So again, agent failures are, it's the primarily the fault of the agent. It was, it should've been able to perform the task, but it wasn't.

63. 12:42

    Agent and infra we coupled together. There's some times where those things are intertwined. It's a little bit hard to tell if it was purely an infra failure or not, but broadly speaking, that agent and infra is, is something along the lines of a CAPTCHA proxy, uh, login or something like that.

64. 12:58

    Uh, and you can see that kind of agent and infra chunk, if we can improve infra, that's a huge bump, uh, to the overall performance of our agents.

65. 13:10

    Finally, the probably biggest across-the-board flaw with agents right now is they are very, very slow. So you can see here, um, this is the average task execution length. Now, it's a little bit skewed because when agents fail, they typically go down a, like, death spiral where they're not able to recognize that they can't do the task, so

66. 13:31

    they just keep trying and trying and trying until they hit a timeout. Uh, but even with that said, we see that agents just take forever to do things. Uh, this is typically because of the agent loop.

67. 13:42

    They have to observe, they have to plan and reason, break the task down. They make mistakes or they fail in their tool calls. They'll have to retry actions, and it just takes them a long time to actually interact with the site, to click around, to navigate the site.

68. 13:55

    Uh, I think the takeaway here is that if you're gonna have an async set and forget application, you might be okay with, with some of this latency. Um, however, for anything that's real time, this is a huge problem, and the latency issue really needs to be figured out for any applications to be effective in real time.

69. 14:17

    Awesome. So, um, those are the findings that we've discovered while benchmarking agents on a wide variety of tasks. Now, taking a step back, we wanna discuss, like, what do these results really mean for you, the AI engineer trying to build with a browser agent, right?

70. 14:34

    And I think there are three, like, big takeaways that we would like to really, uh, impart on you, which is first, um, as Wyatt really alluded to, picking your use case is extremely important, right?

71. 14:47

    And I think the real question you need to ask yourself is, are you building a read or a write use case? Um, as we saw from the results, read use cases today are already pretty performant out of the box, right?

72. 14:58

    Not perfect, but if you're creating like a deep research tool or sort, or some sort of system that uses browser agents to retrieve informations in mass, out-of-the-box products, um, work pretty well here.

73. 15:11

    Now on the other hand, if you're trying to build a product that involves more write functions such as filling in forms or actually doing work, um, and like changing the state of, of some software, uh, just be aware that, like, out-of-the-box browser agents might not be as accurate as you want them to be.

74. 15:28

    And it's not that they can't get there, it's that you're gonna have to test them pretty rigorously before releasing them in some sort of production capacity. And this leads to just building a lot of internal evals, um, for your agents, right?

75. 15:39

    And this is important whether you're building read or write products, but I think it's especially important if you're trying to create, uh, agents that take write-based actions.

76. 15:50

    The second key takeaway, uh, to really hammer home is browser infrastructure matters a ton. Uh, again, this is the second time you've seen this chart, and that's because you can see very apparently that, like, the browser infra you choose can have pretty strong performance swings, right?

77. 16:06

    And so this just means, like, really test the browser infrastructure that you're trying to build on top of. Try multiple providers. A great thing about these browser infrastructure providers today is that they're extremely interoperable.

78. 16:17

    You can switch them in and out to test which ones do better. Um, it is all really use case dependent. Like, different browser infrastructure systems might have better CAPTCHAs for your use case or might have proxies that are or aren't blocked for certain sites that you're trying to access.

79. 16:32

    A great tip is if you're building on a specific set of sites and you feel like you're getting proxy blocked, just, like, shoot your browser infrastructure provider a message and they'll usually be able to unblock you, like, with, with some capacity on their side.

80. 16:45

    Um, so just really make sure you try different providers before you go all in on building some sort of production use case.

81. 16:52

    And I think the last key takeaway here really is, uh, try a hybrid approach. So, like, the report really showed us that although browser agents are good at a lot of things, they're not good at everything, and there's still some room to grow, right?

82. 17:07

    And when we talked to a lot of our customers and a lot of people building browser agents at scale, one of the big techniques that people utilize, especially for production scale use cases, is using a mix of browser agents for tasks or workflows that are more long tail, um, more dynamic, or change often, and mixing that with

83. 17:27

    more deterministic workflows such as Playwright that we know are reliable at scale and that can be, you know, more accurate for a large volume, right? And you can kind of think of this as like laying train tracks for the steps that need constant movement and accuracy, but using the agent for the more long tail diverse, you know,

84. 17:47

    roads and trails that require a little more, um, nuance and capability.

85. 17:54

    And I think, like, looking ahead, we fully anticipate the industry to improve a lot of these problems. Um, I think a few key areas where we're excited to see development on are first just better long context memory.

86. 18:07

    As you saw from Wyatt's stats, like, some write tasks take almost three times the amount of steps as read tasks, right? So having longer term context on how to execute those tasks will be incredibly important for accuracy.

87. 18:20

    Second is I think, um, there is still a massive opportunity to just build more, like, browser infrastructure primitives. Um, we talked about this, but, like, login and auth remains one of the biggest blockers to write-based actions.

88. 18:34

    Along with that, payments is also a huge issue that we haven't even really started touching yet, and I think people who can build these tools to enable browser agents to take these, like, primitive actions on the browser will unlock a, a huge amount of value.

89. 18:47

    And the last thing is that the models that power these browser agents are just, just gonna get better. Uh, and something that actually we're working on is essentially like training environments, uh, and like sandboxes to help essentially train models within a browser computer use environment setting to make them better at, example, tool calling or write actions, things

90. 19:05

    like that, right? So if you want to talk about that, you know, shoot us a ping.

91. 19:10

    But yeah, um, that's really the crux of this talk. Uh, the last thing we want to wrap up with are some really cool examples of browser agent actions that we saw while executing this benchmark that we f- found -- thought were cool and also a little scary.

92. 19:24

    The first is, um, we asked a browser agent to do some actions on GitHub. It got stuck, and it ended up actually talking with GitHub's virtual assistant AI to unblock itself, and this is a little bit of, like, AI agent inception that we thought was really comical to watch as it conversed back and forth.

93. 19:43

    Um, here is a really cool example where we asked a browser agent to post a comment on a Medium article, and that comment ended up being the top liked post.

94. 19:53

    Um, and this was, like, a really funny nod to the Turing test, and, like, really makes you question whether or not we've actually passed this moment.

95. 20:02

    Um, this is a fun example of we asked browser agents to book us some reservations at restaurants, and we kept getting, you know, notifications on our phone of restaurants being booked on our behalf by our browser agent.

96. 20:14

    Um, we went back and canceled those reservations, but it was just a funny, like, externality of real world browser agent testing.

97. 20:22

    And finally, the last one that we actually thought was probably one of the scariest examples, um, I think this was browser use, was blocked by Cloudflare when trying to execute a task.

98. 20:33

    And instead of, uh, just giving up, it went and actually searched for ways on Google to bypass Cloudflare verification. And this was a little bit of a like, whoa, these things are perhaps doing emergent behavior that we couldn't predict, um, without testing it in, in some robust capacity.

99. 20:51

    So yeah, uh, that's our talk. Uh, we hope you enjoyed and really excited for the space. Um, lots of development happening very rapidly, and we wanted to give you guys a current snapshot on what we think the current capabilities are so you guys can build better, and we hope to do this on a recurring basis, um, every

100. 21:06

     few months because everything's moving so fast. Thanks for your time. Have a good day.