Building Agent Interfaces: Lessons from Chrome DevTools (MCP) for Agents

Talk transcript

1. 0:00

   [upbeat music] Let's get started in, in, in interest of time, right?

2. 0:20

   So hi. Welcome. Let's talk about building agent interfaces today. So let me start with a question first. Who in here is already using MCP servers or CLI tools on your,

3. 0:32

   uh, agent? Okay, everybody. That is unsurprising, to be honest. [laughs]

4. 0:37

   Um, who in here have already built MCP servers and deployed them for effect? Okay, it's approximately half of the people.

5. 0:46

   Well, today I'm going to share four engineering lessons from the Chrome, uh, DevTools team on how we build Chrome DevTools for agents and how we deployed it for effect.

6. 0:57

   Quick context setting. Chrome DevTools for humans is used by millions of web developers on a daily basis to debug web pages. It's directly built into Chrome, and developers use it to debug web pages, find errors, audit it, performance profile it, and so on and so on.

7. 1:13

   Uh, right. So n-now let's talk a little bit more about Chrome DevTools for agents. So this is a purpose-built Chrome DevTools, but for agents. How surprising. Um, yeah. Let me briefly show you how it works.

8. 1:28

   So you can see on the left side, Gemini CLI and the prompt is being ent-entered. And now, uh, Gemini CLI has the MCP server being configured, and it opens Chrome on the right side and then does debugging.

9. 1:41

   Uh, uh, I think, yes, this is about performance tracing, so it does a performance trace, analyzes the trace that comes back or the performance insight, acts on it, and then makes the web page faster, uh, validate that it's actually faster afterwards, and it's done.

10. 1:57

    You should be done now. It's nearly done. Sorry, it's a video.

11. 2:01

    Whatever. What I wanted to tell you is like, yeah, this is going to work in any MCP client and, uh, agent harness that is MCP capable. Doesn't really matter.

12. 2:09

    That was Gemini CLI. Also works in Claude Code, Codex, OpenClau, doesn't matter.

13. 2:17

    If you want to have more information, go to this QR code because that QR code is going to, uh, bring you to a web page, and that's going to tell you everything about how you can install it, configure it for your agent harness.

14. 2:30

    Question again. Who in here has already tried it out?

15. 2:35

    Okay, so ten percent. Thank you. [laughs] I love you. I al-also love everybody else, but- [laughing] ... I love the others more. So I was rude. I didn't introduce myself. Uh, my name is Michael Hablich.

16. 2:48

    I am the product manager for Chrome Dev-Developer Tools, uh, at Google. And I'm also a guest lecturer at the university nearby, nearby where I'm living. I have twenty years of experience in tech, developer, tester, QA engineer, project manager, product manager, program manager, and so on.

17. 3:08

    If you have questions afterwards, please talk with me in the hallway track or, uh, connect with me over LinkedIn. Uh, the QR code will bring you to my LinkedIn page.

18. 3:18

    Both is fine. Uh, please do that. I would be super interested to talk w-with you about MCP servers, browse automation tools, and stuff like that. Okay, but now enough advertisement.

19. 3:29

    Let's move on. We ship Chrome DevTools because we saw that coding agents were flying blind. So like one and a half years ago, they were very good with generating code, but they were not able to validate what they actually were doing, right?

20. 3:44

    And it just sucked. But we assumed they are going to be fine if we throw a lot of data at them because, I mean, they are machines, right? The thing is, we were wrong.

21. 3:55

    So this is a, uh, this is the head of a trace file. A trace file, it has all the data about the performance profile.

22. 4:07

    And this is a file, like multiple megabytes of data, and this is like fifty thousand lines of JSON. And we did throw that against common agent harnesses at that point, like one and a half year ago, stuff like that.

23. 4:18

    And without surprise, this is too much data for an agent, for a model to actually reason about, and it blew through the context window. And if you have seen Matt's talk about the dump zone,

24. 4:30

    you're moving the agent into the dump zone at that point. So we thought, okay, we built it wrong. That's not going to work. We need to do something else.

25. 4:36

    So in that case, for example, what we did, our performance tracing endpoints, it can also return that for post-processing, uh, with other tools. But what it's really doing is it's returning Markdown now and semantic summaries.

26. 4:50

    Like this is a example of such a semantic summary, which just gives you information about typical performance metrics like Largest Contentful Paint, INP, and so on and so on.

27. 5:00

    I'm not going to bore you about all the, uh, performance metrics. And we are going to talk about them anyway because they're a very good example of how this is, uh, working.

28. 5:10

    Well, essentially, we didn't force the agent to read the entire book, the trace, but instead we just pointed it at the right sentence and this is the semantic summary.

29. 5:19

    That works quite well. In the end, uh, or in the beginning, agents are a different user class. So that's where when it's kind of like clicked for me like, ah, yeah, they are kind of like a separate user segment.

30. 5:33

    So how do we reason about that? The thing is, agents and humans, they share the intent, they share the goal, right? In our case, for example, both want to identify errors on a page and want to fix those errors,

31. 5:48

    but they think differently. They have different cognitive bottlenecks, more or less. For humans, it's a lot about visual complexity. So humans are very, typically very visual, uh, creatures.

32. 6:02

    And we need layout, we need color in order to find a signal. And this might not be the best example for a appealing UI, but this is just, uh, our console interface where console errors are being, uh, surfaced, and we can also use it as a wrapper interface on Chrome DevTools for humans.

33. 6:19

    But you can u- if you know where to look, you can use it to identify errors on the page very easily. Um, LLMs, on the other hand, they prefer, unsurprisingly, non-visual interfaces, right?

34. 6:31

    They care about schema clarity, data density. So, uh, on the right side, you see a schema for, I think it's for the performance endpoints that we just discussed earlier.

35. 6:40

    Um, yeah, and they really like that. It's a textual, non-visual interface, right?

36. 6:47

    Whatever. This now was my long way of saying, ah, designing for agents actually requiring a few engineering concerns I'm going to talk about today. So there's token burn rate, there's error recovery, there's tool discoverability, and there's also trust boundaries.

37. 7:05

    And I'm going to cover them today very shallow. Let's talk in a whole bit track more.

38. 7:11

    Let's start with token turn, uh, burn rate. So tokens are kind of like a cognitive load for agents, right? Very similar to humans remembering clicks, the right clicks on a visual interface.

39. 7:25

    The thing is, most APIs, uh, todays, uh, they're very often

40. 7:32

    like a fourteen-step checkout flow, as you can see on the left side, for example, for human, or the trace that I showed you before. Kind of like that sucks because every word that is sent back to the model is metered.

41. 7:43

    It costs money, right? Every word, every output token, uh, input token that it gets costs you money. Uh, every reason is happening costs you money. And that just sucks because that translates to monetary cost.

42. 7:55

    So how can we reduce that? Well, let me introduce you to a metric. Of course, uh, let's talk about metrics. It is called tokens per successful outcome. What it does, it balances effectiveness and efficiency.

43. 8:10

    Uh, sounds very hypey, and maybe it is. I don't know. Um, what is it about? So effectiveness is about does the agent complete the entire user journey? Is the functional intent actually fulfilled?

44. 8:25

    Yes or no. And then there's efficiency, which unsurprisingly is about token cost, tool calls, duration.

45. 8:36

    In the end, what tokens per successful outcome tell you is the s- the fuel efficiency of your interface, right?

46. 8:45

    And there's a caveat because there's always a caveat.

47. 8:49

    Fuel efficiency is relatively worthless if you can't reach your destination. So that's why it's called tokens per successful outcome and not tokens per outcome. So make sure that you actually also measure effectiveness, right?

48. 9:05

    And there's one more caveat, and this is

49. 9:10

    you can't measure that globally. I mean, you can do that, but you sh-- maybe you want to do that because it's a nice metric,

50. 9:17

    but it's going to ch-- uh, be tremendously different between different user journeys and task classes. So don't compare them globally, compare them within your user journey that you're measuring that.

51. 9:29

    And what I mean with that is like, for example, in Chrome DevTools, we have the user journey of web scraping, right? So an agent going to website and extracting information, that's relatively cheap.

52. 9:41

    But there's also user journeys that are more intricate, like debugging a website, finding out why the responsive layout is not working. That thing is going to use more tokens, but that is fine because it's a much more, much more intricate, uh, and more interactive session that's happening.

53. 10:01

    Okay, how does this look like in real life? Uh, this is what it looks like in practice and for a project, an internal project that we are working on.

54. 10:08

    And you see a lot of neon bars, which is great. I like neon bars. But I am not going to bore about details. The important thing is the neon bars on the left side, the longer the bar, the more effective a tool, right?

55. 10:23

    The shorter the bar, the less effective a tool for a particular use case. So each of those bars on the left side are about use cases. Uh, which means

56. 10:32

    the smaller bars are probably the ones that we should be focusing on our work next, how to improve that, how to improve the tokens per successful outcome there.

57. 10:42

    Yeah, uh, as you might have already guessed, measuring tokens per successful outcome is not straightforward.

58. 10:49

    The thing is, but even an imperfect measurement is better than simply doing gut-driven, um, decisions. And with that, at least you can do data-informed decisions.

59. 11:03

    Right. In-- Sorry, uh, I had my audio on.

60. 11:11

    In DevTools for agents, we're addressing, uh, token burn from three different, three different angles. First, there's tool categorization, so

61. 11:20

    very straightforward. We hide, hide niche, niche tools, we hide command line parameters. Like, for example, uh, we have tools for Chrome extension debugging.

62. 11:31

    And not everybody is developing Chrome extensions, so why add it to the default context window? There's no point in doing that.

63. 11:38

    Then there's a slim mode, and this one is fun. So what slim mode is doing is like, uh, pushing tool categorization to its limits. It's only exposing, I think, three different tools.

64. 11:50

    Select page, navigate page, and evaluate script. And this is great for a context window, but there's a trade-off. I'm going to talk about a lot about trade-offs today. Um, there's a trade-off Because the last tools you exposed, the last tools also have, uh, to dispose for your, for your agent harness, which means your agent might do extra

65. 12:11

    turns to achieve the same goal. It might not actually have the right tools at the-- that is supposed to actually do something like, for example, getting network requests. You can't do that with a lab-- evaluate script and stuff like that.

66. 12:23

    Yeah. And there's also a CLI interface. Uh, sorry, there's a command line interface that we're offering. Uh, you have seen the previous talk maybe about a code model and all that stuff.

67. 12:32

    We also support that. So it is a MCP server, yes, but there's also a, a command line interface for the same thing, giving you nearly the same functionality. What it enables you, you can have your agent chain commands together to do post-processing.

68. 12:47

    Like in this example, I don't think you can see it. Um, the accessibility tree is extract with a, with a grab, uh, command, and then the result, the ID of the control is being piped into a click command.

69. 12:59

    And that is, of course, sav-saving a lot of tokens doing that because the model doesn't need to process all the tokens. The token, the post-processing is happening on your computer.

70. 13:12

    Right. Um, efficiency is useless if your agents get stuck. So that brings us to error recovery.

71. 13:21

    And, yeah, because every time you were to-- your agent encounters an error, it's going to cost you tokens

72. 13:30

    because it needs to retry, it needs to understand what is happening and stuff like that, and that just sucks.

73. 13:38

    Yeah. Error recovery is a spectrum. And let's talk a little bit about that, what we are doing here. So first, of course, you should add,

74. 13:48

    uh, useful error messages. That sounds obvious. Uh, for a lot of tools, it isn't. [chuckles] And it was also not obvious for all the tools that we actually offered. So we also did, uh, a few iterations on them to actually make the er-- make the error messages good.

75. 14:01

    Like here, for example, uh, an unable to navigate back in a selected page for a particular tool, history entry to navigate, uh, was not found. We actually added the last sentence, and that enabled the agent to self-heal, which is super useful because then the agent doesn't need a human to actually fix the problems, but the agent can

76. 14:19

    s-self-fix the problems. Then there's proactive detours. So be-beneath each of the agents, there's a model, and the model is being trained on certain data.

77. 14:29

    And sometimes the-- there are things where you

78. 14:35

    want to contact the training data, and that's what you can do with proactive detours. Like in this example, um, we detour the agent for performance profiling to our start performance trace tool and not to the lightest audit.

79. 14:50

    And there's diagnostic playbooks. Uh, so we also offer, uh, skills, of course, and we have a skill that's called, uh, troubleshooting. And we see a lot of people have problem in setting up the Chrome DevTools, MCP server correctly, and that troubleshooting skill is then going to kick in and help the human and the agent to f-fix the

80. 15:10

    setup issues. Again, enabling self-healing of the agent.

81. 15:16

    And all of this increase the resilience of your product, uh, of the agent harness that you're building. And this is nice and helps you withstanding mistakes.

82. 15:27

    And now let's talk about discoverability, which is about actually preventing them. So our initial design had one monolithic tool called Debug Webpage. So we only had one tool, Debug Webpage.

83. 15:39

    And you could-- The-- Another agent could send a prompt there and tell it, "Hey, debug this webpage. There is some responsive layout is not working."

84. 15:47

    And it was neat from engineering per-perspective, but it didn't really work. Uh, so we decomposed it into twenty-five different tools. And we thought problem solved. Of course, it wasn't.

85. 15:58

    Because we traded that problem off to another, and that was agents now had twenty-five tools at their disposal. How are we going to find out which one to use when?

86. 16:09

    Well, let's talk about that. According to this paper here, uh, ninety-seven percent of MCP tool descriptions have quality smells.

87. 16:19

    And this matters because the schema is the UI for an agent. So let's make the UI better.

88. 16:27

    And fixing this is a trade-off. As I said, [chuckles] it's always a trade-off. Because of course, you can make the descriptions better, and that's going to increase your context window size.

89. 16:36

    So you probably don't want to have that, or maybe you won't. And also smaller models in particular are not that good with more descriptions because they get biased in using tools they shouldn't be using in the first place.

90. 16:48

    There's a trade-off space. Uh, read the paper. Super interesting. Uh, yeah. But there are a few things you actually should be doing. They are relatively uncontroversial, and that is, um, define purpose.

91. 17:04

    Clearly explain how-- what the tool core function is. Is this working? Ah, yeah, here it is.

92. 17:10

    Come on. Yes. Ah, it says, uh, okay. Clearly explain the tool's, uh, core function, and there's usage guidelines like provide clear activation criteria.

93. 17:21

    And how this looks like in, uh, Chrome DevTools for agents, for example. And, uh,

94. 17:28

    again, performance.trace tool, what we have in there as a, as a description is used to find performance, uh, front-end performance issues and core web vitals, LCP, INP, CLS. Why is this relevant?

95. 17:39

    LCP, INP, and CLS are web performance metrics, and an agent is able to make the connection, oh, I'm going to use that tool if I need to, uh, improve a page load, for example.

96. 17:50

    We're far from finished optimizing that because models and agent harnesses also keep on, uh, changing all the time. So it's kind of like an endless quest for minimum viable description.

97. 18:00

    Yeah, but it is what it is. You can supercharge all of that with skills. As I said, we also have skills, and that is great, uh, in particular, if you have more intricate workflows.

98. 18:10

    But again, there's a trade-off They are not free lunch. If you pile in too many skills, uh, you're going to, [laughs] to shift the problem and run into the same problem again.

99. 18:22

    Uh, agents are going to call your skills even if they shouldn't be calling them. Uh, your context window size is going to increase and all that stuff. The trigger of the shifting, it's not disappearing.

100. 18:33

     Okay, now we have optimized for cost, recovery, and discovery. Let's now talk a little bit about trust, because we don't want to have a backdoor in the system.

101. 18:43

     Chrome DevTools for agents has a feature called auto-connect, and it's kind of like it lets you as a human using your coding agent like Claude Code, share the screen with the agent like, "Hey, I'm stuck here.

102. 18:55

     Uh, please help me debugging the, debug that and fix the problem that I'm seeing here." A-amazing feature. I really like it. Um,

103. 19:05

     and users of course requested the feature, "Hey, why do I need to click allow all the time? I don't want to do that. Please remember my choice." And in a traditional user experience design, that would've been a clear win, right?

104. 19:17

     Because it's just friction, friction that you want to remove. In a world where you are delegating away work to agents and automating a-away agents, you need to think about trust boundaries.

105. 19:30

     And so that's why we actually designed it, uh, with... So that friction is actually by design because we didn't want to have that. And why? Let's talk about that.

106. 19:39

     There's a blog post from, uh, Simon Willison about the lethal trifecta.

107. 19:45

     QR code. You should read it. It's great. Uh, I'm not going to talk, uh, more about that.

108. 19:52

     And utilizing that, there is a free, uh, at least free tiers that I'm thinking about in browser, uh, browsing agents more or less. You have tier one, and that is the local development environment.

109. 20:03

     In a local development environment, you have the human-in-the-loop, and the human wants to grant access to the default Chrome profile, to the data that you already have access to, uh, to the agent in a

110. 20:12

     time-bound manner. And then you have, uh, tier two, and tier two is, uh, agents running in a continuous integration environment, so it's controlled environments, but they're separated away. At that point, you should be using data separation, things like containers of course, but also other things like separate Chrome profiles and stuff like that.

111. 20:31

     Um, if you want to connect to them, we also have a mechanism for that, and that is called, uh, remote debugging port. And third, there's agents with full internet access, and that is YOLO mode essentially, because every webpage out there is able to do some prompt injection attacks to your agent.

112. 20:49

     So make sure that it do the same thing as in tier two, but also in tier three, uh, make sure that they have the domain allow lists and dom- uh, prompt injection mitigations, all that stuff together.

113. 21:02

     Going back to the lethal trifecta, that's what we mostly reason about, uh, tier one, and that's where all those free things are coming together, so that's why we actually say no, the human actually need to consent every time.

114. 21:15

     Key point being is a local agent, tier one, and the browsing agent fleet, tier three, re-your research agents maybe, might share a tool like Chrome DevTools for agents, but they shouldn't share nothing el-else about your security model that you're having, right?

115. 21:31

     Okay. Let's wrap, let me wrap that up. User experience is evolving to incorporate agent experience.

116. 21:38

     An agent is just another type of user, segment of user, also with non-functional requirements,

117. 21:44

     efficiency, discoverability, security, stability, and so on and so on and so on.

118. 21:50

     I shared four takeaways from Chrome DevTools for agents, um, when we are implem-- why we are implementing that. That is measure fuel, fuel efficiency of your interface with tokens per successful outcome, turn errors into recovery playbooks, audit descriptions for intent, and never compromise trust for convenience.

119. 22:11

     Agents are our next users. Let's help them help us. And with that, I wish you a nice remaining conference. [audience applauding] [outro music]