Your Agent Is an Infinite Canvas

Talk transcript

1. 0:00

   [upbeat music] And even though I know some people are still coming in, I've also been informed to start directly on time.

2. 0:20

   So much apologies to anyone who is hustling up here from one of the talks downstairs. You should've known this was the best track and hung out here from hour zero.

3. 0:29

   But, uh, by all means, come in, have fun. Hi there. I am Rachel Lee, Rachel Lee Nabors. Nabors like nearest neighbors, but not spelled that way. I have worked on the standards that powers today web- today's web with Mozilla on Firefox DevTools, the W3C on web standards, like the Web Animations API.

4. 0:47

   I've even been a PM on Microsoft's Edge browser, and you may know me from my pandemic babies on the React team, React.dev and ReactNative.dev. All right. So I've spent the last three years consulting with AI startups and LLM/big browser companies, some of whose logos you may recognize, most of whom you will not, on all things web, AI,

5. 1:08

   and UI. Recently, I have just become newly minted Principal Developer Experience Engineer at Arize. Uh, have you ever had your CTO ruin your agentic flow with a slight change of prompt or an LLM migration?

6. 1:22

   Have you ever been that CTO? [laughs] Anyway, if that is you, you need to check out Arize. Eval's all your friends. Check them out. Now, before my big tech career, I made webcomics for teenage girls on a...

7. 1:36

   Well, it was a, it was a part of the iVillage Network website. I don't think anyone remembers iVillage, but it was kind of a big deal back in the day.

8. 1:44

   Uh, it had four hundred thousand weekly readers. It was powered by ads, and I learned how to build for the web so I could reach those people where they were.

9. 1:53

   It was 2006, and it was a mood, let me tell you. Everyone online was edgy. We were on MySpace, where we had [REDACTED:age] talking about music with [REDACTED:age], and as you can see, I was really into the goth and emo scene.

10. 2:08

    I was the only goth in my village. [clears throat] [laughs]

11. 2:12

    When I got into the web, I believed that the browser is not just a document reader, despite what the CSS wonks will try to get you to believe. It is an infinite canvas that can render anything, not just documents, but video, audio.

12. 2:25

    Whatever you need, there's an API for it, and I have been working to that end on the web ever since. I worked on that web animation API with the W3C, creating such insane demos as the interactive Alice in Wonderland, uh, [laughs] and generally made the web do things that it was not supposed to do. [laughs]

13. 2:44

    Uh- [laughs] And now I write about how agents and AI are changing how we browse and consume content on my little newsletter, uh, The Agentic Web. And I do a fair bit of shenanigans with agents myself.

14. 2:58

    This is, uh, ASSA. It stands for Antisocial Social Agent. It's an MCP app that lets you use Twitter and Bluesky next to each other without actually having to visit the sites or be subjected to other people's algorithms.

15. 3:11

    Anyway, check it out if you like it. Uh, I'm here today to show you how, with the power of the browser's primitives that already exist and have already been specced out right now, you can turn your agent into an infinite canvas, or at least get your website into agents, one way or the other.

16. 3:28

    So I have this habit every year when I'm thinking, "What do I wanna make talks about? What do I wanna do?" I look at the one thing on my laundry list that I keep putting off, uh, in this case, my webcomic site, the archive site that I put together at, like, 2010 and said, "Okay, you're good there.

17. 3:43

    I gotta go now." And it's starting to get 404s, starting to get some broken images. Maybe I shut down some clusters here and there, and it's like, oh dear, the CDN has failed.

18. 3:53

    Anyway, it needed some repair, and I was like, "Well, don't just repair it, make it future-proof." If everyone is working in agents these days, let's meet them where they are.

19. 4:03

    I'm always meeting my audience where they are, and I still get fan mail from the women who grew up with my comics, so maybe I can, uh, get them to try out agents too.

20. 4:11

    All right. So just as a cringe warning, before you go to rachelthegreat.com, 2000s were a vibe. Old comics, younger Rachel Lee. It's just a little bit cringe, okay? Uh, I want to honor the fans, uh, [laughs] and also I'm working a lot with copyrighted images here, and I don't wanna get sued by Marvel, so I'm gonna use my

21. 4:33

    own stuff. The nice thing about having a former career as an artist is I can do whatever I want with my art, no questions asked. Of course, I did have to hose them down with glazing to make sure nobody else would.

22. 4:44

    But that's another story. Check out the repo at the end of this if you want to know more about protecting your content from being used for training purposes. Now, I wanted to make my comics available and navigable.

23. 4:55

    One server, but three different kinds of clients, humans and browsers, humans and agents, and agents and browsers using the web on behalf of their humans. I know, it's complicated.

24. 5:05

    They've got, like, this polycule thing going on. So [clears throat] what we're gonna cover today,

25. 5:11

    MCP tools and transports, how to host them on the web, just like you would a website. MCP apps, how do they work? You've already been told a lot about this.

26. 5:20

    I'm just gonna show you a really cool example and point out the foot guns. And WebMCP, coming to a browser near you. Before we go, how many of you currently host an MCP server?

27. 5:34

    Like, on the web, not locally. [laughs] Okay. Well, I love you guys. Everyone else, you're gonna learn. And how many of you have made an MCP app?

28. 5:44

    All right. How many of you wanna make one? Uh, this is gonna be the next web pages. And lastly, has anyone here played with, uh, with WebMCP?

29. 5:53

    Okay. Well, we'll see if we can change that. So

30. 5:57

    you're here. I assume you know what an MCP server is. I'm, I'm not gonna go into them too much. Um-

31. 6:04

    So we're gonna get started with transports. Now, if you've built for the web, I guess I should ask, how many of you have built websites and hosted them in your lifetimes?

32. 6:14

    All right. So you're familiar, uh, you're probably familiar with HTTP, but you might not be familiar with the concept of transports. Turns out there's more than one protocol you can use to communicate from one surface to another.

33. 6:26

    Uh, transports are how MCP servers communicate with agents, and there's an important difference between when you should use either of these. So stdio stands... God, it sounds terrible when you read it out.

34. 6:38

    Uh, standard input output. You can call it- [laughs]

35. 6:45

    Studio. Like the Phil Collins song, 'cause we're in England, okay? Studio. [laughs] Man, we're gonna call it that for the rest of this one. In fact, my [REDACTED:origin] proofreader who was like, "I didn't think you called it studio."

36. 6:57

    I was like, "Dude, do you want me to call it anything else? I think we're gonna have problems." All right. So the server, it runs as a local process.

37. 7:04

    It's spawned by the client. This is why when you're wiring up a studio-based, um, MCP app, it's always like a string of JSON with a bunch of command line inputs that most of your users don't wanna use, don't wanna configure.

38. 7:17

    Uh, communication, it happens via, as I just said, standard input and output, and the server stays alive throughout the session. HTTP is a little different 'cause we're running over the web.

39. 7:27

    Uh, we are not touching, with a 10-foot pole by the way, any of the security or privacy concerns with either of these layers. That is beyond the scope of today's conversation.

40. 7:36

    But HTTP, the server runs as a web service. It listens at an HTTP endpoint, which you may be familiar with from API development, and communication happens via our best friend, HTTP post requests, and it works well with serverless setups.

41. 7:50

    You know, uh, Vercel keeps, and, and Cloudflare keep trying to sell you those edge functions. Well, we got something to put in them now. All right. With Studio, you have to update a config file with a command to spin up a local server, but with HTTP, your user's expe- oh, my God.

42. 8:07

    Uh, your user experience is as simple as going over to Yawn Settings. Okay, it is buried a little bit, but trust me, when you get... Oh, my God. [laughs] Please, if anyone from Anthropic is watching this, help.

43. 8:20

    Give it a name, and put in a URL, and it is now... Come on, hurry. Come on, come on, come on, come on. Past Rachel Lee. But you can see I can actually type the dang thing out.

44. 8:30

    It's very, uh, that's not how you spell it. There you go. [laughs] It's very... There you go. I put it at /MCP. You hit Add, and now there it is.

45. 8:41

    It's a connector. And I go ahead and go in, and I, I give it all the permissions to always allow because I know this one. But it's not easy, uh, if you're using an HTTP server for your MCP, uh, tools for the user to get them into their, in this case, Claude.

46. 8:57

    All right. So these are the tools I wanted to make available. Uh, it, they look a lot like what's available when they visit the website. You know, they're very similar to that navigation, and they all return structured data in the form of, uh, JSON, except for get transcripts here, and we'll talk a little bit more about this

47. 9:12

    [REDACTED:gender] in a little bit. But point is, list the comics. List the storylines, because the comics include different story arcs and endings and beginnings. List the characters. Search said comics, which is actually searching the transcripts.

48. 9:24

    That's a different talk. And search by character, because I love the cat, I wanna see all the cat. Uh, and the transcript is different. It returns markdown. Boom. All right.

49. 9:35

    Let's take a look at these babies in action. These are just the tools. We haven't gotten to the interactive part. Bear with me. I have a narrative arc. [laughs] [clears throat]

50. 9:46

    All right. It is a bit wordy. Everyone who's like, "Chat's the future," I'm like, "It is not the future." [laughs] This is like, thank you for having a long discussion with me, Claude.

51. 9:59

    Yeah, you did find that I make the comics. Good, Claude. Good baby. [laughs] [clears throat]

52. 10:05

    I would like to make a, a, a, I would like to make the full transcripts and texts available to the context for meta-analysis of themes and character arcs, et cetera, instead of asking Claude to, like, get transcript from 533 different comics.

53. 10:20

    It does, it is not an efficient use of context to do that. I would really love to use MCP resources, which are the perfect vehicle for this, and skills in so many other things.

54. 10:30

    But the spec is loosely defined, it is loosely implemented, and I cannot find a client where I could actively get to these resources. They're on the server. They just ain't visible in your UI.

55. 10:40

    So please, if anyone here is listening who builds, uh, an agent harness, go in. Even bare bones, I'll take it. Give me something to work with here. All right.

56. 10:50

    So I would love for, uh, one or more clients to implement resources. Agents really don't like using MCP tools to go add markdown to their context. If you've ever maintained a documentation site and you thought, "I will add the MCP tools to the documentation site.

57. 11:06

    That should solve the problems with the, the, the coding agent knowing how to..." No. No. It would rather die in a fire than go call, uh, your documentation by MCP tools.

58. 11:17

    So let, that is not how they were intended to be used by God. [clears throat]

59. 11:22

    So resources could also allow harnesses to decide when to pre-prime that context with docs. Like, you're switching over into React mode. Good. Go get all the React resources from React's MCP server.

60. 11:31

    Thank you. Goodbye. All right. Little tirade over. That was your public service announcement. Uh, tools, uh, they can return plain data, JSON, text markdown, but they can also return an interactive app, app, app, app, app, app.

61. 11:45

    So I wanted to put a comic reader in your agent, so while you're reading, you can read some things with some pictures instead of a long wall of text.

62. 11:52

    It's like, "Claude, you are not, uh, my, my best friend in crisis. Why are you wall-of-texting me right now?" [clears throat]

63. 12:00

    It's been said that chat is the lowest common denominator of the user experience, that it is to the future of agentic experiences what the CLI was to software. My mother programmed in COBOL, and when at school they were making us click, double-click on icons to access video games, she was like, "We do not do that in this

64. 12:19

    household. We open DOS, and we do CD blah, blah, blah .exe. That is how programmers do things." But we all know that you don't do that on an iPhone.

65. 12:31

    You tap, and you point and grunt, right? So this is probably a phase for us developers. There's a current what, what I call the chat box landing page or starfish design because it sits there, and it lets the user do all of the work.

66. 12:44

    And you have to have a lot of context in your head when you land. You have to kind of know what this thing can do. Like, if you're on Linear, you're like, "I know this is a ticketing system, and I'm pretty sure I want to see X, Y, or Z."

67. 12:56

    It's great if you're already familiar with the system, but otherwise, it can be a bit challenging not to have any visual cues or guidance. It's sort of like these text games were really popular back in the day, but I wouldn't want to have to do my taxes by being like, "Check mailbox.

68. 13:11

    Is there mail? Whose house is this?" That's not great. [laughs] That's a little, asking, asking your poor customer to do a little bit too much discovery. So fortunately, we have MCP apps, inline, in-agent, rich media experiences created by bundling of HTML, CSS, and JavaScripts into a single file.

69. 13:31

    This sounds so familiar. I'm having flashbacks to, like, DHTML.

70. 13:36

    So this is Get Page. Get Page is my MCP tool baby, and I love it very much. This is it in action. It returns an MCP app. I say, "Yo, I want to read Crow Princess, the comic," to the starfish, um, chat box of everything.

71. 13:53

    And of course, Claude's getting chatty, and it's like, "I'll help you read. Oh, let me just check all of my MCP things. Wow, I found it in the other comics collection.

72. 14:01

    Let me go Get Page that." Here we go. Okay, thank you. And check it out. I can show the commentary. I can show the comments. That's pretty cool. Those are available on the website.

73. 14:12

    It's everything the website can do. It looks just as good as the website. You can navigate forward and backward. But the fun part is you go up here, you can click on text mode, and boom, it shows you the transcript.

74. 14:23

    So you could just read it if you wanted to. That'll come back later. So yeah, it's like a little mini website, comic reader, but it's using all the resources from the website.

75. 14:33

    Okay. Now, how do you do that? Well, I define the tool as Get Page, give it the description. There's an input schema. Of course, I'm gonna have to take, uh, the comic's ID, the storyline ID, the page number, so it can really hone in on where to get this.

76. 14:50

    To be honest, it just needs a storyline ID and the page number. I'm gonna refactor that. Uh, and of course, all of this stuff is required, but most importantly, down here where the magic happens is you pass this little meta attribute, give it UI, and you point it, you point it at a universal resource locator.

77. 15:07

    Boom, right there. Love it. Okay. Important thing to keep in mind is that MCP apps are a little special. They're not like websites. They're, they're an island. Uh, they're a single HTML file, so everything that you need there has to be embedded, Base64.

78. 15:23

    If you're gonna link to something external like a cached CDN font file or something, you gotta add that to essentially a CORS policy. Uh, so this means you're not gonna be dynamically pulling in stuff from other people's servers unless you really are certain about that and enabling it in your app.

79. 15:38

    It's sandboxed in an iframe. You don't get local storage with this, so you're not gonna be tracking state across iframes. You're stuck with it. Um, async tool calling. Okay.

80. 15:49

    And no network access. Ask a, you have to ask server to do things for you with the callServer tool. This gets into a real mother may I situation, which I'll cover in the gotchas.

81. 16:00

    So what does this mean? Uh, it means that becomes super useful to build your MCP app with your design system. In this case, I'm sh- sharing fonts, CSS, uh, and everything from the same server, got the CORS policy set up and everything, and HTML and C- JavaScript components can be bundled.

82. 16:18

    I'm using, uh, Vite single file for this. So if you already have a design system, put your paw up if you do.

83. 16:26

    Okay, good. All the rest of you, oh, God, I'm sorry for you. Um, but a design system is a thing of beauty. All right, bonus round. [sighs]

84. 16:35

    Links require host permission. Remember I said, "Mother may I?" You can't just window open or, you know, href. No, no. No, no. You gotta be like, "app ref current open link, pretty, pretty please."

85. 16:49

    Okay. Uh, remember I said external resources needed CORS? Well, you got this thing called the CSP or content security policy. Fonts, images, other external resources, they're blocked by default, so without that CSP config, the browser would just not load those things.

86. 17:04

    So if you're getting a blank look, and you're wondering what's janked up, check your CSP. And unlike MCP UI, which you may be familiar with, MCP apps can call tools.

87. 17:13

    So I could do that forward and backward navigation with a tool call. But if you're doing that, remember, some tools were not meant to be viewed as a big fat pile of JSON.

88. 17:22

    You want to set, uh, visibility app there just to make sure that the model's not gonna try to use that and return text. It's not gonna go, "Oh, look at this lovely forward and backward.

89. 17:33

    Uh, I will use that." No. Soccer arm. Don't do that. All right. So people are still using browsers, right? There are smarter, uh, more accessible browsers coming out all the time, uh, big and small browsers are shipping with agents built right into them.

90. 17:49

    Sadly, they still struggle with navigating websites because they rely on taking screenshots and guessing how to use the navigation or traversing the DOM, which burns tokens chewing up XML, which is really, uh, not a great use.

91. 18:04

    Th- both of these are compute intensive 'cause you're using visual models, or you're just eating a bunch of HTML to find out where the next URL is. Wouldn't it be great if they just had a way to call the same functions and links you already have in your JavaScript directly from the page that they're visiting?

92. 18:19

    Well, with WebMCP, they can do just that. It makes every HTML page a mini MCP tools server. However, I just wanna point out that WebMCP is to MCP as a JavaScript is to Java.

93. 18:31

    Inspired by riding the hype wave of, but if you actually look under the hood, it's taking inspiration. It is not a one-to-one compliance with the MCP spec. It's really just, like, tools for agents in your browser, yo.

94. 18:46

    All right. So the specs might diverge. Keep that in mind. It comes in two flavors. Since my comics are not forms except for search, I use mostly the imperative model.

95. 18:55

    But with declarative, you can just add toolName and toolDescription attributes to a form. Boom. If you're using JavaScript or functions to process that form, you g- just basically told, uh, you expose that form as a tool to the agent.

96. 19:08

    Uh, great for sites with lots of forms already, if you're one of those people whose site is a dashboard. Using the imperative model, it passed navigator.modelContext.registerTool to par- with parameters, is great for API calls, workflows, uh, data transformation.

97. 19:23

    I used a lot of those. All right. So [sighs] now, if you f- if it's available, uh, if you find model context in Navigator, you can say, "I'm gonna register a tool."

98. 19:35

    You give it a name, give it a description, give it an input schema. My God, it's so similar to MCP. How amazing. How can this not be the same thing?

99. 19:43

    But you also pass it in execution, um, a callback. And in this case, this is a ... What do we do? Next page. This one, uh, basically grabs the link rel next from the page and passes it as the thing that you're supposed to navigate to next.

100. 19:58

     So you could call this to navigate forward and backward on the page with a browser without having to click around. Woo. This is new, so I can only help you imagine how it works, because the standards bodies are still fighting back and forth on it.

101. 20:12

     So I'm gonna show you how to use it using the MCP B extension. This is kind of like a debugging extension. You can see all of the little, uh, tools are exposed over there.

102. 20:22

     I got a little chat window, and this kinda imitates what people might see in Gemini or some future in-browser agent. In this case, I believe I am saying, uh, read this to me.

103. 20:33

     It returns the transcript, and then I might say, I might type in ... Oh, yeah, thank you. Love that transcript. I might type in, like, go to the next page.

104. 20:42

     Come on, go to the next page. Oh, I was speaking into the microphone at this point. That's fine. Anyway,

105. 20:50

     and when I hit, when I hit go, it takes me to the next page. That's just the model calling that function on its own. Yay. That's the wow. Uh, don't applaud yet.

106. 21:01

     So best place to get started to check this out is actually with this interview I did with the creator of WebMCP. It actually started at Amazon. This was something he made to get around a bunch of auth issues, and I love that I screen captured it.

107. 21:13

     Window.navigator.registerTool. Uh, s- there's standards community group, all very lovely. Um, it's not fully supported yet, but you find all the resources in the commentary on that article. So I started my journey with the agentic web with this talk, like, two years ago, and I was like, "The browser's gonna die."

108. 21:33

     But what people didn't take away from the clickbait was that the web isn't dead, gal- y'all. Like, it's just changing. It's evolving. CSS and JavaScript aren't just the language of the web.

109. 21:42

     They're the language of interactive experiences on agents. All right. Here it is in action. I wanna show you that there is a version of this that talks.

110. 21:55

     Brenda looks skeptical while Rachel continues explaining. Rachel, "Heh, Gore left out some important tidbits." Rachel, "What about how plant life will respond to the extra carbon dioxide?" Comma panel two illustration shows tall plants and vegetation growing vigorously.

111. 22:11

     I could keep going, but that's just the Web Speech API there. It sounds terrible. I'm sure you can do something great with ElevenLabs. It's the browser's built-in text-to-speech capability, zero dependency solution.

112. 22:21

     No, no inference required for that. It's right there waiting for you to use it, so are all the other amazing w- uh, APIs available on the web today. So all these APIs, web speech, animation, audio, canvas, Wasm, CSSN, more, they're already there waiting for you to make something with them, and I just can't wait to see what

113. 22:40

     you do with this infinite canvas. [clapping] [upbeat electronic music]