MCP = Mega Context Problem - Matt Carey

Talk transcript

1. 0:00

   [upbeat music] Hello, everyone.

2. 0:17

   Welcome. Quiet down. Quiet down. Hey. Awesome. How is everyone? [cheers] Yeah, good? [claps]

3. 0:31

   Oh, thanks. Um-

4. 0:32

   MCP.

5. 0:32

   Yeah. [laughs] Wanna hear some MCP versus CLI debates? [laughs] Yeah, is that why you all came? Anyway, um, hello, my name is Matt. Uh, I work on MCP and agents at Cloudflare, and welcome to my talk.

6. 0:45

   It's all about how we can make every API a tool for agents. APIs exist in the wild, how can we connect them to agents and make them do things?

7. 0:54

   So I, I really love my job because every day I get to decide, like, uh, if an agent looked like this, would he do this, or would he do this?

8. 1:01

   And I think it's kind of fun. Um, and we often fluctuate between the two of them. Uh, someone does something s- that you think is slightly funny, and then six months later, we're all doing it and claiming it was the best thing in the world.

9. 1:13

   So yeah, it's a really good craic. But the main part of the, the role, I guess, and, like, what I end up doing day to day, is, like, how do we give agents hands?

10. 1:21

    How do we let them interact with the outside world? And you're probably familiar with something like this. This is tool calling, function calling. Uh, it's been around for a while now.

11. 1:29

    The LLM writes a function, you execute the function,

12. 1:33

    bash, bash, bash. The weather in London is eighteen degrees. It's not. It's, like, eight and it's freezing. Um, sad times. And then from there, we went from, uh, bundled tools to something like shared tools.

13. 1:44

    Like, people made tools in their agents, and the-- you probably-- this is, like, all, like, recent history, so everyone's probably aware of this. But before MCP, we had, like, uh, people would bundle all of their tools in their agents, and then they would keep them bundled in their agents.

14. 1:56

    And then if I was, like, trying to interact with Gmail or something, I would make loads of tools for Gmail, bundle them with my agents, and that would be it, and the next person would have to do exactly the same thing.

15. 2:06

    And then we ended up with this, like, big explosion of, uh, MCP and remote MCP about April last year. And the service providers were like, "Oh, we can, we can, like, uh, give everyone MCP tools, and then everyone can use the same standardized tools, and we just make it once, and we provide it as another surface for

16. 2:23

    people to consume our API." Maybe there's a CLI, there's an API, maybe there's like, I don't know, GraphQL API, um, and there's now MCP as, like, another surface. But this got a little bit fun because it was okay with, like, eight tools.

17. 2:39

    But then what happens if you added, like, a few more or a few more or a few more or a few more

18. 2:47

    or a few more? And now you s- you, you're like, "I wanna give an agent access to our whole API surface." And well, that ain't gonna, that ain't gonna happen.

19. 2:56

    Why is that not gonna happen? You've exploded a context window of the agent. You've, like, completely annihilated it. This is one point something million tokens. A-a-and this was the problem that we, like, came across a few, uh, well, around a year ago now.

20. 3:10

    We were trying to give access to the whole of the Cloudflare API to agents. You put all of the, the... You try and make naive tools out of every single API endpoint, and you fully explode a context window.

21. 3:20

    Our OpenAPI spec is two point three million tokens into tools. That's something like one point one million tokens, and that's, like, never gonna fly, even with, like, the biggest foundational models.

22. 3:33

    A-a-and in that time, we were like, "We know this is not necessarily an MCP problem, but it's how everyone else is doing it, so we-we're, we're gonna, we're gonna adapt.

23. 3:43

    We're gonna adapt. We're gonna improvise. Um, and we're gonna split up our API into lots of different product-based MCP servers." So you've probably seen this, like, uh, a company that publishes sixteen MCP servers potentially, and then users have to, uh, interact with the one that they want to use when they want to use it.

24. 3:59

    Um, there's much less context, but the user has to select, and most of the time there's kind of incomplete coverage. So, like,

25. 4:07

    uh, for instance, like, uh, one of our a- one of our product suites, we might have, like, six tools in our MCP server, but the total API maybe has, like, thirty endpoints.

26. 4:16

    Like, you've completely missed some coverage there. A-and this is not, like, fulfilling the goal of, like, how do we make every API a tool for agents? It's actually kind, kind of annoying.

27. 4:27

    Uh, so, so I think we did it all a little bit wrong. Um, well, in Cloudflare, we had sixteen servers very, very quickly. We were hovering around two and a half thousand endpoints.

28. 4:37

    I think we're actually at, like, two thousand six hundred API endpoints now. Um, but w-we, we basically couldn't split up all of these into all of our servers, and the users had to pick the ones that we wanted.

29. 4:46

    What we really needed was progressive discovery of tools. Who's heard of progressive discovery? Anyone heard? Yeah. Cool. And that brings us to the crux of the debates that everyone has on the online.

30. 4:57

    Um, and that is, like, how do we do progressive discovery? And, and, like, is MCP dead? Like, is MC- was MCP, like, a really bad idea? Um, uh, and I'm gonna say, like, I, I don't think it was.

31. 5:07

    MCP is a protocol. All of these can be exposed over MCP. We just shouldn't be dumping loads of tools into context. That's, like, the main thing. We shouldn't be dumping tools in context.

32. 5:16

    Um, and all capabilities, like, i-in the future, we might have prompts and resources more. Skills are basically resources. Um, and we just shouldn't be, like, loading all of those at once.

33. 5:25

    So there's, like, sort of three ways you can get around that problem. Uh, there's a CLI, which, uh, people really like. Uh, there's tool search, or there's a third one that we're gonna come to a little bit later.

34. 5:36

    Um, but, like, how does, how would a CLI work for agents? So this is a, a sandbox in the background, and if I use our, our CLI, and I do something like, uh, I just call Wrangler, we get a bunch of commands.

35. 5:51

    The agent can, like, read these commands, parse these commands, and be like, "Oh, I want to interact with the database. Let's, let's do wrangler d one, and maybe we wanna list."

36. 6:01

    List our databases, whatever. And then after some period of time, and some interactive process apparently, uh, we get, uh, we get, like, the databases I have on my account.

37. 6:12

    And, like, an agent can kind of do this, um, mostly. Uh, and it, and it can call dash dash help to get, like, uh, introspection on, like, which parameters it needs, and th-this, this mostly works, this mostly works.

38. 6:25

    Uh, it's used very popular by things like OpenCloude, and, like, people generally really like CLIs. But you need shell access. This is, like, the main thing. This is, I guess, the crux of this.

39. 6:33

    Like, you have to have shell access, and that's kind of annoying. So for things like Claude Code, they wanted a bit more of a structured way of doing things, so they have, uh, like, tool search.

40. 6:41

    They have a, a search tool which loads the tools that they need when they need them into context. So say I want to, like, create a worker.

41. 6:50

    Uh, what it would do is you, it would take the user, the user question, it would do some sort of keyword matching, and then it would add K equals, say, eight tools to context.

42. 7:00

    And then at some point, the LLM's gonna look at, oh, actually workers create, this is the one we need, and so we're gonna use that one. But the rest of them stay in context.

43. 7:07

    Maybe n- it's not eight, maybe it's six. They change, it changes, but, um, yeah. You end up with like two, two thousand one hundred tokens, and only five hundred of them are being used.

44. 7:14

    But, like, it works. It, it, it's, it works quite well. Um, you only load the tools that are relevant. Uh, and then this last thing is a blog post that, uh, Cloudflare published in the summer of, well, last summer.

45. 7:27

    And it's like, how can we, instead of doing, like, a, um, a static search tool, or instead of, like, enforcing an agent to need a CLI, how can we do something where we just let the agent write code, and we let the agent write code against our API?

46. 7:42

    And it turns out that TypeScript is actually, well, types are a very concise way of representing inputs and outputs, um, in, in a way that an agent can reason about.

47. 7:52

    So say you have all of these endpoints. You have, like, a getWorker scripts or a createAWorker or something like that. We generate these types, uh, and then we let the model, given these types, write some code against these types.

48. 8:06

    So here we're doing Code Mode List Workers. I hope you guys can see that. Uh, and we're gonna try and list some workers. So this might be, like, a user request to list workers.

49. 8:13

    The model generates this code, um, against a typed SDK that we generate from our API. You can generate them from OpenAPI specs. Uh, and then we can run that, and we can, like, list the workers that we have on our account.

50. 8:26

    And we could deploy a worker. That would be fun. Hello, world. And we could put it behind one of the hardest things to do at Cloudflare, which is so weird because it's such a powerful product, but we can add access, which is, like, our, our managed IDP.

51. 8:39

    And now this worker's secure behind access. Kinda cool. With, like, a access policy to only allow me into it and all of this other good stuff. Super, super easy.

52. 8:47

    And an agent can generate all of this code given our types. So this feels like a step in the right direction. Just let the model write code. We benefit from the model getting fa- better.

53. 8:57

    We benefit from, I don't know, or, like, our improving our OpenAPI spec. It's like that should be the source of truth.

54. 9:06

    But we had this, like, kind of weird thing where we thought this was awesome, and we were pretty stoked about it. Uh, but the clients didn't implement it.

55. 9:14

    And then w- and when I mean clients, I've gone into, like, MCP terms now. So the client is the agent. So we'll be referring to the agent as a client from now on.

56. 9:22

    Um, but it, so the clients didn't really implement it. And we're, like, a little bit confused about why this is the case. Like, this was sort of n- eight, nine months ago now, and it's a better way of interacting with, uh, with, with APIs.

57. 9:36

    Just let the model write code against the API, but they didn't implement it. And why not?

58. 9:40

    Uh, and that's because, like, running untrusted code is mega, mega scary. Like, i-if I had said to you a few years ago, "Oh, we're just gonna let a language model write some code that we're gonna gen- we're gonna execute for our users without looking at it, without reading it, without seeing what it does, that might have potentially,

59. 9:58

    like, secrets access." Ideally, it has some secrets access. You'd be like, "That's crazy. That's a CVE, right? Our C-- It's a CVE." Like, it's a vulnerability. That's a problem. [laughs]

60. 10:07

    Like, and now we're proposing you to do this. So it is quite scary. Well, lo-loads of things can go wrong. Uh, we could, it could read a file system, read some secrets that you don't want it to read.

61. 10:17

    It could exfiltrate those secrets into a network request, run infinite loops, consume all your resources, do, like, really scary stuff. Run a crypto miner, you know? That would be, that would be bad.

62. 10:27

    Um, and in the past, people have tried loads of things to let people run code-like solutions. So if anyone's ever written a DSL, um, some sort of, like, JSON spec about how to, uh, and to interpret that as code, that is basically this.

63. 10:40

    Uh, if you ever used one of those integration softwares where you have to do that, that is this. They just don't trust you to write code on their servers.

64. 10:47

    VMs also. People spinning up sandboxes to run code. Big sandboxes, big VMs. That is this. And also code review.

65. 10:56

    But it's kinda lucky 'cause we have a, a pretty cool primitive that solves this, and there will be other primitives that solve this. I just think this is the first, and so it's worth, like, uh, worth shouting about really.

66. 11:07

    Um, and this is, like, how do you run untrusted code in a way that's super safe for you and your infrastructure?

67. 11:15

    And it's kinda like this. So we can execute a worker from a string, and a worker is just a, like a little, is like an isolate, um, in V8.

68. 11:24

    There's many blogs about how all this works. I'm not gonna go into it super deeply. I'm just gonna show you what it can do. Uh, so for instance, we have this, like, um, this piece of code that was generated, and we're gonna run this piece of code that was generated, and this ran on the back end.

69. 11:39

    It didn't run in my browser. It ran in a dynamic worker that's fully isolated. And how, how, I guess how can I prove that to you? Um, if we do this one, we are trying to get some secrets here, process.env.

70. 11:51

    And if, if we print them, there are no secrets. And we also have this weird Cloudflare global. Ooh, interesting. Um,

71. 12:00

    if we turn, that was with node compat on. If we turn node compatibility off, we don't even have, we don't even have a process.env there, and it all errors out.

72. 12:07

    So- We can like influ- we have this like programmable sandbox. It's not quite a sandbox. It's like a very lightweight thing that you can r- put load code into it and then run it.

73. 12:18

    And I'll show some other options later. Like, it's not just us that has this, but we have one that we host for you and goes to like Cloudflare level scale.

74. 12:24

    If you wanna do billions of requests, knock yourself out.

75. 12:28

    And now, like here's one where, um, the agent's written some code that accesses like an external API. And if we run this one, it's like, "This worker is not permitted to access the internet via global functions."

76. 12:39

    Well, maybe we want it to access the internet, and now we can give it access. So it's a programmable sandbox with g- with programmable guardrails, and all we're doing here is like flicking a Boolean i- in the server.

77. 12:50

    That's, that's like all that's happening here. But you can provide like a more in-depth function to be like, "Only access things to these domains," and that's what we do on the Cloudflare MCP.

78. 13:01

    Um, if we go next. Oh, speaking of the Cloudflare MCP, this is where I really hope the demo works. [laughs] Um, so this is an MCP client, uh, in this slide.

79. 13:13

    And if we ask it a question, we're gonna get some- we're gonna get like a auth screen pop up.

80. 13:21

    And then hopefully all this works. Oh, insane.

81. 13:30

    So now we have like complete ... Well, we have read-only access to the whole of the Cloudflare API. All of my Cloudflare infrastructure, I have read-only access to. Um,

82. 13:39

    which is pretty cool. Uh, these are account IDs. Don't worry about them. They're not secrets in, uh, in Cloudflare world.

83. 13:46

    Uh, cool. So, so we just listed a worker, but you could do many more things here, like you can deploy workers from your command line, you can do what we did earlier and add access to something.

84. 13:56

    You could introspect your DNS. You could send emails soon. Um, you can do loads and loads of other stuff, like it's very, very cool, uh, what you can do here.

85. 14:05

    But you have access to the whole of the Cloudflare API, all 2,000 and something endpoints.

86. 14:13

    And I guess, like, this kind of brings up the question like, uh, like where, where are we going with letting agents access external tools? Like what does this look like?

87. 14:22

    Like you have people installing CLIs for everything and running it on their own, running it on their own machine. Maybe running it on a VM. That's kind of cool.

88. 14:29

    Um, you have us being like, "Oh, you could just run untrusted code in this like other, in this other place that's like really isolated." You have people doing tool search.

89. 14:40

    You have people rendering UI as JSON. I don't know. Um,

90. 14:48

    a- and I guess like my, my main thought is that, like we're gonna have so many isolated environments on the, on the web, and there's gonna be loads of infrastructure primitives that allow you to run this type of untrusted code on the web.

91. 15:00

    Because code is actually a very compact, compact plan. Instead of doing tool calls, you can have one tool called code, where the model generates the code of your choice, and then you run it.

92. 15:12

    And that code has so many more degrees of freedom than like an individual tool call. So it makes sense to me that as the models get smarter, this is what, this is what we will do.

93. 15:20

    And people will adapt their infrastructure primitives to do this, so there'll be so many more of this. And you see this starting with like Pydantic Monty, uh, Deno also, and like, uh, we also have it with Workerdee, uh, the dynamic workers that I showed earlier.

94. 15:32

    Like people, more people are gonna build these primitives because they're gonna become more and more useful. So like w- just like a little explanation. Um, this is Workerdee, uh, r- like spawning a dynamic worker in this sandbox and running some code to get a fib sequence.

95. 15:47

    You can do the same thing with Deno, with Deno run, with some

96. 15:52

    ch- questionable checking. I have no idea what that does. [laughs] And then you can also kinda do the same thing with Pydantic, uh, Monty, their new, their new, uh, code interpreter for running untrusted Python.

97. 16:03

    Because it's Python, we have to download Python.

98. 16:06

    Sucks. [laughs] This, this might never work. I actually have no idea.

99. 16:12

    Oh, there we go. Great. Uh, so maybe you can see like where we're trying to go with this. That there was, there was a previous time where no one would ever run untrusted code.

100. 16:21

     That was a CDE, like you would just immediately like you have to like stop allowing that. Uh, and then it seems like LLMs is actually really good for them to run unch- uh, for them to write code that you can run.

101. 16:33

     And so now we're building the primitives to actually enable us to do that, and it feels like we missed out on this whole part of the, the, the tech scene that like we've never tried before.

102. 16:41

     Like in the 1950s, when you wanted to run something on a computer in your local town, you printed out some punch cards, and you stamped them, and you gave them to the guy, and that was kinda like running untrusted code, right?

103. 16:51

     Like that was kind of it. And then when we went to the cloud, we got away from that, and now I think we're gonna go much more back to that, where L- your users can write code 'cause your users are AI, and AI is very good at writing code.

104. 17:05

     And that is how they're gonna interact with your platform, whether through MCP, whether even through like Bash and CLI, like I don't mind. Um, I think they're just gonna write code against your services.

105. 17:14

     And your services have to be ready for this. Like your APIs have to be ready to take a beating because they have to have good rate limiting. 'Cause I can run this in a for loop on multiple sandboxes at once and just hammer your API.

106. 17:26

     Like you have to have some way of protecting against that. Like this is the new world that we're, that we're now gonna be living in.

107. 17:34

     And that's like on the server side, on the, on the services side. Now, what's gonna happen on the client side? So I think that's almost even more interesting because that's the user-facing side of things.

108. 17:45

     Like the user's not gonna see the server. The user doesn't care. The user just, "Why, why is my agent not getting my Gmail emails?" Or, "Why has it deleted my whole inbox box?"

109. 17:53

     They're not gonna like... They're not gonna see that. But on the client side, like there's a lot of innovation that's gonna happen here, and I think we've stalled a little bit recently because actually building an MCP client in particular got really, really hard.

110. 18:06

     Like to, to, to actually build a client that was performative, that worked, you needed to manage stateful connections. You needed to manage resumability between those connections. Um, there's, uh- There's plenty of other reasons why building an MCP client was hard, but, like, it was a pain, an absolute pain.

111. 18:22

     And so people had, like, the most stripped-down clients they possibly could. They mostly offloaded to the MCP SDKs, which are quite bare bones. And n-no one was building these, like, more, um, unique UI experiences on top of that.

112. 18:36

     And I think that, that is gonna come, like, very, very soon. So the, the most, the most obvious thing is we're gonna have programmatic tool calling in the clients.

113. 18:43

     The, the previous slide we just did showing those, um, showing those sandboxes with Workerdee, Deno, and, uh, Pynantic, that is, like, just running untrusted code in a client. People are gonna do that.

114. 18:52

     If your client is remote, you're gonna do it like that. If your client is locally, well, just YOLO it, whatever. Just eval it, you know? And it's gonna be fine.

115. 18:59

     But more people are gonna do this programmatic tool calling. It's gonna happen. And because you're generating code, people are gonna save this code, and they're gonna save it in these, like, mini scripts.

116. 19:10

     And users might be able to decide, "Oh, this action that I just did, that the LLM generated for me, I wanna keep that for later." And then it will be a much faster.

117. 19:19

     So you can see things for things like, um, cron jobs. A user might set up some web scraping job, like, without any knowledge of how web scraping works,

118. 19:28

     and then it generates a script, and that script is ran, like, every day, every two days. And whenever it breaks, because web scraping's, like, pretty brittle, the agent will fix it and resave the script.

119. 19:38

     Like, this stuff is ha- is gonna happen, and I think, like, these saved mini scripts, they only work when you embrace, like, programmatic tool calling, but they really do work.

120. 19:46

     And then, and then the, the last thing, um, is we're probably gonna have many, many more clients. Because they've been so hard to make up until now, and it is gonna get easier, uh, there's actually only ...

121. 19:56

     There's not, there's not a huge amount of really well-used, um, MCP clients.

122. 20:02

     That's gonna change. And with that change, like, more people are gonna be able to make them. More people are gonna deploy agents to the cloud that end up being an MCP client.

123. 20:11

     And I think more people are gonna try and do this, like, stateless agent loop thing. Like, it, it was fine to have sandboxes for every agent, like, running Claude code locally if, if there were a, a million agents in this world.

124. 20:24

     I think when there are, like, 100 agents for each person ... Oh, hello. Let's not do that. Uh, that's gonna be, that's gonna start getting really tough, and you're gonna have to, like, embrace a cloud-native way of doing things, which means that, um, state has to be something you can turn on or off.

125. 20:40

     And this is, I think ... We're, we're nearing the end, but this is my last thing. It's like, um, I work a lot on MCP servers and on the SDK, and this is where I think that bit's going.

126. 20:51

     I think we're going to see MCP as a middleware, um, in an m- when you build an MCP server. When you build an API, when you build an API service, it will be a flag that you can flag on in your favorite framework.

127. 21:02

     The, the SDK itself is getting super, super lightweight, and I think by the end of this year will be, like, natively in every single, at least TypeScript, big, uh, full stack framework.

128. 21:14

     It will just be there natively. Because it will be so small, it will literally just express the protocol in itself, and it will be silly for them not to have it.

129. 21:21

     They'll just have a native integration, and they'll be able to do MCP is true on all of your APIs. And because all of the clients will be doing programmatic tool calling, you can express, like, your thousand APIs from one Next.js app and just do MCP equals true and expose it over, and them as tools over MCP as

130. 21:38

     well. And I think that, I think that will happen. Um, I've been thinking that's gonna happen for a while, but I think we're pretty, really close there. And the last blocker is, like, fixing the SDK really [laughs] so that it's capable of doing that.

131. 21:50

     It's capable of fitting in every single front e- every single bundle, really. And that's the plan. Um,

132. 21:58

     you can find out more at the ... We have a Code Mode blog post that came out pretty recently. It's how we gave agents an entire API in 1,000 tokens.

133. 22:06

     If you have a big API, you should probably do this. A-a- and you observability providers, please just do this, 'cause it's really, really good for people to access your data.

134. 22:14

     Um, and thank you. Um, try out npm i agents. Thank you very much. [audience applauding] [upbeat music]