Remote MCPs: What we learned from shipping — John Welsh, Anthropic

Talk transcript

1. 0:00

   [upbeat music] [audience applauding] Awesome.

2. 0:17

   Thanks so much for coming. Um, I wanted to give a bit of a talk on implementing MCP clients and talking remote MCP at scale within a large organization like Anthropic.

3. 0:28

   Um, I wanted to give first a little introduction from me. Uh, my name's John. I've spent twenty years building large-scale systems and dealing with the problems that that causes, and so I've made a lot of mistakes, and, uh, I'm excited to give maybe some thoughts on avoiding some of those mistakes.

4. 0:45

   I'm currently a member of technical staff here at Anthropic, and I've spent the past few months, um, focusing on tool calling and integration and implementing MCP support for all of our internal, like, external integrations within the org.

5. 0:59

   And so, looking at tool integration with models, we've kind of hit this timeline where, um, models only really got good at calling tools, uh, like, kind of late mid last year.

6. 1:13

   And suddenly everyone got very excited because, like, your model could go and call your Google Drive, and then it could call your Maps, and then it could send a text message to people.

7. 1:22

   And so, there's this huge explosion with, like, very little effort you can make very cool things. And so, um, teams are all trying to move fast. Everyone's moving very fast in AI.

8. 1:31

   Custom endpoints start proliferating for every use case. There's a lot of, like, services popping up with, like, /calltool and /getcontext, and then people, um, start to realize there's additional needs of some authentication.

9. 1:44

   There's been a bunch of stuff there, and this kind of led to some integration chaos where you're duplicating a bunch of functionality around your org. Nothing really works the same.

10. 1:55

    You have an integration that works really well in service A, but then you want to use it in service B, but it- you can't 'cause it's gonna take you three weeks to rewrite it to talk to the new interface.

11. 2:04

    And so we're in this kind of spot, and the place that we came to at Anthropic is realizing that over time, all of these endpoints started to look a lot like MCP.

12. 2:15

    Uh, you, you end up with some get tools, some get resources, some elicitation of, of details. Um,

13. 2:24

    and even if you're not using the entire feature space of MCP, uh, as a whole immediately, like, you're probably gonna go extend into something that kind of looks like it over time.

14. 2:35

    And when I'm talking about MCP here, there's kind of two sides to MCP that in my mind feel a bit unrelated. There's this JSON RP-RPC specification, which is really valuable as engineers.

15. 2:48

    It's like a standard way of sending messages and communicating back and forth between, uh, providers of context for your models and the code that's interacting with the models. And, uh,

16. 2:58

    getting those messages right is the topic of huge debate on, like, the MCP, um, repos. If you're involved with any standardization process ever, you know how those conversations end going.

17. 3:09

    And then on the other side, there's this global transport standard, which is the stuff around Streamable HTTP, OAuth 2.1 session management. And global transport standard is hard because you're trying to get everyone to speak the same language, and so it's really nitty, but there's not a lot of, like...

18. 3:27

    Most of the juice of MCP is in this, the message specification and the way that the, uh, servers are interacting. Um, and so we started asking ourselves, like, "Can we just use MCP for everything?"

19. 3:37

    And we said yes, with the caveat that yes is for everything involved in presenting model context to models. Um, we have this format where your client is sending these messages.

20. 3:51

    Something's responding with these messages. Um, where that stream is going, it really doesn't matter. It can be on the same process. It can be another data center. It can be through a giant pile of, uh, enterprise networking stuff.

21. 4:06

    Um, it doesn't really care at the point that your code is interacting with it. You're just calling a connect to MCP, and you have a, a set of, uh, a set of tools and methods that you can call.

22. 4:16

    So, uh, standardizing on that seems useful. Um, standard-- Why standardize on anything internally? Um, being boring on stuff like this is good. It's not a competitive advantage to be really good at making Google Drive talk to your app.

23. 4:33

    It's just a thing that you need to do. Um, it's not your differentiator. Uh, having a single approach to learn as engineers makes things faster. You can spend your cycles working on interesting problems instead of trying to figure out how to plumb, uh, integration.

24. 4:49

    And, uh, if you're using the same thing everywhere, then, like, each new integration might clean up the field a bit for the next person who comes along. Um, it's, it's ov-overall a good thing i-in cases like this where we're, we're not really doing anything interesting.

25. 5:01

    We're plumbing context between integrations and things that are consuming the integrations. Uh,

26. 5:07

    why standardize on MCP internally? Um, I-- This is where I might make an argument to everyone that there's already ecosystem demand. You have to implement MCP because everyone's implementing MCP, so why do two things?

27. 5:19

    Um, it's becoming an industry standard. There's a large coalition of engineers and organizations that are all involved in building out the standard. Uh, all of the major AI labs are represented in that, so you, you know that as new model capabilities start to be developed, uh, those patterns will be added to the protocol because all the labs

28. 5:40

    want you to use their features. So I think the standardizing on MCP internally for this type of context is a, is a good bet. And one of the things you get with MCP is that it solves problems that you haven't actually run into yet.

29. 5:51

    Like, there's a bunch of stuff in the protocol that exists because there's a problem and a need, and having those solutions at hand when you run into them is really important.

30. 6:00

    So sampling, an example of where this might be valuable in your company, you might have four products that have four different billing models, uh, for reasons because you're building fast.

31. 6:09

    Um, you might have a bunch of different token limits. You might have different ways of tracking usage. This is really painful 'cause you wanna write one- Integration service to connect to your slides and how do you go and, like, hook the billing and the tokens up correctly?

32. 6:24

    And MCP has, uh, already has sampling primitives, so you can build your integration. You can just be like, okay, your integration sends a sampling request over the stream. Uh, the other end of the pipe fulfills that request.

33. 6:36

    You can go and hook it in. Everything works great. And so this is a thing that, uh, uh, a shape problem that might take you a bunch of effort, uh, internally without this, but you already have the answer kind of gift-wrapped for you in the protocol.

34. 6:50

    And so at Anthropic, we're running into some requirements converging. We're starting to see external remote MCP services popping up like mcp.asana.com, which is really cool. We wanted to be able to talk to those.

35. 7:01

    Talking to those is complex because you need external network connectivity, you need authentication. Uh, there's a proliferation of internal agents. People have started building, um, PR review bots and, like, Slack management things and just lots of people have lots of ideas.

36. 7:19

    No one's really sure what's gonna hit, so we're having a huge explosion of LLM-backed services internally. Uh, with that explosion, there's a bunch of security concerns where, uh,

37. 7:30

    you don't really want all of those services to be going and accessing user credentials, uh, y- because that ends up being an, being kind of a nightmare. You don't want, uh, outbound external network connectivity everywhere.

38. 7:43

    Um, auditing becomes really complex. Uh, and so we are looking at this problem. We wanted to be able to build our integrations once and use them anywhere. And so, uh, a model I was introduced to by a mentor of mine and a while ago is the pit of success, which is the idea that, um, if you make

39. 8:03

    the right thing to do the easiest thing to do, then everyone in your org kind of falls into it. And so, uh, we designed a service which is just a piece of shared infrastructure called the MCP gateway that provides a single point of entry and provided engineers just with a connect to MCP call that returns a MCP

40. 8:24

    SDK client session on the end. And we're trying to make that as simple as possible, uh,

41. 8:30

    because that way people will use it if it's the easiest thing to do. Um, we used URL-based routing to route to external servers, internal servers. It doesn't matter. It's all the same call.

42. 8:40

    Uh, we handle all the credential management automatically because you don't wanna be implementing OAuth five times in your company. Uh, it gives you a centralized place for rate limiting and observability.

43. 8:51

    Uh, I have an obligatory diagram here of a bunch of lines going in and out, but, uh, [laughs] here's a, a gateway in the middle. This is kind of the thing, just one more box will solve all our problems.

44. 9:01

    Uh, can I go next? Uh, where is my...

45. 9:08

    Yeah. Uh, so the, uh, the code that we have here, we just made some client libraries where you just MCP gateway connect to MCP. Uh, we pass in a URL, an org ID, account ID.

46. 9:20

    This is, like, a bit simplified. We actually pass a signed token to authenticate 'cause it's accessing credentials, but this is the basic idea. And then importantly, this call returns an MCP SDK object, which means that when new features get added to the protocol, you just update your MCP packages internally.

47. 9:37

    You get those features across the board. Everything works great. The same code seamlessly connects to internal, external integrations. When it comes to transports, uh, and this is a bit high level and hand-wavy because everyone's setup is different, um, internally within your network, it really doesn't matter.

48. 9:54

    You can do anything you want. We've got the standardized transport for connecting to external MCP servers, um, but really just picking the best thing for your org. So we went and picked, uh, WebSockets for our internal transport.

49. 10:10

    Uh, and here's just a quick code example. It's nothing special. We just have a WebSocket, uh, that's being opened. We are sending these JSON-RPC blobs back and forth over the WebSocket.

50. 10:19

    And then if I can make this scroll down,

51. 10:24

    w- at the end, we just pipe those, uh, read streams and write streams into a MCP SDK client session, and we're good to go. We've got MCP going. Um, you might want to do this with gRPC because you wanna wrap these in some multiplexed transport so you don't have to open one WebSocket per connection.

52. 10:41

    That's pretty simple. Also, uh, we have read stream, write, write stream at the end. Uh, starting to see a pattern here. You can do, like, Unix socket transport if you want.

53. 10:50

    You can just have, uh, messages be passed that way. Read stream, write stream at the end. MCP works great. Um, I threw in an enterprise-grade email transport implementation over IMAP, um, which is pretty much the same thing.

54. 11:02

    You just go through. Here is our s- server. We're sending emails back and forth. Uh, "Dear server, I hope this finds you well." [laughing] Uh, "MCP request start," and then we pipe those into a client session at the end.

55. 11:15

    And so it truly doesn't matter. Like, whatever it takes inside your organization is great. We set up this unified authentication model where we're handling OAuth at the gateway, uh, which means that consumers don't have to worry about all that, uh, all that complexity in their apps.

56. 11:33

    Uh, we added a get OAuth authorization URL function and a complete OAuth flow because you might have different endpoints. At Anthropic, we have api.anthropic.com, and we have claude.ai, and we might want those redirects to go back to different places.

57. 11:45

    But, uh, this is tied on the gateway. It's really easy to start a new authentication. Uh, a real advantage of having this put on your gateway is that the credentials are portable.

58. 11:55

    If you have a batch job that you're kicking off, um, your users don't have to reauthenticate to that. You're just calling the same MCP with your internal user ID, and they get everything added correctly.

59. 12:05

    Your also internal servers don't have to worry about your tokens.

60. 12:10

    Um, so your request comes in internally for us. We're hitting a WebSocket connection to MCP gateway, uh, with- Auth token provided as headers to that. Uh, that gateway receives your stored credentials.

61. 12:22

    You create an authenticated SDK client. You just pass in the bearer token to the auth header, uh, and then you're good to go. The MCP client receives a read stream and a write stream, and so you just plumb those read stream and write stream into your internal transport, and you're, and you're, you're good to go.

62. 12:41

    Uh, one of the things that this gives for your org that's not immediately obvious but is really valuable is a central place for all of your context that your models are asking for and all the context that's flowing into your models for your org.

63. 12:56

    Uh, there's some papers written on MCP prompt injection attacks. There is a general risk of, uh, models going and having access to Google Drive and deleting everything in Google Drive.

64. 13:09

    There's some need of, uh, enforcing policy. You might want to be able to, like, ban malicious servers, um, do some content classification on the request, see what's coming in, kind of give an audit.

65. 13:23

    And the really nice thing about this is that because it's MCP, all of your messages are in a standardized format, so it's really easy to hook into that stream and be like, "Okay, here is my tool execution message processor," or, "Here is my tool definition thing," or, "Here's my resource management."

66. 13:37

    And so the payoff that you get from this is, um, adding MCP support to new services is as simple as possible. You just go and import a package. Uh, it doesn't matter what language you're in.

67. 13:48

    We've got multiple languages internally. They all have their own kind of packages. Engineers can focus on building features and not plumbing. Uh, you have the operational simplicity of having a single point of ingress, egress, and standardized message formats.

68. 14:01

    And you get future features for free. As the protocol evolves, you get all of that work, uh, naturally.

69. 14:08

    And so just wanted to go through some takeaways from this that I, I want to p-put to you is that MCP is really just JSON streams, and how you pipe those streams around your infrastructure is a small implementation detail.

70. 14:22

    It's a couple of lines of code to hook the stream into the client SDK that makes the messages. Uh, the, uh, you should standardize on something, anything. I think MCP is a good idea.

71. 14:34

    If you don't think it's a good idea, like, just pick something. Um, your future self will thank you. Uh, build some pits of success. You really want to make the right way to do a thing the easiest way to do a thing, and then everyone just falls into doing the right thing naturally.

72. 14:46

    And also centralizing at the correct layer, so solving some shared problems like auth and external connectivity once allows you to spend your time working on, uh, more interesting problems that are more valuable to you and your, your business.

73. 15:00

    Uh, thanks. That's all I got for you. Uh, thank you so much [audience applauding] for coming out. [upbeat music]