Scaling GitHub for your Agents — Sam Morrow, GitHub

Talk transcript

1. 0:00

   [upbeat music] All right.

2. 0:16

   Hello, London. [clapping] Whoo. And I hope everyone's been enjoying the AI Engineer Europe so far. For- Whoo. There's so many amazing speakers. I've been, like, watching talks and talking to people for days now, and it's been immense.

3. 0:33

   I'm Sam, I lead development of GitHub's MCP Server. And yeah, I'm here to talk about mostly challenges we've faced building and scaling our remote server, how we've overcome them.

4. 0:46

   And, uh, before I start, I just like, I like messing with people, so, you know, here, a quick show of hands. Who's used an MCP server?

5. 0:56

   Good, good. [laughs] Uh, who's used GitHubs? Who has a hot take? No, I'm just trying to... [laughs]

6. 1:02

   Um, [laughs] and, uh, yeah, who-- has anyone built a server or a client?

7. 1:08

   Oh, nice. Quite a few. Um, and yeah, has anyone contributed to the specification?

8. 1:14

   Aw. Oh, yeah. I got, I got one. That's actually the first one, I think, other than the MCP Dev Summit. There was quite a lot of them. [laughs] Uh, but, um, yeah, anyway, it's really awesome to see so many hands, so I'm glad that I've actually come to the right place.

9. 1:28

   But yeah, for, for GitHub, you know, our MCP journey started, well, at least in public, in April last year.

10. 1:35

    And, uh, we actually opened sourced our local MCP in April last year,

11. 1:42

    and we've just turned [REDACTED:age], so I'm super stoked by that. But, um, yeah, back then, right, there was a tremendous buzz. Uh, we were the most starred repo on GitHub of, of the particular week, and, uh, like, the exposure meant we got a high volume of public contributions, uh, rapidly filling gaps in plat-platform coverage that

12. 2:03

    people kind of wanted to add tools and things and, you know, not everything was perfect, right? After a m-month or so of new features, agents, in some ways, were getting worse at using GitHub, and context windows were getting blown out quicker.

13. 2:16

    And, uh, you know, we picked, I think, over a hundred tools, and certainly at the time, that was just too many.

14. 2:24

    Uh, LangChain had already produced research, uh, they published in February that year, you know, of the exact kind of problems we were seeing.

15. 2:33

    More tools don't make better agents. You know, they get confused and forgetful. Well, I say more tools, like, more context and more tools shoved directly into the context, to be precise.

16. 2:44

    But, uh, yeah, GitHub's a really expansive platform, and we provided tools, you know, for repos, issues, PRs, actions, projects, like, even more things. Uh, but the hard part of solving this was, like, we didn't wanna prevent users from having the tools individually that they needed, and they used.

17. 3:02

    Uh, and suffice to say, our user ba-base is pretty diverse. And probably even, like, on GitHub platform at the moment, there might be, like, one or two Claudes as well. [laughs]

18. 3:11

    Uh, and for the record, uh, there's a team of us who work on it. [laughs] It's not just me. Uh, and my team is awesome. But, uh, yeah, we-- so to try and fix some of this, you know, I quick-quickly added this thing, tool sets, which was, you know, a kind of grouping concept of related product tools, and

19. 3:28

    users could just pick which ones they wanted and configure it. Uh, I also, like, added a dynamic tool selection thing, uh, where agents could f-discover sets of tools and then turn on in chunks.

20. 3:41

    And, uh, we never released it, but I made a kind of RAG version of the same, um, you know, for kind of semantic tool search and discovery. But it, uh, like, what, what do you think happened even in spite of all this stuff?

21. 3:56

    Context float. [laughs] Everyone used the default settings. It was really annoying because, like, in a way, we had all these elegant solutions. Uh, all they did was require users to actually, you know, configure the JSON a little bit, and most users just don't.

22. 4:15

    Uh, maybe it's even partially a spec problems 'cause, uh, you know, for, like, every proposal so far for grouping to the MCP specification, for various reasons, has been rejected, and there have been several attempts.

23. 4:28

    Uh, and like, in a sense, like, every mode or configuration we add, y-you know, one could argue is papering over potential gaps, uh, like, or gaps in client implementations.

24. 4:40

    So like, uh, as an example, uh, we have a read-only mode, and, uh, roughly seventeen percent of our users use it, but it maps one-to-one to the read-only... Oh, sorry.

25. 4:51

    Yeah, the read-only hint annotation. But, like, no client exposes that as a method of filtering servers. I think some gateways now do, but anyway, it's a, it's an interesting, easy win for more enterprise use cases, where people often only want that.

26. 5:06

    But, uh, yeah, we needed to find better solutions to context reduction, and, uh, you don't need to worry too much about the specifics. This is dated now. But, uh, like, we started trying to optimize, and we looked at the use-- the usage patterns on our remote server.

27. 5:21

    And initially, you know, we cut the, uh, amount of con-context used by focusing the tools more specifically to the general case and based on usage to, like, about forty-nine percent reduction of the initial load.

28. 5:36

    And then we subsequently also grouped CRUD tools and brought that down even more. And I th- like, I think you get about forty tools if you use the default configuration, and then you can kind of expand or contract that based on your own preference.

29. 5:49

    But, uh, yeah. Like, it's easy to customize. And, uh, we've also, like, uh, recently had a massive push to, you know, reduce output tokens of a lot of tools as well.

30. 6:00

    And, um, in this example- You know, just by tailoring exactly what comes with the list pull request, it's like actually lost more than 75% of the tokens used in the output.

31. 6:12

    So, you know, in terms of how token hungry GitHub server is, like it's, it's, it's a moving target. We're constantly changing things that improve it. And, uh, if you haven't used it in a while, like it's likely very different from a few months ago even.

32. 6:26

    And, um, yeah. Anyway, like, and we haven't ruled out more advanced approaches like code mode, and we're always experimenting internally. But, uh, on the heels of this, we also dug into our data and we found some more opportunities.

33. 6:41

    So yeah, like, uh, we made a big push to reduce tool failures as well, and the success rate is roughly, I think, over 95% at this point. But, uh, like not all failure is preventable 'cause agents don't necessarily know which repos they have write permission on.

34. 6:59

    They still hallucinate. But, uh, we've been able to identify significant numbers of errors that could be overcome, mostly by encoding a sort of agent intent into our tool surface.

35. 7:11

    And, you know, you might have to make five API calls to make it more robust, but you know, in that case, we do that in the server side to reduce round trips 'cause that, you know, saves context, saves time, and usually, um, makes a massively better experience, you know, makes the agents more successful.

36. 7:32

    And yeah, we also started to run evals last year. Um,

37. 7:36

    I'm not gonna go into detail. The, that link takes you to a blog article that my colleague Xenia wrote about doing it. But, uh, like one of the gists is instead of micro optimizing individual tool descriptions, you know, you try to test them against each other to try and make sure that they're called at the right times

38. 7:54

    and not called at the wrong time, so that in the pull of each other, they don't fight for like, you know... Like the perfect tool description that makes the agent call it all the time is terrible, as is the reverse of that.

39. 8:06

    So you need to try and get that as tight as possible. Um, but yeah, this could be a whole other talk.

40. 8:14

    Security, on the other hand, is something that's like a kind of constant menace in all of this. I've seen lots of people talking about this, um, and it's a real problem in some ways for us because, you know, we have a lot of people using plain text access tokens for MCP in the wild, and, uh, usually they're

41. 8:33

    stored somewhere the agent can access. They're frequently long-lived, they're often overprivileged, and they're kind of sat there just waiting to be abused. Uh, end users, like I, I don't think they're choosing this, you know?

42. 8:46

    Like it's, it's actually hard to make configuration easy and secure at the same time, and clients have to make use of system key rings or encrypted storage and like VS Code does.

43. 8:58

    Uh, but, uh, the, you know, the MCP spec also provided a better way with remote HTTP, which, you know, is all the way back to April last year as well.

44. 9:08

    Um, and we embraced this, of course. Um, and we wanted to make secure connection path of least resistance. Uh, we didn't want users to have to download a local runtime.

45. 9:18

    And, you know, our remote server supports OAuth 2.1, and my team even helped add the proof key for code exchange support, which is commonly known as PKCE, to GitHub's authorization server to improve the security posture for client apps.

46. 9:33

    Um, but as I said, we hoped OAuth would be the path of least resistance, and, uh, again, perhaps some of you might know what happened.

47. 9:44

    Everyone expected us to support the dynamic client registration, and for us, like it created more problems than it solves because, like if you implement it kind of properly, it's hard not to have unbounded growth of app databases and challenges of how you would bucket them for rate limits, and there isn't a reliable app identity.

48. 10:04

    So we just considered it and rejected it, and like we feel like it's a, it was a well-intentioned mistake. And we're, you know, we're not the only s- authorization server to not support this.

49. 10:16

    And, um, uh, even, um, like MCP itself, right? It decided that client ID metadata is probably the way to go, and I can't promise that we're going to support it, but I promise that I am trying to get us to support it.

50. 10:35

    And that should make logging in like massively easier. But, um, yeah, more on that in the future. And also, speaking of security, some of you may have seen this.

51. 10:45

    Um, this was a fun day. [laughs] But, uh,

52. 10:51

    like, you know, Invariant Labs published this and, you know, like it's a correct sort of correctly done prompt injection exfil attack for getting private data out of GitHub. And, um, the thing is, it, you know, they called specifically GitHub's MCP server out, and I think that

53. 11:08

    we, you know, we do provide the tools that can enable that if you just kind of enable them all. But, uh,

54. 11:15

    it applies to almost every agent set up, whether they use MCP or not, or whether they use GitHub MCP, you know? Like the lethal trifecta stuff, which I'm not gonna rehash now 'cause I think many of you have probably seen it, or you can look it up, like Simon's, Simon Willison's blog post on that's excellent.

55. 11:30

    But, you know, uh, the utility of agents is in conflict, direct conflict with kind of protecting this stuff and it's like, it's an active space trying to work out how to prevent these problems, but, uh, it's not solved and it's very much not unique to GitHub, and we have users with wildly different risk profiles, you know?

56. 11:49

    Like, um, we, you know, we even have people that have like air-gapped GitHub enterprise server instances in like much more secure... And then, you know, uh, obviously the Cloud Bros, et cetera, are also- Just running straight to GitHub with, like, you know, probably full token access to the agent and everything, and that's kind of also interesting, right?

57. 12:13

    And, like, I'm, I'm not naysaying any of this. It's just, it's cool to kind of see what people do and see if we can actually support the different use cases and security postures while everyone experiments with this stuff.

58. 12:26

    And, uh, we also kind of use, like, lean on Auth to, uh, manage tools as well, and this is something I'm pretty happy with. Um,

59. 12:36

    if you log into GitHub MCP with a PAT token, that we just immediately filter the tools down by the scopes that the token has. You, uh, you don't have to do anything other than give it the token.

60. 12:50

    On OAuth, we support Step Up Auth, so, you know, you can get a s- we could return a scope challenge, and then it will interactively ask the user if they want to allow the scope.

61. 13:01

    And if you do, then you can, uh, like, continue the tool call. It doesn't fail, which I think is also nice. And then VS Code, for example, supports that.

62. 13:10

    And I initially worked on this with them just because they already have a token to use GitHub. And what they wanted was that if their baked-in token doesn't have permissions to use everything, that it, instead of just failing, there was a mechanism for users having a clean install and then an app scoping later if they need it.

63. 13:29

    And yeah, lastly, server tokens as well. Like, they didn't have a, like, on actions and things. They didn't have a user, so user-specific tools are kind of out there.

64. 13:38

    And then by removing those, we're just removing kind of constant sources of failure and wasted context at the same time.

65. 13:47

    Uh, we run a completely sort of stateless server setup. And, um, we have been using Redis for session storage. You know, it's standard observability and debug kind of stack.

66. 14:01

    Like, this is not a weird picture. But I guess one of the weird things for some people is a lot of people are running a stateful MCP server process in the singular and have kind of struggled with how you get it into this shape.

67. 14:14

    But, um, for us, like, we did a few things because it's very dynamic. But, like, one of the fun things we did is, um, we, uh, we actually make a brand-new, in the SDK sense, a brand-new server instance on every single request, and we add the tools to it at the start.

68. 14:32

    So whatever your configuration is, it just builds this, and then you get what you've asked for or what you're allowed to use because some things have policies that impact whether you've got tools or not.

69. 14:44

    Um, and yeah, like, we've been able to scale to, at this point, we serve around seven million tool calls a week. And, you know, we don't have session affinity.

70. 14:55

    Uh, the, even the sessions, we generally only use them to identif- it's the only way to identify the self-reported client identity that comes through MCP. So it's useful for us to understand, like, what clients people are using the server with.

71. 15:07

    So yeah, like, we use sessions for that. But, um,

72. 15:13

    yeah, we also have, uh, like, wanted to bring experiments to all of you and everyone. And, um, we have this thing that's in Insiders Mode, and all it, all it does is it, it turns on certain feature flags and things for experiments that we're happy to just ship to anyone who wants to use them.

73. 15:33

    And, uh, this just takes you to the documentation. But, um, like, an example of something that we haven't released generally yet but is on Insiders is our MCP apps.

74. 15:43

    And, like, just, you know, I, I set up the example before I came in, but, like, it's quite nice when you're talking to the agent to have the opportunity to kind of edit the AI-generated, uh, issue, especially if you're, you know, you're working heavily in professional open source stuff, and you wanna make sure that it's you posting,

75. 16:02

    and it's not gonna get closed as a sort of bot-generated thing. It, like, this is a nice human-in-the-loop thing that MCP enables, and, like, I much, you know, I, I wasn't sure how much I would like it at first, but then I've come to love it because I kind of care about how my issues and things are

76. 16:20

    received by people, and this is just a really great way to make sure that I can, I can check that. Um,

77. 16:30

    so yeah, like, in terms of where I think it's going, like, something along these lines. I think a near future, you know, server discovery will hopefully be automatic, and tool, tool use will probably become more compositional, like Bash or piping tools into other tools, streaming data through them, or like, you know, Cloudflare's code mode approach or Anthropic's

78. 16:49

    tool search tool API, which just landed in Claude Code a couple of weeks ago. And OpenAI recently added a similar API as w- so OpenAI added a similar API too.

79. 17:00

    And, uh, you know, I ex- I fully expect that, like, thousands of tools will be normal very soon. We're trying to iron out all the problems that prevented it in the first place.

80. 17:09

    And that'll probably reverse many of the fewer tools decisions. And, uh, users hopefully won't even have to know what MCP is. They'll just convey what it is they want to do, and the Auth setup and, like, you know, the tool selection, things will become truly autonomous.

81. 17:26

    And I don't think we're that far away from this, but we're, we're kind of in this experimental phase where we're not really there yet. But, um, I think harnesses like Py are also interesting because you can build a weird client that maybe optimizes this in a really good way yourself.

82. 17:43

    So I would encourage people to experiment with crazy clients. I, I feel like you, you never know, you could be, like, the next, um,

83. 17:52

    uh, [laughs] like, well, if you're super lucky, you could be, like, the next Claude, right? You could publish something that goes so viral it totally changes the agentic game. Uh, I wanted to end on a high and look at some numbers.

84. 18:06

    So- Like GitHub itself, it's actually got over 11 million Docker downloads of our standard IO server, which is by f- not, like, by far not the most used version of it either.

85. 18:18

    Um, we've got 126 contributors now, and over 2,300 issues and PRs, which it's been over seven a day f- like every single day for over a year now, which I do look at almost every single thing eventually.

86. 18:33

    So it's been like [laughs] quite a year. Um, I mean, I, uh, other ... Some repos have it even worse, but, like, I also love it, so please keep doing it.

87. 18:42

    Um, and yeah, we've almost 4,000 forks, which blows my mind. I kinda wanna know, like, the weirder things that people have done that they haven't contributed back. Uh, yeah, nearly 30,000 stars, and, uh, we're fast approaching eight million tool calls a week.

88. 18:57

    And GitHub itself is also facing a new challenge. [laughs]

89. 19:07

    This is really intense, right? And it shows no sign of slowing down. Uh, I still want you to keep opening issues and PRs for us. Like, we will cope, but, you know, this is new territory. [laughs]

90. 19:17

    And, um, uh, you know, everything's, like, mildly on fire for everyone, I think, these days, and it's just exciting and fun. But, uh, yeah, thank you so much for having me. [audience applauds]

91. 19:36

    I think I got, like, 30 seconds. I don't know if anyone has anything they wanna ask, but

92. 19:42

    ... What's, what's your take on piping tool calls? Um, I, you know what, I think, like, things like trying out MCP CLIs and things like that is a fun avenue.

93. 19:54

    I don't think it's entirely ironed out, but, like, one thing you can do, take the read-only tools from some MCP, wrap it in a CLI, and just give it a proper help, and just see, see how the agent does.

94. 20:05

    Like, stuff like that is surprisingly effective. And, you know, I, like, like I say, I want people to mess with this stuff. So I would encourage you to just try it if you're interested.

95. 20:15

    All right, I'm zero seconds. I will answer you, but in person, if that's okay. [outro music]