MCP Is Not Good Yet — David Cramer, Sentry

Talk transcript

1. 0:00

   [upbeat music] Um, welcome everybody.

2. 0:17

   It's a little bit last minute, so bear with me. If you don't know me, uh, I started Sentry a long time ago, David Cramer. I'm sort of an engineer, sort of an executive, sort of a founder.

3. 0:26

   Uh, I would like to think I have rational opinions, so that's mostly what this is. Um,

4. 0:33

   I don't think you're gonna learn anything here. Maybe you will, I don't know. I personally think this is not that complicated, it's just big scary words. So, if you do, great.

5. 0:44

   If you don't, maybe you walk away and you're like, "Yeah, I thought that's what it was." We've done it.

6. 0:50

   Mostly, uh, I was asked a couple days ago while I snuck my way into this conference if I could fill a slot, and so filling a slot was like, "Oh, come give some hot takes, maybe spice it up a little bit," so that's what we're gonna do.

7. 1:01

   It's not gonna be too much of a rant. If you know me, I like to rant, but, you know, we'll, we'll dial back for this one a little bit.

8. 1:06

   So, what is an MCP? Uh, you know, I, I gotta say, this is like one of the wildest phenomenons I've... It's like the new crypto wave or something. Everybody's like, "Yeah, MCP, we don't know what it is, but we're here for it."

9. 1:18

   And you find a lot of these sort of like opinions around, you know, how it should be, how it shouldn't be, and, uh, you know, what I often find is people who have these opinions have not built anything, or at least not built the thing they're talking about.

10. 1:29

    I built Sentry's MCP server mostly as a fun project, um, so take this for what it is. Um, it's also Sentry's MCP server. These are biased opinions towards what Sentry is.

11. 1:38

    If you're not familiar with Sentry, you probably should be, but we do application monitoring. We do a bunch of stuff. Um, if you have bugs on the internet, they probably go to us.

12. 1:48

    Uh, and so it's in context of a B2B, a SaaS business. A lot of you probably work at enterprise companies, if you will, so think about it that way.

13. 1:55

    But the way we think about MCP is it is a pluggable architecture for agents. Full stop. That's it. It's pretty simple to reason about.

14. 2:03

    And again, all of this is contextualized in an enterprise cloud service kind of way. There's a lot of other variations of how you might adapt MCP. There's tool chains that make sense locally.

15. 2:14

    We're talking about we run cloud services. That's most of the industry. We're B2B, we're enterprise. I think a lot of this actually still applies, um, but take that with a grain of salt.

16. 2:25

    So, how we think about MCP with Sentry, particularly because this, again, relevant here, we fix bugs. There are things like Cursor, where you also fix bugs. What if we could all fix bugs together?

17. 2:37

    And so everything's contextualized in that, and I think

18. 2:40

    there's this whole thing of like, how do we be relevant? That's like the, the name of the game for every single company in the world right now. Um, it's like, "Oh, how, how do we become an AI company?"

19. 2:49

    We too are now an AI company. Um, but Sentry has a lot of bugs. I fix those in my editor. Wouldn't it be cool if the bugs could be inside my editor sometimes?

20. 2:59

    That's a great example of where maybe an MCP is useful, but at the very least we're gonna pretend it's useful. So, that's the context here. Um, but it all comes back to like probably the reason everybody's here is like, "How do I become relevant?

21. 3:10

    I've got an AI mandate. I've got infinite money to spend all of a sudden for some reason that didn't exist yesterday. How do we get involved?" Okay? So everybody, probably same stage.

22. 3:18

    I know how this works, so. All right. Um, we built this a few months ago. We are not first to market with an MCP,

23. 3:28

    and the reason why is because the- there's two interfaces for MCPs. I'm gonna focus on a remote interface, but there's also the standard I/O. You probably learned about that or know something about that.

24. 3:36

    I don't think standard I/O is super useful for businesses like ours. I'll talk about that. But, but sort of the analogy of why MCP is useful, and this is VS Code Insiders, which you just, uh, heard from Harold, but like they do a pretty good job.

25. 3:48

    They're the only ones with OAuth support that's like useful today. Cursor promised me end of week, and I'll hold them to that. Um, but it works pretty well. You plug in Sentry's MCP, you can look up data from Sentry in a bunch of curated workflows.

26. 4:00

    You can maybe fix some bugs. Maybe easier than it was before, or at least more fun than it was before. Um, and for the sake of this, I- I needed a screen grab, so last night I'm like, literally last night I'm working on these slides and I go into VS Code.

27. 4:11

    I'm like, "I'm just gonna plug it in. I don't have time to futz around if the thing's gonna break," and so I use the VS Code. And I'm like, "Okay, I'll just do a thing where it's like fix all my bugs for me." [laughs]

28. 4:20

    And then immediately it does like 20 API queries to Sentry. Probably cost me like five bucks to run this thing. Um,

29. 4:27

    but it did start fixing some bugs. Uh, I don't know if the fixes were good, mind you. They're, they're probably garbage. Um, but it, it does a thing, right?

30. 4:34

    It's like it brought context into the editor, which is what we want, and that context was provided by somebody else, Sentry in this case. So that is like one of the interesting things.

31. 4:43

    It's one of the interesting things we think about and why MCP is like in- like valuable to sort of a traditional, uh, I don't know, we're kind of an enterprise company, but like, uh, like every company in the world.

32. 4:53

    And that's part of why we're all hopping on it. It's pretty accessible, and then that's what I'm gonna talk about. It is actually super accessible. So,

33. 5:00

    this is you. This was me. Um, and this is why I have opinions about it now. It's like, oh, it's just an API, it plugs in. We've got an API, we've got some OAuth going on.

34. 5:11

    You know, we had our own OAuth provider. You know, a lot of you might use something like a WorkOS or, I don't know, pick one of these authentication services that just gives you it out of the box.

35. 5:18

    If you have that, you're pretty much ready to go, which is pretty cool. It's actually like a pretty low boilerplate, uh, implementation. Um, but then you quickly learn that it's actually not that easy.

36. 5:28

    And so first you kinda go into this OAuth like dance and you're like, "Oh, okay, like, uh, yeah, we're gonna do this," but it needs OAuth 2.1 and nobody in the world supports this thing.

37. 5:36

    Like, it's like, I don't know how old it is, but I had never heard of it before MCP. Um, and so there's a little bit of complexity there. But you're like, "Okay, it's almost there.

38. 5:43

    It's OAuth. We've got that. We can plug it into our API." You kinda get it working. In our case, we use Cloudflare Shim, which basically lets us proxy our OAuth 2 API on top of Cloudflare Workers, which has a 2.1 client registration thing.

39. 5:56

    I don't know if anybody's talked about that. TLDR, it's complicated. Um- But it's not that complicated. This was built in a couple days, mind you, and I'm also an executive at the company, so it's like, uh, if I can do it, everybody can do it.

40. 6:08

    Um, but you go through the OAuth flow, and then you're like, [claps] "Cool," but the, the robots don't know actually how to reason about giant JSON payloads that were not built for them.

41. 6:17

    And this is actually where I think a lot of people break down. There was like a big conversation. This is sort of one of my first opinions, if you will, what I might call common sense, is that MCP is not a thing that just sits on top of OpenAPI.

42. 6:30

    Like, you cannot just be like, "I got an API. I'm gonna expose all those endpoints as tools." You're gonna get the worst results you can possibly imagine. You're gonna be like, "Oh, this doesn't make any sense."

43. 6:38

    You have to massage everything. You have to design around the system. But, like, generally speaking, and I'll talk a little bit about this, like y- you need to really think about how would you use an agent today?

44. 6:46

    How do the models react to what you do when you provide them context, which is what this really is for, and design a system around that. So might leverage your API.

45. 6:54

    It is not your API. And then you get past that, and you wire it up to things like Cursor and VS Code, and you're like, "Why is this breaking all the time?"

46. 7:02

    You can't, you can't, uh, solve for that one. It's just... You gotta wait for everybody to catch up. Um, they're almost there. Uh, you know, handful of clients support native authentication now.

47. 7:11

    They're kind of stable. Um, to VS Code's credit, it hasn't broken much recently. Cursor's broken quite a lot on me, but they're both great. Don't get me wrong. Claude has support.

48. 7:20

    Claude Code has sort of support, but not really. Um, so I guess it might work, it might not. I think particularly in the developer ecosystem, we're much more ahead of the curve.

49. 7:31

    And so if you're trying to adapt your services to third-party agents that are in our ecosystem, like these editors, you've probably got a good shot of it working tomorrow.

50. 7:41

    If, I don't know, it's Salesforce or something, I have no idea. So, so you're, you're kind of beholden to, like, the clients and the implementation because again, it's a plug-in architecture for agents.

51. 7:50

    Um, there's a lot of other use cases that are not just third parties, but that's kind of the focus. And so I'm gonna try to be constructive from here.

52. 7:57

    Let's see. We got nine minutes. Um, just a few learnings, and I'm happy to talk more about this later. I'll be around. Um,

53. 8:05

    you'll probably... Somebody in this room is gonna disagree with this, but you should only care about OAuth if you're a B2B SaaS company like me. Um,

54. 8:13

    and particularly you care about OAuth with remote environments for the most part. If you're like, how do I integrate my services into various agents?

55. 8:22

    I want bugs to exist in Cursor. I wanna run a cloud service, and I wanna run a cloud service for the exact same reason I've always wanted to run a cloud service.

56. 8:29

    Because I can iterate on it. I can ship fast. I can dial in security. All of the advantages, it turns out, are exactly the same because technology has not changed.

57. 8:37

    And so if I were you, and you're not building something hyper-specific that is like a local device-centric thing, just focus on the remote MCP server, focus on the OAuth specification, and just, like, don't worry about it.

58. 8:49

    The, the problems will solve themselves. Security will solve itself 'cause there's a whole world of security problems, and the standard I/O interface is filled with most of them. Um, I'm not gonna talk about that.

59. 8:58

    I'm sure there's some other talks here about prompt injection, but it is like very, very, very scary. Do not allow random MCP tools in your organization. Um, trust people that have earned trust.

60. 9:09

    Don't download random packages off the internet. Uh, it will be a very bad time for your organization. Um, I did mention this. Claude Desktop has, I think, full OAuth support right now in production, in GA.

61. 9:19

    VS Code Insiders has it. Um, these are great because you just drop in the MCP URL, and it handles everything from there. Cursor, like I said, I think this week.

62. 9:26

    Um, I don't know about anybody else. I don't pay attention much beyond anybody else, and I think Claude Code has not... At least I've not seen anything. Um, and then there's a bunch, like a long tail, right?

63. 9:35

    So works pretty well. There is this mcp-remote package, which is how we shipped all this stuff. It works okay. I applaud early adopters for getting this out. It's not a great experience, and you'll find a lot of this is not a great user experience.

64. 9:47

    It's rough. It's beta. That's fine. Um, this is the biggest thing, going back to OpenAPI then. You actually have to spend the, the calories. You can't just be like, "Ha ha, we proxied OpenAPI and exposed it as tools."

65. 9:57

    It's gonna do nothing. And so what the right answer here is, who knows? Um, our version of this, and I'll talk a little bit about why, is like

66. 10:06

    we return Markdown. We've, we've taken some API endpoints, and we've directly translated some of the response to Markdown. But it's intentional. It's like, I wanna get a bug out of Sentry.

67. 10:16

    I'm just gonna give you the bare essentials. I'm gonna give it in a structured way that a human can reason about because generally speaking, if a human can reason about it, the language model can reason about it because it's effectively pattern matching on language.

68. 10:27

    Um, it can kind of figure out JSON here and there, but if, if you actually push it, you're gonna find it breaks all the time. So just use something like Markdown.

69. 10:34

    Um, it's not scientific. I think there's a lack of science in a lot of this. It's hard. Just go with whatever works. But you, you have to really think about, you don't control the, the consumer.

70. 10:42

    You don't control the model, and so you're kind of like this least common denominator thing. And so think about that, but you need to design the system, and you need to treat it as like you are providing context to an agent that you don't know what the agent is doing, right?

71. 10:54

    And so that's the name of the game is context. That same thing, uh, sorry, here's an example of that. Forgot what my slides were. Uh, we just, like, give kind of a reasonable description of tools as the first version of context, um, which sometimes you hit token limits with all this, so there's some other challenges.

72. 11:09

    We give a reasonable description of a tool with the hopes that clients figure out how to make use of this context. So it can call the right tool. It can call it when it needs to.

73. 11:18

    It can choose one tool over the other tool, which is a really, unfortunately, hard problem for it to figure out. Um, mostly straightforward. Errors, same thing. You gotta design the errors.

74. 11:27

    They are still context because just like a human can't figure out how to call your API, the machine also can't figure out how to call your API. In my example, I'm like, "Fix all my bugs for me," and it, it queries like every organization in Sentry that I have access to.

75. 11:40

    It queries all... It's like, like 20 API calls when it should've been one, even with all this context. So the... We are a long ways from this being great, but it's like a glimmer, right?

76. 11:48

    So, you know, in this case, it's like, oh, you didn't pass the, the thing, or rather you passed an invalid value for the thing. Give it a real human response.

77. 11:56

    This is now more important than ever because again, it's not just a sort of machine reasoning about it where you can hard code all this stuff. It's abstract. You don't know who's reasoning about it.

78. 12:06

    The biggest thing, and this is sort of leading to like the, my overarching view of the world, is like you don't, you have no control, which already is a problem.

79. 12:13

    You are also passing the cost on in a lot of these cases. So you actually kinda need to be mindful. So another reason to not just be like, "Here's my API, I'm just gonna return everything to you."

80. 12:21

    Because all of a sudden, you know, that, that call, if you will, that tool call that could've been a dollar might be $10 now because of the amount of tokens you needed.

81. 12:28

    And more importantly, it might just not work. Like early on in, and I don't know if VS Code and/or OpenAI, I don't know who's to blame, fixed this, but like there was, and may still be, a limit to the amount of tokens or description lengths of tools.

82. 12:42

    Makes sense, right? You wanna constrain the cost of every API call, but all of a sudden now you have problems again. So you gotta be really thoughtful about this.

83. 12:48

    This is gonna be evolved. And I think the big thing is like if you build one of these, it's not set and forget. Like I- we're still updating this thing every week, tweaking it here and there, trying to look at like what's happening and evolving it, right?

84. 12:57

    But the biggest thing, and this is sort of my, my takeaway, my, my very, very strong belief is like you just need to really focus on building agents. MCP is a plugin architecture.

85. 13:05

    There's a lot of value behind it, but the like the inherent value of a lot of what LLMs are bringing is this sort of agent architecture, which by the way, is just a service architecture with a fancy new word on it.

86. 13:14

    Common sense kinda stuff, right? Um, and so we've done this in Sentry. It does not work well with MCP yet for, for what it's worth. There is no streaming responses for tools yet, and that's a big problem when you think about sort of this agent to agent, and I don't mean this in like the Google way.

87. 13:27

    I mean in like the generalized point of view of agent to agent. Um, but it gives you control, and it's, it's the same as all software. If you have control, you can be responsible for the success, for the failure.

88. 13:37

    I can be responsible for the prompt that dictates how the tool is called. I can be responsible for the result from the tool. I can make many calls behind the scenes and wrap those up.

89. 13:46

    So I, I just get a lot more control if I pick up the cost of that agent. I control the model even, right? And so I think this is, this is my big bet, and I think this is where B2B's gonna shine is when we start exposing agents through the MCP architecture.

90. 13:59

    Again, treating MCP as a plugin architecture. We've done that with one of ours, which is this thing. We keep renaming it, so bear with me. It's like called Seer now, but it's just like Sentry's got a lot of data on what's broken in your application.

91. 14:10

    We do this thing where we do this really high-quality root cause analysis that's done via an agent. Um, we expose that root cause analysis, mostly to our UI to be fair.

92. 14:19

    We also expose it to the MCP, but because it doesn't do streaming, we have to do like some polling check where it's like, "Okay, start the job and then let's check in on it a few times."

93. 14:27

    But then there, because of the way agents work, it just gives up at some point. So yeah, it, it's a little complicated, but again, beta testing, the promise is there.

94. 14:34

    Um, but when this works, I, I really think this is gonna be the value unlock for a lot of us. Again, MCP does a lot of things. It's an abstract protocol.

95. 14:41

    Um, but the agent analogy is really good. Um, aside, all this is open source. You can find Sentry's MCP somewhere on the internet. You'll find it on GitHub. I should say fair source.

96. 14:50

    There's some complexity there. This is what the agent looks like in the UI. Check it out if you haven't. We'll be around. Give me feedback. Um, I think the last thing I wanna, wanna sort of part with is just like this stuff is not that hard.

97. 15:02

    Um, it's quite broken all the time, but it's not that hard. I, again, I built it in two days. I got a lot of jobs to do at the company.

98. 15:08

    You can just go build it and try it out and learn and like all this stuff is pretty obvious. I think the lesson we've learned at Sentry, uh, or still are learning, I should say, uh, everybody is scared of all this stuff because there's fancy new words for everything, but the fancy new words are just new words

99. 15:22

    for the same thing. It's just a new p- like coat of paint, right? You know, MCP is just a plugin architecture. Agents are just services, like the, the LLM calls or MCP calls, actually half of them, tools are just API calls with a new response format, right?

100. 15:34

     So it's pretty accessible to do all this. There's a lot of great like technology that's been going on in here. Um, like I said, we used a lot of Cloudflare tech.

101. 15:40

     We did not use Cloudflare at all before this, and then in a couple days we're like, "Cool, we can shim up a thing on, on Workers. They've got an OAuth proxy for us.

102. 15:48

     Problem solved." And this is important 'cause we don't run WebSocket infrastructure at Sentry. It's just not a thing we had, right? And unfortunately, the protocol requires something like that, which makes it a little bit annoying to adopt, but, but again, it's not that hard.

103. 15:59

     It's pretty easy to adopt. Uh, try it out. You'll probably hit a lot of bugs, but just stick with it. I, I think this one will stick around. Um, but I would really dial in the, the thinking around agents and how you're optimizing for context in the workflows you understand for your data.

104. 16:12

     Uh, with that said, I will be around the rest of the afternoon probably at our, our booth in the expo hall if you wanna come chat, uh, come say hi.

105. 16:20

     I'm always happy to like rant about other things or give you my semi-informed opinions. Um, I'm not an AI guy to be clear, but, um, cool. With that, you know, thanks everybody for, for showing up to this talk and this wild conference which is interesting, and I'll call it there. [outro music]