The State of MCP Observability: Observable.tools — Alex Volkov and Benjamin Eckel, Weights & Biases and Dylibso

[upbeat music] Hey, folks.

Um, my name is Alex Volkov. I'm an AI evangelist with Weights & Biases.

I'm Benjamin Eckel. I am co-founder, CTO of Dylibso. We're creators of [REDACTED:url]run.

All right. And we're here to talk to you about MCP observability.

Hey, Ben, I want to ask you a question.

Yeah, sure.

As somebody who worked at Datadog before and somebody who runs multiple MCP servers and, uh, clients on production, uh, something that happened-- advice that happened, something in my agent, uh, in production the other day.

Okay. Uh, yeah, I mean, we've been running MCP clients and servers in production since the beginning. Uh, yeah, but wait, aren't you, like, working at an observability company, Weights & Biases?

And don't you work on, like, what's it called? Weave?

Yep, that's true. I, I work on Weave and-- but since I started adding some powers to my agent via MCP, all that observability that I'm used to from just having my own code run end-to-end has gone a little bit dark.

Gotcha.

So this is what we're here to talk to you guys about. Um, the rise of MCP is creating an observability blind spot. As AI agents become more, uh, prevalent, the problem can compound with more and more tools via MCPs, the less they, the developers can know about the end-to-end ha-happenings within their agent.

Yeah. Um, yeah, so on [REDACTED:url]run, we're running both clients and servers, and because it's a new ecosystem, we've had to, like, cobble together a lot of our own ways to do observability.

And I've been looking around, and it seems like everyone is sort of doing this in isolation. They're sort of solving the same problems. Um, so you know, we wanted to bring the community together on this issue.

And so today we're gonna talk about the state of observability in the MCP ecosystem.

Yep. So why do we care about this, and why do we think that you guys should care about this? So if you don't have the ability to quickly understand why things went wrong on production, where they went wrong, and how, your ability to quickly respond is greatly diminished.

And we care deeply about-- We both build tools that need MCP observability, and we, we support MCP, and we both care deeply about developer experience as well.

Yeah. It's, it's really important to me because enterprise engineering teams don't ship something to production unless they know for sure that they're gonna be able to identify security and reliability problems before their customers do.

Um, and that's why they invest a ton of money in observability platforms. And, uh, so if you're gonna ship MCP to these production environments, you must seamlessly integrate with these ober-observability platforms.

Yep. So because we care deeply about, uh, developer experience at, uh, W&B Weave, uh, I'm happy to announce here on stage that Weave supports [REDACTED:url] Yay. As long as you're a developer of both the client and the server, all you need to do is set this MCP_TRACE_LIST_OPERATIONS environment variable on your client and server, and, uh, we'll show

you the, the list tool calls, and we'll show you the, the, the duration of your m-MCP calls. This works currently with our Python-based clients, and this is how it looks, super quick.

With the red ar-ir-arrows, you can see the client traces, for example, and with the blue arrows, you can see we're pointing to the calculate BMI tool and, and the other tool, and that's it.

Observability solved, right?

Yeah.

Let's, let's get off the stage. We're done.

No, wait a second. [chuckles] So, uh, what about this, like, calculate BMI tool, this, uh, MCP server? Why can't I see into that?

Um, you-- Yeah, we're working on this.

Uh, yeah. Also, this seems like this is specific to Weave, right? Um, is there not, like, a vendor-neutral way to do this that's standardized?

Yeah, that's right. Uh, this is a, uh, bespoke integration that we built into Weave, into our SDKs in Python. And while working on this, while our developers have been building this, like, um, integration within our MCP tooling, I was advocating internally and externally that sh- we should align with the open nature of MCP as a concept and

created Observable.tools. Maybe some of you have seen this. This is a manifesto to drive a conversation that this is a problem that needs solving. And, uh, between observability providers, uh, such as us and other folks that's been on stage before and gonna be on the Evil Shark tomorrow, uh, to do observability in a vendor-neutral and standardized way.

And so while working on Observable tools, I realized, I s-- I did some search, realized that a vendor-neutral, scalable way to add observability exists, uh, and there could be a great way to marry the two open protocols to work together.

Yeah, exactly. Uh, fortunately, MCP-powered agents are really just another distributed system, and we've been doing that for decades. So OpenTelemetry is just the way that's, that we've, like, settled on doing that.

Um, we're gonna talk about OTel a little bit. Uh, if you're not, uh, familiar with it, we need to learn about a few primitives first.

So the main primitive that we need to learn about is the trace. So a trace is kind of like an atomic operation in your system. It's made up of a tree-like structure of steps that we call spans.

And a span represents the duration and some arbitrary metadata for each step. And what this step is exactly is completely up to you to define. It can be as high level as, like, an HTTP request.

It can be as low level as a tiny little function call. Um, here's an example of, like, a checkout experience, an API for a checkout. The size and position of each of these spans correspond to how long it took and where it sits in the call graph, respectively.

And just from this data, you can tell a lot about a system and how to observe it.

Um, the other primitive you need to be aware of is sinks. So a sink is kind of like a centralized database where all your telemetry goes. But often they come in the form of this, like, whole platform with, like, a UI and dashboards and alerting and monitoring and all those things.

So there's a lot of logos here, Ben.

Yeah.

Uh, basically, a sink is an open standard way for folks like collectors to, like, receive those spans. As long as the developer instrumented their application code in a certain standard spec way, everybody can just receive those in, in the same unified way, right?

Exactly, yeah. It's-- If you squint, it's just kind of like a bunch of databases that all support the same schema and wire protocol.

And in fact-

And you could switch them out

... and in fact, they don't have to change much of their code or even change the code at all.

No, shouldn't need to.

It could be just config, right?

Right.

Uh, by the way, LLM observability tools like W&B Weave and some friends, uh, Simon from Logpraw here, uh, before and some other friends, uh, all have switched to support OTel as well.

OpenTelemetry is becoming, like, this global standard.

Great. Uh, yeah, another great thing about having a centralized sink, uh, is the last concept, distributed tracing. So going back to our checkout endpoint, if the, uh, fraud service sends its span to the same sink, then we can stitch back to- together the traces and show the whole context.

So maybe you're kinda seeing where the MCP server stuff comes in here.

Yeah. So, hey, Ben, i- if it's possible via the o- integration to the open protocol, um, what if I want to use MCP servers that other people host, like GitHub, like Stripe, like other folks?

Yeah, it's a good question. So, um, with MCP-enabled agents, or really just any distributed system, there are kind of two scenarios. There's when the client and server are in different domains, and then there's when they're in the same domain.

And by domain here, I don't necessarily mean the literal definition. I just mean, like, the administration, administrative domain of control, right? Like, do, like, do you own this MCP server?

Do you own this MCP client, or is it a third-party thing? So your GitHub Stripe example is, like, a great example of, like, the different domain scenario. So, um, this is a trace of an agent that is executing the prompt, "Read and summarize the top article on Hacker News."

So it's gonna reach out to this, like, remote fetch server to read Hacker News. But it appears to us in the trace as a single service span because it's ou- it runs outside of our domain of control, so it appears to black box to us. [clears throat]

Um, but suppose we do own the server, like maybe it's running in a da- different data center than the client. Um, how do we get actually the whole context?

Uh, it's pretty simple. So with distributed tracing and context propagation, we can have the remote fetch server send its spans to the same sink as the client, and the sink will just stitch together the missing, uh, parts of the trace back for us.

So in this graphic, you can see that we can now break into that fetch server, and we can see what it's doing. It's making some HTTP requests that's taking roughly three hundred fifty milliseconds, and then it's doing a little, uh, crunching to, to create some markdown.

Okay, so that, that is great in theory, and we ran through this. We could have a whole hour talking about OTel, not that we got an hour. Uh, but how do we can actually marry those two protocols together, right?

Uh, is there a standard way? Did the MCP spec folk deploy a way for us for observability?

Um, not quite. It was, it was, uh, pretty tricky to get, to get working. Um, it does work today, but, uh, it required a little bit more work than it should have.

So in order to do this, we need to, as I said, propagate the trace context from the client to the server. So here's a TypeScript example, and when we call a tool in the client, um, we're gonna extract our current span, and we're going to, uh, pass it along to the server.

And we achieve this by basically just shuttling the data through the protocol's meta payload.

And, uh, now that we're inside the server, uh, this would be like in the fetch server, we can pull that trace context out, inherit it as our current span, and then when we send our spans off to the sink, uh, it, it's as if it came from that parent span, and they-- the sink can stitch it back

together.

Ben, this is awesome. So you basically used an undocumented kind of property of the p- uh, sending the payload, together with the payload between clients and servers, um, to pass along the data that OTel needs to connect those things together, right?

Yeah, sort of. I just kind of had to abuse the lower-level interface reserved for the protocol. But a higher level way should be provided through tooling, and that's something we should talk about a little bit later in the talk.

Yep. And so-

Oh, yeah. So by the way, this is, uh, this is not just, um, a screenshot. This is a, a working demo. So, um, it's a lot more code than what I showed in the slide.

So if you wanna actually go see how this works and adapt this for your needs, uh, go check out this GitHub link. And I think actually you did that to, to get it to work with Weave, right?

Yeah. So now that we know how to pass context after you, you, you showed me the way, uh, let's see how amazing the solution actually is in practice. While Weave MCP, the thing I showed you guys before, was a bespoke solution baked into our Python SDK for Weave.

The huge benefit of MCP, generally, not only observability related, is that servers and clients don't have to run on the same environment or share the same code or be from the same programming language.

So w- while we were working on the Python SDK, you built an agent in TypeScript.

Mm-hmm.

And so because W&Weave-- W&B Weave supports OTel, OpenTelemetry, and it's an open protocol, uh, your TypeScript agent, it took me a few minutes to-- by a f- without changing much code to just send those traces into Weave from a TypeScript agent and not necessarily from a Python agent.

So here, uh, here you could see in the green the, the client traces are in the green, and then the server traces actually show what happens within those calls, uh,

on kind of the, the server side as well.

Yeah, that's really cool. So how did, how did you actually get the traces into Weave?

So this is very, very simple, way simpler than before. Uh, we just define W&B Weave as the OTLP endpoint, a standard that you kind of, like, showed me around, uh, and then folks can send their traces into wandb.ai/otel.

And all you need to do in addition to this is authorize. So add authorization headers and specify which project you wanna go into.

Cool.

Yep. So while we talk to you about observability, while I was working on this, I had a magic moment happening with [REDACTED:url] I wanted to share this with everybody and-

Yeah, okay. I love a good-

-with you as well

... MCP story.

Yeah. So, um, I used Claude Opus four that just came out to Weave-ify your agent that you built-

Yeah

... and to add this, uh, uh, MCP observability. And W&B Weave is gonna get a little meta, stay with us, uh, also has an MCP server.

Okay, what, what does it do?

So we have an MCP server that lets your agents or, or chats, et cetera, talk to your traces and see the data and summarize the data for you. Okay, so we have this [REDACTED:url]

It's been configured in my Windsurf, uh, and, and Clau- uh, Code, uh, Opus four [laughs] uh, was able to use this MCP server to kind of work through it. So here you see an example.

Um, the agent basically started working on your code, and then decided, "Okay, I'm gonna run the code," and then said, "Okay, I'm gonna go and actually see if the traces showed up at, at W&B Weave."

Then it noticed that they showed up, but they showed up incorrectly. So some input or output, a specific parameter that it needed to do, it didn't know how to do.

It wasn't part of the documentation. And so, uh, the next moment just absolutely blew my mind. This Opus-4 discovered that our MCP server exposes a support bot. So essentially another agent, uh, decided to write a query for it, uh, received the, the right information after a while, and acted upon this information, learned how to fix the thing

that it needed to fix, fixed it, and then went back to Weave and, uh, uh, to notice whether or not the fix was correct. So my, um, [laughs] my coding agent talked to another agent via support via MCP that it discovered on its own.

I didn't even know that this ability exists to work on your coding agent in, in things. Things got a little bit meta, and my head was like absolute... I was sitting like this while all this happened.

Didn't touch the keyboard once.

That's awesome. Yeah. It's pretty meta. Uh, yeah, before we go, I also wanted to have, uh, take a moment to have an announcement. So, um, mcp run will also be exporting telemetry to O- OTel compatible sinks.

Um, so as I mentioned before, we run both servers and clients. Uh, so for servers, we have this concept called profiles, and these allow you to, like, slice and dice multiple MCP servers into one single virtual server.

And on, on, uh, we also have the, an MCP client called Task, and this is like a single prompt agent that can be triggered via URL or a schedule, and it also just sort of marries with the idea of profiles.

Um, but yeah, soon you'll be able to get OTel out of both of these and hopefully, you know, we'll, uh, connect up to Weights & Biases and have a little party.

Yeah. You can send those to Weave straight from [REDACTED:url]run.

Okay. So, uh, to recap, um, observability is here at, in MCP today, but it's not evenly distributed. Uh, OTel should get you most of the way there, but the community needs to come together, uh, create, creating tooling and conventions to make it smoother.

Um, you shouldn't need to be an expert in observability to, like, get this stuff working.

So how do you get involved? Well, AI engineers, just start thinking about observability via MCP tooling and whether or not you're getting, uh, observability to the end-to-end of, of your execution chain.

Um, for tool builders and, uh, platform providers, we should join and work on higher level SDKs. So, uh, Arize's open inference, for example, is a great start, but all of us should help with the instrumentation for our clients who use bespoke SDKs to work on conventions also together.

Ben, can you explain semantic conventions super quick?

Yeah, sure. So as we learned earlier, um, spans, they carry user-defined attributes, right? So if they're user-defined, how does the sink know that a span is actually, say, an HTTP request with a 200 status code?

Or how does it know that it's an MCP tool call that has an error? Um, that's where semantic co- conventions come in. Um, and you can be a part of defining what the conventions are for agents that all observability platforms agree on.

And if you're interested in this, I would suggest going to check out the, uh, GenAI semantic conventions effort by the OTel team. And, um, yeah, lastly, for platform builders such as mcp run, um, you know, go add OTel support, help review RFCs, and finally, yeah, just come, like, talk to us about ideas because we're just...

Everything's just kinda coming together. Everything's so new and fresh, and we don't really know exactly what to do.

There's an additional track here at, at, uh, AI Engineer that's called the hallway track, and I've learned more about-

Oh, okay. [laughs]

... the stuff that we're talking about, uh, out there-

Heard of that

... by actually talking to people who implement this than I learned while preparing, uh, before this talk. It's quite incredible. So, um, and Ben?

Yeah, sure. Um, yeah, so again, I'm Ben. Uh, my call to action here would just be go check out mcp run. You can get a free account, try it out.

Uh, yeah, that's it.

And I'm Alex. Uh, uh, check out W&B Weave MCP OP to learn how to trace MCP, uh, with OTel. Uh, I'm also... I did the Observable Tools Initiative. I would love for you to check out the manifesto to see if this resonates with you, to join forces to talk about observability.

And, uh, we, uh, yeah, pl- please visit us at the booth. We have some very interesting surprises for you. We have a robotic dog right here, uh, that's observable.

I also run the ThursdAI podcast. I wanna send Swix a huge, huge shout-out for, uh, having, uh, uh, giving me the support to show up here and give... If you guys are interested in AI news, we're gonna record a, an episode tomorrow.

That's it. Thank you so much. [outro music]