AI Engineer Summit 2025
OpenLLMetry is all you need
Read the talk
OpenLLMetry: bringing LLM applications into OpenTelemetry
Logs, metrics and traces already describe cloud applications. OpenLLMetry extends that machinery to model calls, vector databases and AI frameworks.
From a talk by Nir Gazit
Before you start: Basic familiarity with application code and client libraries is helpful; no prior OpenTelemetry experience is required.
How do you observe what an application is doing?
How do you see what is happening inside a cloud application—and keep that visibility in the observability platform you already use? OpenLLMetry, introduced by Nir Gazit, CEO of Traceloop, starts with an existing foundation: OpenTelemetry. Before adding LLMs to the picture, it helps to understand what that foundation provides.
OpenTelemetry is a CNCF-maintained project that standardizes cloud observability. Gazit describes it as one of CNCF’s largest projects after Kubernetes; the relevant comparison is project activity, rather than adoption. CNCF’s mid-year 2024 analysis ranked OpenTelemetry second in project velocity for July 2023–July 2024. Its practical appeal is broad backend support: Splunk, Datadog, Dynatrace, New Relic, Grafana and Honeycomb are among the platforms Gazit names. A common telemetry format lets application instrumentation work with those existing destinations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Events, aggregates and multistep requests
Gazit introduces OpenTelemetry through its protocol, which standardizes logs, metrics and traces. More fully, OpenTelemetry is a framework and toolkit containing protocols, APIs, SDKs and other components. Start with the simplest signal: a log records an event at some point during an application’s lifetime, optionally with metadata. If you have written a Python print statement, you have already used the basic idea. For example, this records a request event and its identifier:
python
import json
request_id = "request-123"
print(json.dumps({
"event": "request_received",
"request_id": request_id,
}))
The event says something happened. By itself, it does not describe an aggregate trend or the request’s complete journey.
Metrics describe behavior in aggregate: across days, across users or across another grouping. Traditional cloud metrics include CPU usage, memory usage and latency. For a GenAI application, the corresponding questions include how many tokens it consumes, how long calls take and how often they fail.
| Application context | Example metrics |
|---|---|
| Cloud infrastructure | CPU usage, memory usage, latency |
| GenAI application | Token usage, latency, error rate |
Latency appears in both rows because adding a model does not remove the need to understand response time.
Tracing follows a multistep process. Gazit identifies tracing as OpenTelemetry’s first defined signal. In a microservice architecture, a trace connects the work performed as a user request passes through several services. The same structure fits GenAI applications: chains, workflows and agents can make several calls and execute tools before producing a result. A trace preserves the relationship between those steps, giving the individual operations a shared request context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a standard to working telemetry
A format alone cannot collect data from an application. OpenTelemetry supplies an ecosystem for producing and moving that data, beginning with SDKs for explicit instrumentation. At the time of the talk, Gazit reports SDK support for eleven languages, including Python, TypeScript, Go and C++.
The components divide the work into three roles:
| Component | Role |
|---|---|
| SDK | Explicitly emit logs, metrics and trace spans from application code |
| Instrumentation | Automatically capture telemetry from supported libraries |
| Collector | Process telemetry before exporting it to a backend |
With an SDK, you decide where to emit a signal and write the call yourself. Instrumentation supplies that behavior for supported dependencies. A Collector handles the subsequent processing stage before the data reaches an observability platform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Automatic instrumentation wraps the client library
Consider an application that accesses Postgres. Rather than manually adding telemetry around every database operation, you can use instrumentation for the database client. In the mechanism Gazit describes, the instrumentation monkey-patches the client library: it wraps the library’s behavior inside the application and emits telemetry when the application uses it. This places capture on the application side, where the outgoing operations are visible.
That is the source of the automatic visibility: the instrumented dependency produces telemetry as existing application code calls it. Gazit describes the latency impact as almost negligible, but supplies no workload, baseline or measurement conditions. The talk therefore establishes the capture mechanism, while leaving its overhead for a particular application unmeasured.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control what leaves the application environment
Collectors are self-deployable components that can run inside your own cloud environment, including on Kubernetes. They provide a processing point between telemetry generation and delivery to an external platform.
Their useful operations include:
- Filtering: remove telemetry that is not useful for your application’s observability needs.
- Sensitive-data handling: obscure personally identifiable information or other sensitive values before export.
- Multiple destinations: send telemetry to more than one provider.
Open-source Collector components provide these processing capabilities, but sensitive-data handling requires configuration; deploying a Collector does not automatically establish a privacy policy. This layer gives you a place to apply that policy before forwarding the data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extend the same machinery to GenAI dependencies
OpenLLMetry applies this instrumentation approach to foundation models, vector databases and GenAI frameworks. The resulting telemetry uses OpenTelemetry, so the destination can remain the platform the application team already uses. Gazit names Datadog, Sentry, Grafana Tempo and Dynatrace as trace destinations. Automatic collection still needs a correctly configured export destination.
At the time of the talk, Gazit reports more than forty providers supported through community-built instrumentations. The integrations span several layers of an AI application:
- Model services: OpenAI, Anthropic, Cohere, Gemini and Bedrock.
- Vector databases: Pinecone and Chroma.
- Application frameworks: LangChain, LlamaIndex, CrewAI and Haystack.
The aim is automatic visibility into these dependencies through the same logs, metrics and traces model. The precise signals and fields available depend on the integration; the list does not establish identical coverage for every library. Once the appropriate instrumentation and export configuration are in place, supported operations can emit telemetry without manual emission around every call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What Pinecone instrumentation exposes
The Pinecone integration makes the idea concrete. Gazit describes visibility into outgoing queries and indexing activity, followed by the ability to investigate returned vectors. That connects the operation sent by the application to the retrieval results it receives.
The demonstrated scope includes returned vector data, vector distances, scores and latencies in standard OpenTelemetry format. These provide different views of an operation: the query shows what was requested, the returned data and scores help explain the retrieval result, and latency describes how long the operation took. This is the useful extension beyond observing only whether a database call succeeded.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep instrumentation independent of the backend
A shared telemetry standard separates application instrumentation from the choice of observability platform. OpenLLMetry connects LLM applications to an existing backend through OpenTelemetry. Gazit characterizes switching platforms as a configuration change because compatible platforms accept the same telemetry format. That portability applies to telemetry export: storage, visualization and backend-specific functionality remain the responsibility of the receiving tools. The application can retain its instrumentation while changing where its telemetry goes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
OpenTelemetry extensions for observing LLM applications, with installation instructions and supported integrations.
An introduction to telemetry signals, instrumentation, SDKs, protocols and observability backends.
Guidance for receiving, processing and exporting telemetry to multiple backends.
Further reading
Nir Gazit's October 2023 introduction explains the project's origins and open-protocol approach to LLM observability.
Updates since the talk
Current settings for disabling content capture globally or controlling it for individual workflows.
Read the complete timestamped transcript
- 0:01
Hey everyone, I'm Nir, the CEO of Traceloop, and today I'm going to talk to you a bit about OpenLLMetry, which is a nice open source project we've built.
- 0:09
You can probably hear from the name OpenLLMetry that it originates in another project called OpenTelemetry. And in case you're not familiar with OpenTelemetry, I'm going to spend the next couple of minutes explaining to you what OpenTelemetry is.
- 0:22
So, you know, we're not talking about GenAI yet. We're not talking, talking about LLMs, just plain simple cloud observability. OpenTelemetry is an open source project. It's, uh, maintained by the CNCF.
- 0:34
It's one of the largest projects, uh, out there after Kubernetes, eh, that is maintained by the CNCF, and it standardizes a way to do cloud observability, um, in your, you know, cloud environment.
- 0:49
Currently, it's supported by every major observability platform from Splunk, Datadog, Dynatrace, New Relic, Grafana, Honeycomb, and many, many others. So if you're using OpenTelemetry, you can use it in conjunction with any of these platforms, uh, pretty easily.
- 1:04
But what is OpenTelemetry exactly? OpenTelemetry is a protocol, first and foremost, that, uh, standardizes the way to do logging metrics and traces, uh, in your, uh, cloud application. Logging, I think, uh, I don't need to explain to you what is, what is logging exactly, uh, because if you've ever written some Python, you know, script and you've, you've
- 1:26
written print, then, then you've done some logging. So logging is an arbitrary event that you can send, uh, anytime you want, uh, in the life cycle, lifetime of your application, and it just emitted as is, and you can view it, uh, later on, possibly with some metadata.
- 1:43
Metrics, on the other hand, is completely different. Metrics is something you wanna see on an aggregate level. You wanna see how it behaves across days or across users or, you know, whatever you want.
- 1:53
And when we're thinking about metrics in the traditional cloud world, we are probably talking about CPU usage, uh, memory usage, or, um, latency. And if we wanna, you know, talk a bit about GenAI, if you're thinking about which metrics, uh, we wanna see when we are, uh, building a GenAI-based applications, it's probably things like token usage, latency,
- 2:16
error rate, uh, and so on. Lastly, OpenTelemetry defines tracing. Actually, this was the first thing that was defined, uh, with, uh, OpenTelemetry, but, uh, it's the, uh, I would say the least trivial one.
- 2:31
So tracing is basically tracking of a multi-step process. So again, thinking about, you know, cloud environments, you have microservices, they're talking to each other, and you wanna see some process that spans across multiple microservices.
- 2:43
You wanna use a trace for that. And then you can see, you know, how a certain request coming from the user is processed across these, uh, microservices. Uh, specifically for GenAI, I think tracing is actually pretty common because we ha-- we are using, you know, a lot of multi-step processes, whether it's chains or workflows, uh, or even,
- 3:04
you know, agents or multi-step processes that ma-- that, uh, interact and run, uh, tools.
- 3:13
So yeah, logging metrics and traces, this is what OpenTelemetry as a protocol defined.
- 3:20
But it doesn't stop there because, you know, what can you do with a protocol? So OpenTelemetry is also an ecosystem, and it contains, uh, SDKs, instrumentations, and collectors. SDKs are the way you can, you know, manually send out these logs, metrics, and traces, uh, from your application.
- 3:38
It-- OpenTelemetry currently has eleven different languages, uh, supported in eleven different languages, uh, of SDKs from Python, TypeScript, Go, C++, and many others. You know, every language you're probably using has an OpenTelemetry SDK.
- 3:56
Instrumentations are a way to do this automatically. So remember, if you're using an SDK and you're, you wanna send out logs, metrics, and traces, you need to do it manually.
- 4:05
You need to actually send out a log or emit a trace or a span, uh, or a metric. But instrumentations can do it, uh, automatically, and we're gonna, we're gonna talk about it in a bit.
- 4:16
And lastly, collectors allow you to do some processing to your observability data before you send it out to whatever, uh, uh, observability platform you're using.
- 4:28
So what are instrumentations? Instrumentations are a way, as I said, to automatically get some observability data, some visibility into, uh, some parts of your application. Let's say you have-- you're using a, an SQL server, then you can use an instrumentation for that SQL server like Postgres and get some, uh, logs, metrics, and traces automatically.
- 4:53
Uh, the way these instrumentations work is that they monkey patch the client library that you're using within your application and then emit all the data that is, that, that you wanna, probably wanna see in your, uh, in, in your observability platform.
- 5:07
Everything happens on the application side, and they're designed, uh, magnificently on an engineering level so that, uh, the latency impact is almost negligible and, uh, you get a, you know, a nice view of everything that's happening in your system without doing anything.
- 5:26
Collectors are, uh, self-deployable components you can deploy in your own, uh, you know, cloud environment, on Kubernetes or whatever you want, and can provide you some pre-processing before you send data out to whatever, you know, platform you're using.
- 5:42
For example, if you want to filter out some data that is not important for you, or you want to obscure PII or obscure sensitive data and hide it, you can use a collector to do it.
- 5:51
And, and those, you know, ready-made components that you can just deploy, and they're completely open source, have a lot of these, uh, features just built in. And lastly, if you also wanna send this out, send the, you know, the observability data out to multiple providers, you can also do it with a collector.
- 6:08
I think at this point you're probably asking me, "Hey, Nir, you talked a lot about OpenTelemetry, but we are in a GenAI conference. What, what does that has to do with, with GenAI?"
- 6:17
This is where OpenLLMetry comes in. We took this amazing project called OpenTelemetry and extended it to support a lot of, you know, GenAI, uh, frameworks, uh, foundation models, and, uh, vector databases that some of, you know, you can see some of the logos, uh, that, of the instrumentations we built here.
- 6:37
So we extended it to support all of these, you know, amazing products. And because we are relying on OpenTelemetry, you can then get observability in whatever platform you're using.
- 6:47
You wanna, you wanna see, you know, traces within Datadog? Just use OpenLLMetry. You wanna see them in Sentry? You wanna see them in Grafana Tempo or in Dynatrace? Just, you know, just use OpenLLMetry, configure it correctly, and that's it.
- 6:59
You get logs, metrics, and traces automatically in your favorite platform. It's kinda nice.
- 7:07
Um, we've worked, we've worked a lot on, on, on building the instrumentations with our community, and now we have more than forty different, uh, providers. So we're talking about foundation models like OpenAI, Anthropic, Cohere, Gemini, Bedrock, and many others.
- 7:21
Uh, we're also talking about vector databases like Pinecone, Chroma, and many others. And we're also, we also have, uh, support for frameworks like LangChain, LlamaIndex, Qwery, and Haystack. So, uh, you have, you know, the instrumentations that automatically emit logs, metrics, and traces, and then just connect it to whatever platform you want, and you get it, uh, out
- 7:42
of the box. And remember, because these are instrumentations, it's, it's kinda done automatically, so, uh, it's, it's pretty cool. It's like a magic.
- 7:53
Just, just to give you an example, what, what is, what an instrumentation for, let's say, Pinecone would look like. So, uh, if the, the Pinecone instrumentation will contain, you know, ways to see the queries going out to Pinecone, uh, see indexing, uh, happening within Pinecone, and also inve-- and also ability to investigate vectors returned from Pinecone.
- 8:15
So you wanna see, you know, the data, the distances, uh, the vector distances that Pinecone returned, or, uh, scores, latencies, all of these are available in the standard OpenTelemetry format.
- 8:28
So OpenLLMetry, it's, it's a great, you know, way to, to connect, uh, LLM-based applications to whatever platform you're currently using. And because it's a standard protocol, you're, you're never tied to a specific platform.
- 8:44
You can easily switch between platforms. It's just a matter of, you know, a configuration change, uh, because all of the platforms that support OpenTelemetry, uh, support with the same, with the exact same format.
- 8:56
Uh, that's it. If you have any questions, uh, I'm available. I will be in the conference. I'm available in the conference or otherwise at, uh, [REDACTED:email_address]. Thank you very much.