OpenLLMetry is all you need

Talk transcript

1. 0:01

   Hey everyone, I'm Nir, the CEO of Traceloop, and today I'm going to talk to you a bit about OpenLLMetry, which is a nice open source project we've built.

2. 0:09

   You can probably hear from the name OpenLLMetry that it originates in another project called OpenTelemetry. And in case you're not familiar with OpenTelemetry, I'm going to spend the next couple of minutes explaining to you what OpenTelemetry is.

3. 0:22

   So, you know, we're not talking about GenAI yet. We're not talking, talking about LLMs, just plain simple cloud observability. OpenTelemetry is an open source project. It's, uh, maintained by the CNCF.

4. 0:34

   It's one of the largest projects, uh, out there after Kubernetes, eh, that is maintained by the CNCF, and it standardizes a way to do cloud observability, um, in your, you know, cloud environment.

5. 0:49

   Currently, it's supported by every major observability platform from Splunk, Datadog, Dynatrace, New Relic, Grafana, Honeycomb, and many, many others. So if you're using OpenTelemetry, you can use it in conjunction with any of these platforms, uh, pretty easily.

6. 1:04

   But what is OpenTelemetry exactly? OpenTelemetry is a protocol, first and foremost, that, uh, standardizes the way to do logging metrics and traces, uh, in your, uh, cloud application. Logging, I think, uh, I don't need to explain to you what is, what is logging exactly, uh, because if you've ever written some Python, you know, script and you've, you've

7. 1:26

   written print, then, then you've done some logging. So logging is an arbitrary event that you can send, uh, anytime you want, uh, in the life cycle, lifetime of your application, and it just emitted as is, and you can view it, uh, later on, possibly with some metadata.

8. 1:43

   Metrics, on the other hand, is completely different. Metrics is something you wanna see on an aggregate level. You wanna see how it behaves across days or across users or, you know, whatever you want.

9. 1:53

   And when we're thinking about metrics in the traditional cloud world, we are probably talking about CPU usage, uh, memory usage, or, um, latency. And if we wanna, you know, talk a bit about GenAI, if you're thinking about which metrics, uh, we wanna see when we are, uh, building a GenAI-based applications, it's probably things like token usage, latency,

10. 2:16

    error rate, uh, and so on. Lastly, OpenTelemetry defines tracing. Actually, this was the first thing that was defined, uh, with, uh, OpenTelemetry, but, uh, it's the, uh, I would say the least trivial one.

11. 2:31

    So tracing is basically tracking of a multi-step process. So again, thinking about, you know, cloud environments, you have microservices, they're talking to each other, and you wanna see some process that spans across multiple microservices.

12. 2:43

    You wanna use a trace for that. And then you can see, you know, how a certain request coming from the user is processed across these, uh, microservices. Uh, specifically for GenAI, I think tracing is actually pretty common because we ha-- we are using, you know, a lot of multi-step processes, whether it's chains or workflows, uh, or even,

13. 3:04

    you know, agents or multi-step processes that ma-- that, uh, interact and run, uh, tools.

14. 3:13

    So yeah, logging metrics and traces, this is what OpenTelemetry as a protocol defined.

15. 3:20

    But it doesn't stop there because, you know, what can you do with a protocol? So OpenTelemetry is also an ecosystem, and it contains, uh, SDKs, instrumentations, and collectors. SDKs are the way you can, you know, manually send out these logs, metrics, and traces, uh, from your application.

16. 3:38

    It-- OpenTelemetry currently has eleven different languages, uh, supported in eleven different languages, uh, of SDKs from Python, TypeScript, Go, C++, and many others. You know, every language you're probably using has an OpenTelemetry SDK.

17. 3:56

    Instrumentations are a way to do this automatically. So remember, if you're using an SDK and you're, you wanna send out logs, metrics, and traces, you need to do it manually.

18. 4:05

    You need to actually send out a log or emit a trace or a span, uh, or a metric. But instrumentations can do it, uh, automatically, and we're gonna, we're gonna talk about it in a bit.

19. 4:16

    And lastly, collectors allow you to do some processing to your observability data before you send it out to whatever, uh, uh, observability platform you're using.

20. 4:28

    So what are instrumentations? Instrumentations are a way, as I said, to automatically get some observability data, some visibility into, uh, some parts of your application. Let's say you have-- you're using a, an SQL server, then you can use an instrumentation for that SQL server like Postgres and get some, uh, logs, metrics, and traces automatically.

21. 4:53

    Uh, the way these instrumentations work is that they monkey patch the client library that you're using within your application and then emit all the data that is, that, that you wanna, probably wanna see in your, uh, in, in your observability platform.

22. 5:07

    Everything happens on the application side, and they're designed, uh, magnificently on an engineering level so that, uh, the latency impact is almost negligible and, uh, you get a, you know, a nice view of everything that's happening in your system without doing anything.

23. 5:26

    Collectors are, uh, self-deployable components you can deploy in your own, uh, you know, cloud environment, on Kubernetes or whatever you want, and can provide you some pre-processing before you send data out to whatever, you know, platform you're using.

24. 5:42

    For example, if you want to filter out some data that is not important for you, or you want to obscure PII or obscure sensitive data and hide it, you can use a collector to do it.

25. 5:51

    And, and those, you know, ready-made components that you can just deploy, and they're completely open source, have a lot of these, uh, features just built in. And lastly, if you also wanna send this out, send the, you know, the observability data out to multiple providers, you can also do it with a collector.

26. 6:08

    I think at this point you're probably asking me, "Hey, Nir, you talked a lot about OpenTelemetry, but we are in a GenAI conference. What, what does that has to do with, with GenAI?"

27. 6:17

    This is where OpenLLMetry comes in. We took this amazing project called OpenTelemetry and extended it to support a lot of, you know, GenAI, uh, frameworks, uh, foundation models, and, uh, vector databases that some of, you know, you can see some of the logos, uh, that, of the instrumentations we built here.

28. 6:37

    So we extended it to support all of these, you know, amazing products. And because we are relying on OpenTelemetry, you can then get observability in whatever platform you're using.

29. 6:47

    You wanna, you wanna see, you know, traces within Datadog? Just use OpenLLMetry. You wanna see them in Sentry? You wanna see them in Grafana Tempo or in Dynatrace? Just, you know, just use OpenLLMetry, configure it correctly, and that's it.

30. 6:59

    You get logs, metrics, and traces automatically in your favorite platform. It's kinda nice.

31. 7:07

    Um, we've worked, we've worked a lot on, on, on building the instrumentations with our community, and now we have more than forty different, uh, providers. So we're talking about foundation models like OpenAI, Anthropic, Cohere, Gemini, Bedrock, and many others.

32. 7:21

    Uh, we're also talking about vector databases like Pinecone, Chroma, and many others. And we're also, we also have, uh, support for frameworks like LangChain, LlamaIndex, Qwery, and Haystack. So, uh, you have, you know, the instrumentations that automatically emit logs, metrics, and traces, and then just connect it to whatever platform you want, and you get it, uh, out

33. 7:42

    of the box. And remember, because these are instrumentations, it's, it's kinda done automatically, so, uh, it's, it's pretty cool. It's like a magic.

34. 7:53

    Just, just to give you an example, what, what is, what an instrumentation for, let's say, Pinecone would look like. So, uh, if the, the Pinecone instrumentation will contain, you know, ways to see the queries going out to Pinecone, uh, see indexing, uh, happening within Pinecone, and also inve-- and also ability to investigate vectors returned from Pinecone.

35. 8:15

    So you wanna see, you know, the data, the distances, uh, the vector distances that Pinecone returned, or, uh, scores, latencies, all of these are available in the standard OpenTelemetry format.

36. 8:28

    So OpenLLMetry, it's, it's a great, you know, way to, to connect, uh, LLM-based applications to whatever platform you're currently using. And because it's a standard protocol, you're, you're never tied to a specific platform.

37. 8:44

    You can easily switch between platforms. It's just a matter of, you know, a configuration change, uh, because all of the platforms that support OpenTelemetry, uh, support with the same, with the exact same format.

38. 8:56

    Uh, that's it. If you have any questions, uh, I'm available. I will be in the conference. I'm available in the conference or otherwise at, uh, [REDACTED:email_address]. Thank you very much.