AI Engineer World's Fair 2025
Maximize GPU Efficiency with Continuous Profiling for GPUs
Read the talk
Finding Where GPU Time Goes with Continuous Profiling
Matthias Loibl walks from sampled CPU stacks to GPU telemetry and GPU-time profiles, showing how to investigate idle accelerators and trace device work back to its callers.
From a talk by Matthias Loibl
Before you start: Familiarity with function call stacks, CPU–GPU workloads, and basic Linux deployment concepts will help you follow the profiling examples.
Where is the machine spending its time?
Where does a program spend its time—and which work could you remove or make cheaper? Profiling supplies the measurements needed to answer that question. Matthias Loibl tentatively traces its history to IBM work in the 1970s, but the practical questions remain familiar: how much memory does a program use, where does CPU or GPU time go, which instructions execute, and how frequently and for how long do functions run?
Those measurements connect performance engineering to infrastructure cost. Loibl offers a hypothetical: a 10% software performance improvement might let an operator turn off 10% of its servers. The point is the possibility of reducing the capacity needed for a workload, not a guaranteed conversion from faster software to a smaller bill.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trade event completeness for continuous observation
A profiler can record every event, or periodically observe what is happening. Recording everything provides detail, but collection cost and data volume make that approach difficult to sustain continuously.
| Approach | What it records | Tradeoff |
|---|---|---|
| Tracing | Every event | Detailed history; high cost and data volume |
| Sampling | Observations at selected intervals | Lower collection volume; individual events can be missed |
Loibl's sampling example uses a ten-second window with an adjustable rate, such as 20 or 100 samples per second. The rate controls how often the profiler looks, rather than how often the application performs work.
Loibl associates 100 samples per second with less than 1% CPU overhead and cites approximately 4 MB of memory overhead. He does not supply the workload, hardware, or measurement conditions for those figures. The broader tradeoff is clear: sampling will miss events, but collecting continuously gives recurring work more opportunities to appear. A stack that executes only once matters less to this view than work that repeatedly consumes resources.
Each observation captures a call stack: the chain of functions calling one another at that moment. For CPU profiling, that means recording the stack executing on the CPU; for allocation profiling, it can mean recording the stack responsible for allocating memory. Repeated observations build a picture of where the application spends its resources without requiring a complete event history.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Observe the production environment
The useful profile is the one from the environment running the workload. A developer's machine does not reproduce production, so low overhead matters partly because it makes production observation practical. Loibl's collector uses Linux eBPF, drawing on kernel facilities to gather profiles. He describes starting one collector to profile applications without changing or individually instrumenting them.
Loibl introduces himself as the Berlin-based director of Polar Signals Cloud and a maintainer of Prometheus, Prometheus Operator, and Parca. He presents Parca as the open-source counterpart to the profiling work, before turning from established CPU and memory collection to the GPU preview.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use GPU metrics to choose the investigation
After three or four years working on CPU and memory profiling, the team began a GPU profiling preview earlier in the year of the talk. The first view uses the NVIDIA Management Library, NVML, to collect GPU metrics. In the demonstration, the blue utilization line represents the overall node, while the orange line represents one GPU process, identified by its process ID. Memory utilization and clock speed appear alongside those signals. A utilization drop identifies a time interval worth investigating: what stopped the GPU from staying busy?
The next metrics help distinguish possible constraints:
- Power: Compare measured power usage with the dashed power-limit line.
- Temperature: Look for heat that could lead to throttling. Loibl warns about sustained temperatures around 80°C; that is an approximate warning, not a threshold shared by every GPU.
- PCIe throughput: Investigate whether transfers between CPU and GPU are limiting the workload. In the displayed convention, negative values represent receiving and positive values represent sending; Loibl points to an example of sending 10 MB/s.
These signals narrow the question from “Why is utilization low?” to a more specific investigation of power, cooling, or data movement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find what the CPU does while the GPU waits
GPU telemetry becomes more useful when it shares a time axis with stored CPU profiles. The collector already captures CPU stacks with eBPF, so an interval of interest in the GPU metrics can lead directly into host-side execution.
- Find a time interval worth investigating in the metrics.
- Inspect the corresponding CPU activity. Loibl selects activity on CPU zero near the end of the example.
- Drag over that time range to open a flame chart.
- Follow the call stack to see what the CPU was executing while the GPU was not fully utilized.
In the selected profile, Python eventually calls into CUDA functions. The chart supplies the calling context that the utilization curve alone cannot show.
A common explanation is that the CPU is busy loading data and fails to keep the GPU supplied with work. The host can therefore be active while the accelerator remains underutilized. This is why looking only at GPU utilization leaves an important part of the workload unexplained.
The same investigation is not limited to Python. Loibl names Rust calling CUDA, compiled-language stacks generally, and runtime support including Python, Ruby, and the JVM. The underlying profiling approach also applies beyond accelerator workloads—to web servers, databases, and vector databases whose performance depends on where execution time goes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Attribute GPU execution time to its callers
CPU profiles explain host activity, but they do not directly answer how much GPU time a calling function causes. Loibl next introduces GPU-time profiling, a new preview demonstrated that morning, distinct from the earlier GPU metrics preview. The small example shows branches for matmul_naive and generate_random_matrix: matrix multiplication and random-matrix generation become identifiable work in a profile.
Loibl describes obtaining a start timestamp when a CUDA call stack appears on the CPU and a termination timestamp for the GPU kernel, then using the interval to determine the kernel's GPU duration. Once matching device-execution timestamps are available, the duration calculation is straightforward:
python
def gpu_duration_ns(start_ns: int, end_ns: int) -> int:
if end_ns < start_ns:
raise ValueError("Kernel end precedes kernel start")
return end_ns - start_ns
The difficult part is obtaining and matching the right events. CUDA launches are asynchronous, and the talk does not specify the launch-to-completion correlation mechanism; timing only a host launch call would not establish device execution time.
The resulting view attributes GPU time to individual functions. Loibl then moves from the small CUDA example to a larger profile, following a Python main function down through the calling stack into libcuda. The host call path provides the context for the device work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read widths, leaves, and colors separately
The larger chart prompts a useful question: does it show CPU time, GPU time, or both? Its calling path comes from the CPU, while its width represents attributed GPU time. Loibl explains that each stack's leaf is the function consuming time on the GPU. The chart connects the host's chain of calls to device execution rather than simply combining two unrelated timing measurements.
A second question concerns the colors. They distinguish binaries, not timing categories. Blue identifies Python in this example; Loibl tentatively identifies one of the lower binaries as CUDA.
| Chart feature | Meaning |
|---|---|
| Width | Attributed GPU time |
| Path to the leaf | CPU calling stack |
| Leaf | Function consuming GPU time |
| Color | Binary associated with the frame |
Keeping those meanings separate lets the reader follow which application code led to device work without mistaking a color for a measure of expense.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploy the collector
The deployment described in the talk starts with a Linux host and eBPF. The collector can run as a binary managed by systemd, in Docker, or as a Kubernetes DaemonSet. For Kubernetes, Loibl describes obtaining the manifest YAML, supplying a token, and deploying it. This is the talk's collector setup: the later CUDA profiling release additionally documents a GPU-process injection library and agent configuration, so the preview's setup should not be treated as a complete current CUDA installation procedure.
Existing CPU and memory profiling customers were beginning to integrate GPU profiling into their platforms. Loibl names Turbopuffer as interested in improving its vector engine's performance, without reporting a quantified GPU improvement.
The presentation closes with a booth invitation and an event-specific offer: two free consultation hours for the first ten people to sign up, plus discounts for seed and Series A startups.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
NVIDIA's interface for monitoring GPU utilization, memory, temperature, power, clocks, and active processes.
Open-source continuous profiling with eBPF collection, profile storage, querying, and local build instructions.
Further reading
Loibl and Vincent's conference companion covers GPU metrics, per-process utilization, and a preview of GPU-time flame graphs.
Versioned guidance for inference performance measurement, including GPU clocks, cooling, and thermal throttling.
Updates since the talk
The later CUDA profiler release explains how CUPTI execution records connect GPU durations to CPU launch stacks through USDT probes and eBPF.
Read the complete timestamped transcript
- 0:00
[upbeat music] Great. Uh, thank you for, for coming.
- 0:17
Um, I'm gonna talk about maximizing GPU efficiency with continuous profiling for GPUs. Um, so what is profiling? Profiling is pretty much as old as programming. I think it was, like, firstly-- uh, first done in, like, the 1970s.
- 0:32
I think some IBM folks were, were trying to figure out what was happening on their computers back then. So it's been around basically forever in computer science. And, uh, what are we doing with profiling?
- 0:46
We are profiling, um, basically, uh, anything that, that we can to, [chuckles] to inform our, um, view of the world. We wanna see, like, the memory or CPU and GPU, um, time spent.
- 0:59
We want to see the usage of the individual instructions and the frequency and duration of these function calls. So, um, yeah, a lot of different approaches to profiling, but, um, yeah, it's generally speaking, super, super important to, to performance engineering.
- 1:16
So why would we do this? Uh, obviously to improve performance, and also we can save money. So if we, um, improve our, um, software by, like, ten percent, we might be able to just, like, turn off ten percent of our, like, servers and save a bunch of money, right?
- 1:32
Um, so that would be great. And, um, there are two different kinds of, of profiling typically, um, that, that we're seeing these days. So one is tracing profiling, um, and that is you record each and every e-event all the time constantly.
- 1:48
But, um, obviously, that's great for, like, getting, like, the best, uh, possible, uh, view onto the system, but it's, like, pretty high cost, um, and generates a lot of data, so it's, like, hard to, to do, uh, continuously.
- 2:05
And that is why we're, uh, doing sampled, uh, profiling. So what we do is basically we sample for a certain duration, like ten seconds, and we, uh, only sample a hundred times per second or, like, twenty times per second, et cetera.
- 2:19
Uh, you can tweak that, uh, how often do you wanna profile, um, and, and, and sample. So, like, a hundred times per second isn't that much for a CPU, and that's why you get, like, less than, like a percent overhead on, on the CPU and, like, only like four megabytes of overhead, uh, for the memory profiling.
- 2:38
Um, you will most definitely miss things, but if you do it always on, um, you will eventually see most of the relevant things, right? Like, one stack that executed once isn't, like, relevant to us anyway.
- 2:53
Like, we wanna see the, the big picture.
- 2:57
Um, so yeah, this is basically what we're, what, what we're doing. We, we see, like, the stacks on the left-hand side executing, and these are, like, the functions that are calling each other, and we are just, um, like twenty times per second or a hundred times per second, taking, taking note, um, on what exact, uh, stack we're
- 3:16
seeing on the CPU, um, or which stack is, like, allocating, uh, et cetera.
- 3:24
Yeah, and that allows us to, like, do it always on, do it in production. Um, your machine is not the production environment, so it is pretty important to be able to do this in production and actually see what's happening, uh, out there in the real world and do it with lo-- uh, low overhead.
- 3:42
And we are actually using, uh, Linux eBPF.
- 3:46
And because we're u-using something, um, that the kernel is doing, we, we don't even have to, uh, change any of your, uh, applications. That means, um, you start one thing and it will start, um, profiling all of your applications.
- 4:02
So you don't really have to instrument. Quickly about me. I'm Matthias Loibl, flew in from, uh, Berlin, Germany, and I'm the director of Polar Signals Cloud, and I'm also a maintainer of Prometheus, Prometheus Operator, Parca as the open source, uh, version of all of what I'm talking about today, and some other projects.
- 4:26
So, um, we are basically here for, like, GPUs, right? And we just earlier this year, uh, after, like, working on CPU and memory profiling for the last three or four years, um, started, um, a preview on GPU profiling.
- 4:43
So I'm gonna talk about this today and, uh, why, why we think it's pretty, pretty great. Um, as you can see in this, uh, screenshot, um, we're talking to NVIDIA NVML to get these metrics out of, uh, your GPU.
- 4:58
So we can see in the blue ch-- the blue line on top, we can see the overall utilization of the node, and then the orange, uh, line is one particular process on the GPU.
- 5:11
Um, so we can see-- Over here, we can see the process ID. So we see individual processes, but we also see the overall, uh, nodes utilization, um, further down the memory utilization and the clock speed, et cetera.
- 5:26
And that will kind of inform, um, where we wanna look at, um, the performance of our system, right? So sometimes we can see the utilization drop down, and that might be something that we wanna investigate to really make sure that we are using, uh, our GPUs to the fullest.
- 5:42
Uh, just couple of more metrics we are collecting. So there's, like, the power utilization, and the dashed line is the power limit, and then the temperature. Temperature sometimes is important because, like, eventually, if you're, like, always at, like, eighty degrees Celsius, you're, you're gonna get throttled, um, by the GPU, um, quite significantly.
- 6:03
And then obviously PCIe throughput Um, it's interesting, are you bound by the data you are transferring between CPU and GPU? Perfect. Yeah, so, uh, just to repeat, like, um, the negative one is, uh, receiving whereas the, um, positive ones are sending 10 megabytes per second, uh, through PCIe.
- 6:24
And then we can, uh, use all of those metrics to correlate from the CPU prof- uh, from the s- uh, GPU metrics to the CPU, um, profiles that we're storing.
- 6:34
So we're, like, collecting, like we have done the last f- three or four years, uh, using eBPF, those, uh, CPU stacks. Um, and we wanna, like, see what is happening on the CPU.
- 6:46
So in this case we might wanna look at a particular stack on, uh, CPU zero, um, right before the end because there was some activity, for example. So we can drag and drop and select a particular, uh, time range, and then we are presented vis- with a flame chart.
- 7:03
Um, and in the flame charts we can see what the CPU is doing while the GPU is not fully utilized. So in, in this case we're, we, we can see that, uh, Python is actually calling, um, eventually the, the CUDA, uh, functions further down.
- 7:21
Um, but oftentimes you, you will see that, like, the CPU is, um, pretty actively, uh, trying to load data and being busy that way and not, not, uh, keeping the GPU busy.
- 7:34
Um, if you are, um, using Python we can see it. If you're using Rust to integrate, uh, with CUDA, for example, um, that also works, but any compiled language is going to show up, uh, in those stack traces, and even some of the interpreted languages are going to show up, uh, like Ruby, Python, um, uh, JVM, et
- 7:53
cetera. So while there's a focus at this conference, um, that we're talking about GPUs here, it really works with, like, any language and, and any application. So web servers, databases, and vector databases for example, um, also are interested in improving their performance obviously.
- 8:13
Uh, something super exciting that we first, uh, uh, introduced this morning, so this is, like, um, super fresh, uh, and hot off the press, is GPU time profiling. So, um, as you heard, like, I, I was talking about, like, these GPU profiles, and we, we look how, how much, uh, time is spent on, uh, individual functions on
- 8:35
the CPU. But we are, like, more in- interested in, uh, GPU time spent, uh, by these functions. So here's, like, a small example of CUDA functions, and basically what we do is we tell the Linux kernel to, um...
- 8:52
Whenever there's a CUDA stack getting put on the, on the CPU, to tell us the start time of that, uh, function, and then eventually tell us the s- uh, time when that, uh, kernel terminates, and then we know the duration of how much time, uh, that particular kernel was spending on, on the GPU.
- 9:11
Um, and, and that's super interesting obviously because now we can actually see how much, uh, GPU time these individual, um, functions are taking on the GPU. And here's a bit m- more of a real [chuckles] world example.
- 9:26
So, um, at the top we can see, uh, uh, on the... Yeah, on the right-hand side we can see, like, the main function in Python, and then calling down into, uh, libcuda down here.
- 9:37
And the width of these, um, stacks that we're seeing is, like, the actual time that we had these functions, uh, take up in, in the GPU. So this is showing CPU and GPU?
- 9:50
Yeah, so this is, like, basically, um, the stack on the CPU- Like- ... down to here, and then the leaf of each stack is the function that was taking time on the GPU.
- 10:03
Okay. W- what do the colors mean? Uh, the colors are different, um,
- 10:09
binaries in this case that are running on your machine. So that's why, like, blue up here for example is, is Python, and then there's, like, some, some I think CUDA, uh, down here.
- 10:21
Yeah, great question. Uh, how do you get started? Because we, um, we run, uh, on Linux using eBPF. You have a binary that you can, that you can run, uh, using systemd or Docker works as well, but we also have a daemon set for Kubernetes.
- 10:39
Um, and you deploy that, you get the m- manifest YAML and give it a token. Um, and then some of our customers are already using it for CPU and memory profiling, and they're starting to also integrate, um, their platforms with our GPU profiling.
- 10:55
Uh, especially like Turbopuffer, um, are, are interested in, in improving their performance of their, uh, vector engine, right?
- 11:06
And that's really it. Um, please visit, visit our booth. Um, um, you get, um... The first 10 people get, like, to sign up for a consultation get two hours for free if you want to, and we can also do discounts for C and CSA startups.
- 11:22
And that's really it. Thank you so much. [outro music]