# Maximize GPU Efficiency with Continuous Profiling for GPUs

Matthias Loibl

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/wt8gzWR6auQ-maximize-gpu-efficiency-continuous-profiling-gpus) · [Video](https://www.youtube.com/watch?v=wt8gzWR6auQ)

## Finding Where GPU Time Goes with Continuous Profiling

Matthias Loibl walks from sampled CPU stacks to GPU telemetry and GPU-time profiles, showing how to investigate idle accelerators and trace device work back to its callers.

Familiarity with function call stacks, CPU–GPU workloads, and basic Linux deployment concepts will help you follow the profiling examples.

### Where is the machine spending its time?

[0:17](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=17s)

Where does a program spend its time—and which work could you remove or make cheaper? **Profiling** supplies the measurements needed to answer that question. Matthias Loibl tentatively traces its history to IBM work in the 1970s, but the practical questions remain familiar: how much memory does a program use, where does CPU or GPU time go, which instructions execute, and how frequently and for how long do functions run?

![Slide titled “What?” defines profiling and lists space, time complexity, instruction usage, and frequency and duration of function calls.](https://ai.engineer/api/talk-essay-media/28f8068967eac30334dc671a74f69fc32e48d6ce573396d5d68cea876a6b0bc2)

Profiling measures memory, CPU and GPU time, instruction usage, and function calls.

Those measurements connect performance engineering to infrastructure cost. Loibl offers a hypothetical: a 10% software performance improvement might let an operator turn off 10% of its servers. The point is the possibility of reducing the capacity needed for a workload, not a guaranteed conversion from faster software to a smaller bill.

### Trade event completeness for continuous observation

[1:32](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=92s)

A profiler can record every event, or periodically observe what is happening. Recording everything provides detail, but collection cost and data volume make that approach difficult to sustain continuously.

| Approach | What it records | Tradeoff |
|---|---|---|
| Tracing | Every event | Detailed history; high cost and data volume |
| Sampling | Observations at selected intervals | Lower collection volume; individual events can be missed |

Loibl's sampling example uses a ten-second window with an adjustable rate, such as 20 or 100 samples per second. The rate controls how often the profiler looks, rather than how often the application performs work.

Loibl associates 100 samples per second with less than 1% CPU overhead and cites approximately 4 MB of memory overhead. He does not supply the workload, hardware, or measurement conditions for those figures. The broader tradeoff is clear: sampling will miss events, but collecting continuously gives recurring work more opportunities to appear. A stack that executes only once matters less to this view than work that repeatedly consumes resources.

Each observation captures a **call stack**: the chain of functions calling one another at that moment. For CPU profiling, that means recording the stack executing on the CPU; for allocation profiling, it can mean recording the stack responsible for allocating memory. Repeated observations build a picture of where the application spends its resources without requiring a complete event history.

### Observe the production environment

[3:24](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=204s)

**The useful profile is the one from the environment running the workload.** A developer's machine does not reproduce production, so low overhead matters partly because it makes production observation practical. Loibl's collector uses Linux **eBPF**, drawing on kernel facilities to gather profiles. He describes starting one collector to profile applications without changing or individually instrumenting them.

Loibl introduces himself as the Berlin-based director of [Polar Signals Cloud](https://www.polarsignals.com/) and a maintainer of Prometheus, Prometheus Operator, and [Parca](https://github.com/parca-dev/parca). He presents Parca as the open-source counterpart to the profiling work, before turning from established CPU and memory collection to the GPU preview.

### Use GPU metrics to choose the investigation

[4:26](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=266s)

After three or four years working on CPU and memory profiling, the team began a GPU profiling preview earlier in the year of the talk. The first view uses the [NVIDIA Management Library, NVML](https://developer.nvidia.com/management-library-nvml), to collect GPU metrics. In the demonstration, the blue utilization line represents the overall node, while the orange line represents one GPU process, identified by its process ID. Memory utilization and clock speed appear alongside those signals. A utilization drop identifies a time interval worth investigating: what stopped the GPU from staying busy?

The next metrics help distinguish possible constraints:

- **Power:** Compare measured power usage with the dashed power-limit line.
- **Temperature:** Look for heat that could lead to throttling. Loibl warns about sustained temperatures around 80°C; that is an approximate warning, not a threshold shared by every GPU.
- **PCIe throughput:** Investigate whether transfers between CPU and GPU are limiting the workload. In the displayed convention, negative values represent receiving and positive values represent sending; Loibl points to an example of sending 10 MB/s.

These signals narrow the question from “Why is utilization low?” to a more specific investigation of power, cooling, or data movement.

### Find what the CPU does while the GPU waits

[6:24](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=384s)

GPU telemetry becomes more useful when it shares a time axis with stored CPU profiles. The collector already captures CPU stacks with eBPF, so an interval of interest in the GPU metrics can lead directly into host-side execution.

1. Find a time interval worth investigating in the metrics.
2. Inspect the corresponding CPU activity. Loibl selects activity on CPU zero near the end of the example.
3. Drag over that time range to open a flame chart.
4. Follow the call stack to see what the CPU was executing while the GPU was not fully utilized.

In the selected profile, Python eventually calls into CUDA functions. The chart supplies the calling context that the utilization curve alone cannot show.

![Slide titled “Visualize with flame charts” shows a vertical colored call stack with a function-details tooltip.](https://ai.engineer/api/talk-essay-media/fb532d13d9e98c7b968f5c680ffeee08e8003bd498813f8ee6cb0c15ca0dec87)

A CPU flame chart helps investigate what happens while the GPU is not fully utilized.

A common explanation is that the CPU is busy loading data and fails to keep the GPU supplied with work. The host can therefore be active while the accelerator remains underutilized. This is why looking only at GPU utilization leaves an important part of the workload unexplained.

The same investigation is not limited to Python. Loibl names Rust calling CUDA, compiled-language stacks generally, and runtime support including Python, Ruby, and the JVM. The underlying profiling approach also applies beyond accelerator workloads—to web servers, databases, and vector databases whose performance depends on where execution time goes.

### Attribute GPU execution time to its callers

[8:13](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=493s)

CPU profiles explain host activity, but they do not directly answer how much GPU time a calling function causes. Loibl next introduces **GPU-time profiling**, a new preview demonstrated that morning, distinct from the earlier GPU metrics preview. The small example shows branches for `matmul_naive` and `generate_random_matrix`: matrix multiplication and random-matrix generation become identifiable work in a profile.

Loibl describes obtaining a start timestamp when a CUDA call stack appears on the CPU and a termination timestamp for the GPU kernel, then using the interval to determine the kernel's GPU duration. Once matching device-execution timestamps are available, the duration calculation is straightforward:

```python
def gpu_duration_ns(start_ns: int, end_ns: int) -> int:
    if end_ns < start_ns:
        raise ValueError("Kernel end precedes kernel start")
    return end_ns - start_ns
```

The difficult part is obtaining and matching the right events. CUDA launches are asynchronous, and the talk does not specify the launch-to-completion correlation mechanism; timing only a host launch call would not establish device execution time.

![Slide titled “GPU-time profiling” shows horizontal colored stack bars branching into matmul_naive and generate_random_matrix.](https://ai.engineer/api/talk-essay-media/effbac0ee1a9434d8f870bd8745600c281281e7650ea72d29fb985898a776f43)

A small GPU-time profile shows branches for matrix multiplication and random-matrix generation.

The resulting view attributes GPU time to individual functions. Loibl then moves from the small CUDA example to a larger profile, following a Python main function down through the calling stack into `libcuda`. The host call path provides the context for the device work.

### Read widths, leaves, and colors separately

[9:37](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=577s)

The larger chart prompts a useful question: does it show CPU time, GPU time, or both? **Its calling path comes from the CPU, while its width represents attributed GPU time.** Loibl explains that each stack's leaf is the function consuming time on the GPU. The chart connects the host's chain of calls to device execution rather than simply combining two unrelated timing measurements.

A second question concerns the colors. They distinguish binaries, not timing categories. Blue identifies Python in this example; Loibl tentatively identifies one of the lower binaries as CUDA.

| Chart feature | Meaning |
|---|---|
| Width | Attributed GPU time |
| Path to the leaf | CPU calling stack |
| Leaf | Function consuming GPU time |
| Color | Binary associated with the frame |

Keeping those meanings separate lets the reader follow which application code led to device work without mistaking a color for a measure of expense.

![GPU-time profiling slide with a branching chart of purple, blue, green, and cyan horizontal bars.](https://ai.engineer/api/talk-essay-media/b930191a636ce586f7a7e0cd25432bbd8a3df32c5ebf3f69ce7cc50c94763b99)

The larger GPU-time profile contains colored branches of varying widths and depths.

### Deploy the collector

[10:21](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=621s)

The deployment described in the talk starts with a Linux host and eBPF. The collector can run as a binary managed by `systemd`, in Docker, or as a Kubernetes `DaemonSet`. For Kubernetes, Loibl describes obtaining the manifest YAML, supplying a token, and deploying it. This is the talk's collector setup: the [later CUDA profiling release](https://www.polarsignals.com/blog/posts/2025/10/22/gpu-profiling) additionally documents a GPU-process injection library and agent configuration, so the preview's setup should not be treated as a complete current CUDA installation procedure.

Existing CPU and memory profiling customers were beginning to integrate GPU profiling into their platforms. Loibl names Turbopuffer as interested in improving its vector engine's performance, without reporting a quantified GPU improvement.

The presentation closes with a booth invitation and an event-specific offer: two free consultation hours for the first ten people to sign up, plus discounts for seed and Series A startups.

## Resources

- [Latest in Continuous Profiling for GPUs](https://www.polarsignals.com/blog/posts/2025/06/04/latest-in-continuous-profiling-for-gpus) — Loibl and Vincent's conference companion covers GPU metrics, per-process utilization, and a preview of GPU-time flame graphs.
- [NVIDIA Management Library](https://developer.nvidia.com/management-library-nvml) — NVIDIA's interface for monitoring GPU utilization, memory, temperature, power, clocks, and active processes.
- [Parca](https://github.com/parca-dev/parca) — Open-source continuous profiling with eBPF collection, profile storage, querying, and local build instructions.
- [Continuous NVIDIA CUDA Profiling In Production](https://www.polarsignals.com/blog/posts/2025/10/22/gpu-profiling) — The later CUDA profiler release explains how CUPTI execution records connect GPU durations to CPU launch stacks through USDT probes and eBPF.
- [NVIDIA TensorRT Performance Best Practices](https://developer.nvidia.com/docs/drive/drive-os/7.0.3/public/drive-os-tensorrt-developer-guide/best-practices.html) — Versioned guidance for inference performance measurement, including GPU clocks, cooling, and thermal throttling.

## Slides

### Profiling is as old as programming

The speaker describes profiling as nearly as old as programming and tentatively dates early attempts to the 1970s.

[0:23](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=23s) [Slide says profiling is as old as programming.](https://assets.ai.engineer/sites/aiecode2025/sha256/bd1d25df7dc92d1173085ed873ae4cfddb48e2424fbed94e20b4bdd2b2fd1bf1.webp)

Profiling

As old as programming

Polar Signals

AIE
aws

### What is profiling?

Profiling uses dynamic program analysis to measure resource consumption, including memory, CPU and GPU time, instruction usage, and the frequency and duration of function calls.

[0:43](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=43s) [Profiling measures resource consumption, including memory, CPU and GPU time, instruction usage, and function call frequency and duration.](https://assets.ai.engineer/sites/aiecode2025/sha256/f03d1ec8022b3637a9cd94ffa1405ea5e0c3874b314f65996d20902ec0fd6377.webp)

What?

Profiling is a form of dynamic program analysis that measures resource consumption, for example:

• the space (memory)
• time complexity of a program (CPU and GPU),
• usage of instructions,
• frequency and duration of function calls

https://en.wikipedia.org/wiki/Profiling_(computer_programming)

AIE
Polar Signals

### Why?

Profiling can improve performance and save money. The speaker suggests that a 10% software improvement might let a team turn off 10% of its servers.

[1:14](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=74s) [A slide with the word “Why?” in purple.](https://assets.ai.engineer/sites/aiecode2025/sha256/e5caa8f24382120566bf3f52f0c5bc1b51fc7c8e0dc0ee1270c87bf560c747d4.webp)

Why?

Polar Signals

AIE
Microsoft
smol ai

### Tracing

Tracing records every event continuously, giving a detailed view of the system but generating lots of data at high cost. That makes it hard to use continuously.

[1:36](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=96s) [Tracing records every event constantly and has high costs.](https://assets.ai.engineer/sites/aiecode2025/sha256/dcbed45e74c345a2b47a4f7dcbcbd4a101a6169bc354f90f7cba4a36f4757bb1.webp)

Tracing

• Recording each and every event constantly
• High costs

Polar Signals
AIE
Microsoft
smol ai

### Sampling

Sampling periodically observes call stacks, trading missed events for low overhead. The slide cites <0.5% CPU and ~4MB memory overhead.

[2:02](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=122s) [Sampling periodically observes call stacks for a set duration, with examples of sampling frequency and low overhead.](https://assets.ai.engineer/sites/aiecode2025/sha256/56cdd9330938d716ddb002324ffec75f6dfe4c0fff57898f3d4791248e42170d.webp)

Sampling

• Sample for a certain duration
  ◦ Eg. 10 seconds
• Periodically observe function call stack
  ◦ Eg. 100x per second
• Low overhead*
  ◦ <0.5% CPU
  ◦ ~4MB memory

*[illegible]s://cloud.google.com/blog/products/management-to
[illegible]-tests-cloud-profiler-adds-negligible-overhead

Polar Signals

Interface text:
AIE
Microsoft
smol ai

### What is profiling?

The colored columns depict function call stacks. Sampling records which stack is executing on the CPU or allocating, such as 20 or 100 times per second.

[3:06](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=186s) [A diagram shows sampled function call stacks as colored blocks, with a legend for one stack.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd32154a7d51f38a8e9ede37029b5aef8e4418a70123b211b964eb7e57aef7ac.webp)

What is profiling?

a()
c()
g()
f()

Polar Signals
AIE
Microsoft
smol ai

### What is profiling?

Profiling samples call stacks 20 or 100 times per second, recording which functions are executing on the CPU or allocating.

[3:13](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=193s) [A colored call-stack sampling diagram highlights another stack with a different function legend.](https://assets.ai.engineer/sites/aiecode2025/sha256/0847ef9870eb427ab701971cb2c9a8d18955b76dc52ceb2a6d31b58ec5e96e86.webp)

What is profiling?

a()
c()
b()

Polar Signals

Interface text:
AIE
Microsoft
smol ai

### What is profiling?

Linux eBPF enables always-on profiling in production with low overhead. The speaker says one profiler can cover applications without requiring code changes.

[3:22](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=202s) [Profiling is always on, runs in production, has low overhead, and requires zero configuration.](https://assets.ai.engineer/sites/aiecode2025/sha256/28a96287b4f7fecf83393ff5d1f1f1cf03756ee427c212165a50cdfb876ac73c.webp)

What is profiling?

• Always on
• In production
• Low Overhead
• Zero configuration

Polar Signals

Interface text:
AIE

### About me

Matthias Loibl directs Polar Signals Cloud and maintains open-source projects including Prometheus, Prometheus Operator, Parca, and Pyrra.

[4:04](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=244s) [Speaker introduction listing a Polar Signals Cloud role and open-source projects.](https://assets.ai.engineer/sites/aiecode2025/sha256/b8bf871e2719d2a30ecd3c25a34dc273e93d0ee2c7a0e8321d5ad0f1e2c8a47b.webp)

About me

• Director of Polar Signals Cloud
• Open Source Maintainer
  ◦ Prometheus
  ◦ Prometheus Operator
  ◦ Parca
  ◦ Pyrra

@metalmatze

Polar Signals

Interface text:
AIE

### Get the right metrics to maximize GPU utilization

Power stays below its dashed limit while temperature rises and then falls. PCIe throughput shows traffic in both directions: negative values represent receiving and positive values sending.

[5:41](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=341s) [GPU dashboard charts show power, temperature, and PCIe throughput over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/eba2b37f37a9b71cd9e361807204bbab03caa9830e79b06451befae9b75b2c65.webp)

Get the right metrics to maximize GPU utilization

See at a glance when a GPU is not utilized the way you expect using high resolution GPU hardware metrics.

Partial chart:
Y-axis tick: 500 MHz
Timeline: 17:20, 17:21, 17:22, 17:23, 17:24, 17:25
X-axis: Time

Power
Y-axis ticks: 20 W, 40 W, 60 W
Timeline: 17:20, 17:21, 17:22, 17:23, 17:24, 17:25
X-axis: Time

Temperature
Y-axis ticks: 50 °C, 55 °C, 60 °C
Timeline: 17:20, 17:21, 17:22, 17:23, 17:24, 17:25
X-axis: Time

PCIe Throughput
Y-axis ticks: -30 MB/s, -20 MB/s, -10 MB/s, 0 Bytes/s, 10 MB/s
Timeline: 17:20, 17:21, 17:22, 17:23, 17:24, 17:25
X-axis: Time

Interface text:
AIE
aws

### Correlate CPU usage with CPU strips

Temperature and PCIe throughput charts sit above CPU activity strips. The speaker describes selecting a time range on CPU 0 to inspect its stack in a flame chart.

[6:22](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=382s) [Temperature and PCIe throughput charts appear above colored CPU activity strips.](https://assets.ai.engineer/sites/aiecode2025/sha256/85eafa50b3b2fe3617ffb416bcd942a7fcbc7a4172b2ac0b2fdf3fdfd86948b8.webp)

Correlate CPU Usage with CPU strips

Understand down to the nanoseconds when the CPU was active and when not.

[Partially visible chart]
Y-axis tick: 20 W
Timeline: 17:20, 17:21, 17:22, 17:23, 17:24, 17:25
Time

Temperature
Y-axis ticks: 50 °C, 55 °C, 60 °C
Timeline: 17:20, 17:21, 17:22, 17:23, 17:24, 17:25
Time

PCIe Throughput
Y-axis ticks: 10 MB/s, 0 Bytes/s, -10 MB/s, -20 MB/s, -30 MB/s
Timeline: 17:20, 17:21, 17:22, 17:23, 17:24, 17:25
Time

CPU activity strips
Timeline: 1.25m, 2.5m, 3.75m, 5[illegible]
{cpu: 0}
{cpu: 1}
{cpu: 2}
{cpu: 3}

Interface text:
© 2025 Polar Signals. All rights reserved.
Support  Status  Pricing  Contact Sales
AIE
Microsoft
smol ai

### Visualize with flame charts

After selecting a CPU time range, the flame chart shows a Python call stack while the GPU is not fully utilized. The speaker traces the calls toward CUDA functions.

[6:56](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=416s) [A flame chart displays a Python call stack and a function detail popup.](https://assets.ai.engineer/sites/aiecode2025/sha256/8881f334ae58739eb491756e6bcc42942e22a723176f5db0165711abfb0e9e2d.webp)

Visualize with flame charts

If the GPU wasn't fully utilized, what was the CPU doing that wasn't making the GPU do work?

Timeline: 68.64ms, 137.27ms, 205.91ms, 274.55ms, 343.18ms, 411.82ms, 480.45ms, 549.09ms, 617.73ms, 686.36ms

Selected function: _PyObject_VectorcallTstate
Cumulative: 52.63 ms (100.00%)
Flat: 0 ns (0.00%)
Timestamp: 2025:06:03T17:21:15.428 (UTC)
File: ...de/internal/pycore_call.h +92
Address: 0x54cd31
Inlined: yes
Binary: python3.12
Build Id: 37451b37c71cb46f8ccb27cb3cdb...

Call stack labels: __GI__[illegible], start_thre[illegible], pythread_[illegible], thread_run[illegible], method_vec[illegible], _PyObject_[illegible], _PyEval_Ev[illegible], <interpret[illegible], Thread._b[illegible], Thread.run[illegible], _pin_memor[illegible], tupledeall[illegible], Py_XDECREF, Py_DECREF, list_deall[illegible], THPVariabl[illegible]

Showing 52,631,578 values.

Interface text:
Group by
Function Name
More
Contains
Filter by function
Share
Right click to show context menu
© 2025 Polar Signals. All rights reserved.
Support
Status
Pricing
Contact Sales
AIE
aws

### Languages we can see in stacks

Stack traces can include compiled languages such as Rust and interpreted languages such as Python and Ruby. The slide pairs language logos with CUDA and PyTorch.

[7:33](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=453s) [Logos for programming languages and GPU tools shown as examples of stacks.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b7d0b8d3f894d97902019b1ad0e3ba548d21f27575300fda5780cd34f1fb698.webp)

Languages we can see in stacks

NVIDIA
CUDA
PyTorch

GO
C++
C
R

Polar Signals

Interface text:
AIE
aws

### GPU-time profiling: CUDA example

The CUDA example branches into matrix multiplication and random matrix generation. The profiler uses function start and kernel termination times to calculate GPU duration.

[8:22](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=502s) [A stack chart splits into matrix multiplication and random matrix generation functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/2da675d07ac94dfe3cc658f0edb697630708242dcaed65c9c2354fb7bc2f836c.webp)

GPU-time profiling

root
_start
__libc_start_main_impl
__libc_start_call_main
main
x
matmul_naive
__device_stub__Z12matmul_naivePKfS0_Pf
generate_random_matrix
__device_stub__Z22generate_random_matrixPfm

Showing 890,432,142,752 values.

Polar Signals

Interface text:
AIE
Microsoft
smol ai

### How to get started

Polar Signals runs on Linux using eBPF. Deploy its Kubernetes DaemonSet with a tokenized manifest, or run the binary using systemd or Docker.

[10:20](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=620s) [Getting-started slide describes a Kubernetes installation command and mentions Docker, systemd, and a binary.](https://assets.ai.engineer/sites/aiecode2025/sha256/73763e26f19bfc3741f5d42b782b2cdbc0e1fde572d4ed95312809cae8eca4ad.webp)

How to get started

With just one command and the easiest onboarding guide you’ll ever see, you can start saving costs and optimizing performance in your infrastructure.

All it takes is a single command.

$ kubectl apply -f https://api.polarsignals.com/api/manifests.yaml?token=<your-token-here>

Also Docker, sytemd, and binary

Polar Signals

Interface text:
AIE
Microsoft
smol ai

[10:28](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=628s) [Polar Signals onboarding slide showing a Kubernetes installation command and listing Docker, systemd, and a binary as alternatives.](https://assets.ai.engineer/sites/aiecode2025/sha256/333a6162fede442daa4f01f7c1b0adaf3b9a6dde4d8fb20c2d33b204cdad9d5a.webp)

How to get started

With just one command and the easiest onboarding guide you’ll ever see, you can start saving costs and optimizing performance in your infrastructure.

All it takes is a single command.

$ kubectl apply -f https://api.polarsignals.com/api/manifests.yaml?token=<your-token-here>

Also Docker, sytemd, and binary

Polar Signals

Interface text:
AIE
aws

## Full transcript

[0:00](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=0s) [upbeat music] Great. Uh, thank you for, for coming.

[0:17](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=17s) Um, I'm gonna talk about maximizing GPU efficiency with continuous profiling for GPUs. Um, so what is profiling? Profiling is pretty much as old as programming. I think it was, like, firstly-- uh, first done in, like, the 1970s.

[0:32](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=32s) I think some IBM folks were, were trying to figure out what was happening on their computers back then. So it's been around basically forever in computer science. And, uh, what are we doing with profiling?

[0:46](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=46s) We are profiling, um, basically, uh, anything that, that we can to, [chuckles] to inform our, um, view of the world. We wanna see, like, the memory or CPU and GPU, um, time spent.

[0:59](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=59s) We want to see the usage of the individual instructions and the frequency and duration of these function calls. So, um, yeah, a lot of different approaches to profiling, but, um, yeah, it's generally speaking, super, super important to, to performance engineering.

[1:16](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=76s) So why would we do this? Uh, obviously to improve performance, and also we can save money. So if we, um, improve our, um, software by, like, ten percent, we might be able to just, like, turn off ten percent of our, like, servers and save a bunch of money, right?

[1:32](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=92s) Um, so that would be great. And, um, there are two different kinds of, of profiling typically, um, that, that we're seeing these days. So one is tracing profiling, um, and that is you record each and every e-event all the time constantly.

[1:48](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=108s) But, um, obviously, that's great for, like, getting, like, the best, uh, possible, uh, view onto the system, but it's, like, pretty high cost, um, and generates a lot of data, so it's, like, hard to, to do, uh, continuously.

[2:05](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=125s) And that is why we're, uh, doing sampled, uh, profiling. So what we do is basically we sample for a certain duration, like ten seconds, and we, uh, only sample a hundred times per second or, like, twenty times per second, et cetera.

[2:19](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=139s) Uh, you can tweak that, uh, how often do you wanna profile, um, and, and, and sample. So, like, a hundred times per second isn't that much for a CPU, and that's why you get, like, less than, like a percent overhead on, on the CPU and, like, only like four megabytes of overhead, uh, for the memory profiling.

[2:38](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=158s) Um, you will most definitely miss things, but if you do it always on, um, you will eventually see most of the relevant things, right? Like, one stack that executed once isn't, like, relevant to us anyway.

[2:53](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=173s) Like, we wanna see the, the big picture.

[2:57](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=177s) Um, so yeah, this is basically what we're, what, what we're doing. We, we see, like, the stacks on the left-hand side executing, and these are, like, the functions that are calling each other, and we are just, um, like twenty times per second or a hundred times per second, taking, taking note, um, on what exact, uh, stack we're

[3:16](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=196s) seeing on the CPU, um, or which stack is, like, allocating, uh, et cetera.

[3:24](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=204s) Yeah, and that allows us to, like, do it always on, do it in production. Um, your machine is not the production environment, so it is pretty important to be able to do this in production and actually see what's happening, uh, out there in the real world and do it with lo-- uh, low overhead.

[3:42](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=222s) And we are actually using, uh, Linux eBPF.

[3:46](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=226s) And because we're u-using something, um, that the kernel is doing, we, we don't even have to, uh, change any of your, uh, applications. That means, um, you start one thing and it will start, um, profiling all of your applications.

[4:02](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=242s) So you don't really have to instrument. Quickly about me. I'm Matthias Loibl, flew in from, uh, Berlin, Germany, and I'm the director of Polar Signals Cloud, and I'm also a maintainer of Prometheus, Prometheus Operator, Parca as the open source, uh, version of all of what I'm talking about today, and some other projects.

[4:26](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=266s) So, um, we are basically here for, like, GPUs, right? And we just earlier this year, uh, after, like, working on CPU and memory profiling for the last three or four years, um, started, um, a preview on GPU profiling.

[4:43](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=283s) So I'm gonna talk about this today and, uh, why, why we think it's pretty, pretty great. Um, as you can see in this, uh, screenshot, um, we're talking to NVIDIA NVML to get these metrics out of, uh, your GPU.

[4:58](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=298s) So we can see in the blue ch-- the blue line on top, we can see the overall utilization of the node, and then the orange, uh, line is one particular process on the GPU.

[5:11](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=311s) Um, so we can see-- Over here, we can see the process ID. So we see individual processes, but we also see the overall, uh, nodes utilization, um, further down the memory utilization and the clock speed, et cetera.

[5:26](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=326s) And that will kind of inform, um, where we wanna look at, um, the performance of our system, right? So sometimes we can see the utilization drop down, and that might be something that we wanna investigate to really make sure that we are using, uh, our GPUs to the fullest.

[5:42](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=342s) Uh, just couple of more metrics we are collecting. So there's, like, the power utilization, and the dashed line is the power limit, and then the temperature. Temperature sometimes is important because, like, eventually, if you're, like, always at, like, eighty degrees Celsius, you're, you're gonna get throttled, um, by the GPU, um, quite significantly.

[6:03](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=363s) And then obviously PCIe throughput Um, it's interesting, are you bound by the data you are transferring between CPU and GPU? Perfect. Yeah, so, uh, just to repeat, like, um, the negative one is, uh, receiving whereas the, um, positive ones are sending 10 megabytes per second, uh, through PCIe.

[6:24](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=384s) And then we can, uh, use all of those metrics to correlate from the CPU prof- uh, from the s- uh, GPU metrics to the CPU, um, profiles that we're storing.

[6:34](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=394s) So we're, like, collecting, like we have done the last f- three or four years, uh, using eBPF, those, uh, CPU stacks. Um, and we wanna, like, see what is happening on the CPU.

[6:46](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=406s) So in this case we might wanna look at a particular stack on, uh, CPU zero, um, right before the end because there was some activity, for example. So we can drag and drop and select a particular, uh, time range, and then we are presented vis- with a flame chart.

[7:03](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=423s) Um, and in the flame charts we can see what the CPU is doing while the GPU is not fully utilized. So in, in this case we're, we, we can see that, uh, Python is actually calling, um, eventually the, the CUDA, uh, functions further down.

[7:21](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=441s) Um, but oftentimes you, you will see that, like, the CPU is, um, pretty actively, uh, trying to load data and being busy that way and not, not, uh, keeping the GPU busy.

[7:34](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=454s) Um, if you are, um, using Python we can see it. If you're using Rust to integrate, uh, with CUDA, for example, um, that also works, but any compiled language is going to show up, uh, in those stack traces, and even some of the interpreted languages are going to show up, uh, like Ruby, Python, um, uh, JVM, et

[7:53](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=473s) cetera. So while there's a focus at this conference, um, that we're talking about GPUs here, it really works with, like, any language and, and any application. So web servers, databases, and vector databases for example, um, also are interested in improving their performance obviously.

[8:13](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=493s) Uh, something super exciting that we first, uh, uh, introduced this morning, so this is, like, um, super fresh, uh, and hot off the press, is GPU time profiling. So, um, as you heard, like, I, I was talking about, like, these CPU profiles, and we, we look how, how much, uh, time is spent on, uh, individual functions on

[8:35](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=515s) the CPU. But we are, like, more in- interested in, uh, GPU time spent, uh, by these functions. So here's, like, a small example of CUDA functions, and basically what we do is we tell the Linux kernel to, um...

[8:52](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=532s) Whenever there's a CUDA stack getting put on the, on the CPU, to tell us the start time of that, uh, function, and then eventually tell us the s- uh, time when that, uh, kernel terminates, and then we know the duration of how much time, uh, that particular kernel was spending on, on the GPU.

[9:11](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=551s) Um, and, and that's super interesting obviously because now we can actually see how much, uh, GPU time these individual, um, functions are taking on the GPU. And here's a bit m- more of a real [chuckles] world example.

[9:26](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=566s) So, um, at the top we can see, uh, uh, on the... Yeah, on the right-hand side we can see, like, the main function in Python, and then calling down into, uh, libcuda down here.

[9:37](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=577s) And the width of these, um, stacks that we're seeing is, like, the actual time that we had these functions, uh, take up in, in the GPU. So this is showing CPU and GPU?

[9:50](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=590s) Yeah, so this is, like, basically, um, the stack on the CPU- Like- ... down to here, and then the leaf of each stack is the function that was taking time on the GPU.

[10:03](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=603s) Okay. W- what do the colors mean? Uh, the colors are different, um,

[10:09](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=609s) binaries in this case that are running on your machine. So that's why, like, blue up here for example is, is Python, and then there's, like, some, some I think CUDA, uh, down here.

[10:21](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=621s) Yeah, great question. Uh, how do you get started? Because we, um, we run, uh, on Linux using eBPF. You have a binary that you can, that you can run, uh, using systemd or Docker works as well, but we also have a DaemonSet for Kubernetes.

[10:39](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=639s) Um, and you deploy that, you get the m- manifest YAML and give it a token. Um, and then some of our customers are already using it for CPU and memory profiling, and they're starting to also integrate, um, their platforms with our GPU profiling.

[10:55](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=655s) Uh, especially like Turbopuffer, um, are, are interested in, in improving their performance of their, uh, vector engine, right?

[11:06](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=666s) And that's really it. Um, please visit, visit our booth. Um, um, you get, um... The first 10 people get, like, to sign up for a consultation get two hours for free if you want to, and we can also do discounts for seed and Series A startups.

[11:22](https://www.youtube.com/watch?v=wt8gzWR6auQ&t=682s) And that's really it. Thank you so much. [outro music]
