Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

Talk transcript

1. 0:01

   Hello, everyone. I'm Mozhgan Kabiri Chimeh, Developer Relations Manager at NVIDIA, where I work closely with developers building and deploying AI systems. Today, we're looking at running LLMs locally, practical LLM performance on DGX Spark.

2. 0:20

   This isn't a theoretical talk, it's a data-backed journey through the trade-offs of modern AI infrastructure. Findings are based on hands-on experiments with the goal of understanding what's actually practical on a single system.

3. 0:38

   The evolution in AI puts greater demand on developer systems, creating two main challenges. You either run out of memory or you do not have access to the right software stack, then you end up on pushing everything to the cloud or data center.

4. 0:57

   As models move from experiments to production, concerns like cost predictability, data residency, and deterministic latency take center stage. And iteration speed often depends, often depends on access to shared infrastructure.

5. 1:16

   And since your work will be scheduled against other competing workloads, it causes delays in development work as well. So the question becomes: Can we bring some of that workflow closer to where development actually happens?

6. 1:31

   To maximize developer productivity, local solutions to these challenges are required.

7. 1:39

   DGX Spark is designed from the ground up to build and run AI and can be used as a standalone system.

8. 1:48

   It's powered by the GB10 Grace Blackwell Superchip, combining CPU and GPU with a unified memory architecture. With hundred and twenty-eight gigabyte of unified memory and FP4 support, it enables developers to work with models of up to around two hundred billion parameters locally on a system that fits under the, under a desk

9. 2:13

   or on top of your desk. It runs the same NVIDIA AI software stack used in production environments, meaning workflows can move from desktop to data center or cloud with minimal changes.

10. 2:29

    The key idea here is not replacing the cloud, but bringing powerful AI development closer to the developer.

11. 2:40

    This is my setup. Everything runs locally and is reproducible. To serve the models, I used vLLM.

12. 2:49

    With the set of Qwen models across different sizes and precision formats, I ran this inside an NVIDIA optimized container. This would ensure that our environment is identical to what you would deploy in a data center.

13. 3:04

    Instead of just showing raw results, I want to show you the how.

14. 3:09

    I built an automated benchmarking harness. Every model run from one point five billion to fourteen billion follows the same strict protocol: environment isolation via Docker, three mandatory warm-up runs, and background GPU metrics logging at one-second interval.

15. 3:28

    On the left, you are looking at the orchestrator script. For every execution, the script automatically generates a unique directory using a precise timestamp and a sanitized model ID.

16. 3:44

    For every model run, it captures the full model's endpoint response and necessary metrics. On the right, you see the result, a clean versioned artifact of the run. It contains everything to verify the findings, the metadata, and the text results from the benchmarks.

17. 4:06

    On the lower right, you can see an example command for getting things started.

18. 4:13

    Now, let's look at the actual measurement logic. In an AI application, end-to-end latency is important, but time to first token is the metric that defines the user's perceived performance.

19. 4:29

    If the first token arrives instantly, the application feels responsive, and the script here shows how we timestamp the very first chunk of a streaming response. In this script, we are not just calling an API and waiting for a result, we are explicitly handling the streaming response from the vLLM server.

20. 4:54

    If you look at the highlighted block in the stream_once function, you'll see the timestamping logic.

21. 5:03

    To explore this in practice, I ran a series of experiments using vLLM on DGX Spark. I tested different models from the smaller instruction models to larger optimized variants, all under the same setup.

22. 5:19

    Everything was served locally. The goal here wasn't to push theoretical limits, but to understand realistic behavior in a developer workflow. Let's dive into the raw performance data I captured.

23. 5:34

    This bar chart represents the completion token per second across the test suite.

24. 5:42

    At the far left, you see the one point five billion instruct model delivering a massive sixty-one point seventy-three tokens per second. But the most interesting data point for us is the fourteen billion NVFP4 model Despite being nearly ten times larger than

25. 6:07

    the one point five billion model, it still achieves twenty point nineteen tokens per second.

26. 6:16

    I would say this is a critical engineering sweet spot. By leveraging NVIDIA's NVFP4 four-bit floating-point quantization, we were-- we are able to maintain a sophisticated high-intelligent model at a throughput that is still faster than the average human reading speed.

27. 6:39

    What becomes very clear here is how aggressively throughput drops as model scale-- as models scale and how much quantization helps. Have a look at the fourteen billion base model.

28. 6:56

    It drops to just eight point four T tokens per second.

29. 7:01

    This proves that on Blackwell archite-- on Blackwell hardware, the choice of quantization format is just as important as the hardware itself. It's what allows the DGX Spark to bridge the gap between research toy and production prototyping engine.

30. 7:20

    The Spark allowed me to experiment with different precision formats locally and understand the trade-offs in the real-time.

31. 7:31

    While throughput is a measure of raw power, time to first token is the metric that defines user experience. It determines whether an application feels instant or broken. In other words, it reflects how quickly the model starts responding.

32. 7:51

    As we see in the results, the DGX Spark delivers exceptional responsiveness.

33. 7:58

    The increase for larger models is expected. More parameters mean more computation before the first token is generated.

34. 8:07

    Out of all from this chart, the interesting ones is a comparison between the fourteen billion parameter models, the base model, and the NVFP4 model. As we can see, as we can see, the fourteen billion NVFP4 is three point four times faster to first token than

35. 8:33

    the unoptimized fourteen base model. A key takeaway here from this data is that memory capacity is not the same as the memory bandwidth. While the DGX Spark's hundred and twenty-eight gigabytes of unified memory allows us to fit massive models up to two hundred billion parameters, our throughput

36. 8:58

    is still governed by how efficiently we can move data.

37. 9:04

    This is why NVFP4 is the hero here.

38. 9:10

    It effectively increases our intelligence per byte, allowing a fourteen billion model to feel as responsive as much smaller one.

39. 9:23

    After running all of these, three things stood out to me on when to use the DGX Spark. For steady-state workloads, privacy-sensitive data, and rapid prototyping. It allows you to build and fine-tune locally with the exact same software stack used in DGX Cloud.

40. 9:47

    Visit build.nvidia.com/spark to access the playbooks and software stack I used for these benchmarks. And in one line, run locally, iterate quickly, and when ready, scale to data center or cloud.

41. 10:04

    That's the workflow DGX Spark enables.