# Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

Mozhgan Kabiri Chimeh

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/c5-kx2bwoCk-running-llms-locally-practical-llm-performance-on) · [Video](https://www.youtube.com/watch?v=c5-kx2bwoCk)

## Running LLMs locally: capacity, throughput, and responsiveness on DGX Spark

Local Qwen benchmarks on DGX Spark show how a repeatable serving setup and quantization affect both generation speed and the wait for a response to begin.

Familiarity with LLM inference, model parameter counts, and basic Python will help you follow the benchmark and streaming example.

### What keeps an AI workload off the desktop?

[0:01](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=1s)

What happens when an AI experiment outgrows your development machine? Often, one of two constraints forces the move: the model does not fit in memory, or the required software stack is unavailable locally. Mozhgan Kabiri Chimeh, NVIDIA Developer Relations Manager, approaches that problem through hands-on experiments aimed at finding what is practical on a single system.

![Slide showing two monitors: one marked with a red cross for insufficient memory, the other with a warning for the required software stack. A line below says work that cannot run locally must move to cloud or datacenter.](https://ai.engineer/api/talk-essay-media/017b75b3587d5b743047bb5d5a2b3ed769d70d67079d6ca68ab81a034a1a6b65)

Local AI development faces memory and software-stack constraints.

Moving the workload to shared infrastructure introduces another dependency. Cost predictability, data residency, and deterministic latency become concerns as experiments approach production, while competing workloads can delay the next development run. The practical question is whether more of that iteration can happen where the developer works, without waiting for shared resources.

### A desktop system with a portable software stack

[1:39](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=99s)

[DGX Spark](https://www.nvidia.com/en-gb/products/workstations/dgx-spark/) is a standalone system built around the **GB10 Grace Blackwell Superchip**, combining CPU and GPU with unified memory. Its 128 GB of unified memory and FP4 support underpin the advertised capacity to work with models of up to around 200 billion parameters. That is a model-capacity claim, not a measured generation rate or a guarantee for every context length and concurrency setting.

The other part of the proposition is software continuity. Spark runs the NVIDIA AI software stack used in production environments, allowing workflows to move from the desktop to a data center or cloud with minimal changes. Local hardware supplies a place to develop and iterate; cloud resources remain part of the eventual scaling path.

### Make every benchmark run inspectable

[2:40](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=160s)

The experiment serves **Qwen** models locally through [vLLM](https://build.nvidia.com/spark/vllm), using an NVIDIA-optimized container across model sizes and precision formats. The container establishes a consistent software environment intended to match the data-center serving stack. The linked serving guide is a living resource; its current recipes do not identify the exact container or software versions used for these measurements.

An automated harness applies the same protocol to models ranging from 1.5B to 14B parameters:

1. **Isolate the environment** with Docker.
2. **Perform three mandatory warm-up runs** before measurement.
3. **Log GPU metrics every second** in the background.
4. **Create a unique run directory** from a precise timestamp and a sanitized model ID.
5. **Save the evidence**: full model endpoint responses, metrics, metadata, and benchmark test results.

This makes each result traceable to a particular execution. A throughput number is useful only if the surrounding run information survives with it.

The methodology slide pairs the orchestration script with its resulting directory of artifacts and example startup commands. The directory is as important as the launcher: it preserves the information needed to inspect a run after the server has stopped.

![Methodology slide with a script on the left, an annotated run-directory listing on the right, and example launch commands below. Yellow outlines highlight GPU logging and benchmark calls.](https://ai.engineer/api/talk-essay-media/c25f673030ae3fc3fc5477c57aeb8de21bfce73af1ac61930b59cdd72e7e18cc)

The reproducible harness pairs an orchestration script with saved run artifacts and startup commands.

### Measure when the response begins

[4:13](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=253s)

End-to-end latency measures how long a request takes to finish. **Time to first token** addresses a different part of the experience: how long the user waits before the response begins. A fast first response can make an application feel responsive even while the rest of the answer is still being generated.

The demonstrated `stream_once` function explicitly consumes the vLLM streaming response and timestamps its first chunk. The essential Python timing pattern is:

```python
from time import perf_counter
from openai import OpenAI


def stream_once(client: OpenAI, model: str, prompt: str) -> dict:
    started = perf_counter()
    first_chunk_at = None
    parts = []

    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    try:
        for chunk in stream:
            if first_chunk_at is None:
                first_chunk_at = perf_counter()
            for choice in chunk.choices:
                content = choice.delta.content
                if content:
                    parts.append(content)
    finally:
        stream.close()

    finished = perf_counter()
    return {
        "first_chunk_s": (
            None if first_chunk_at is None
            else first_chunk_at - started
        ),
        "end_to_end_s": finished - started,
        "text": "".join(parts),
    }
```

Here the timer records the first yielded chunk, even if it contains no visible text. That preserves the described first-chunk proxy; the talk does not specify token-event filtering. Current [vLLM per-request metrics](https://docs.vllm.ai/en/latest/features/per_request_metrics/) define server-side TTFT from scheduling to the first generated output token, so that metric is not interchangeable with this client-side measurement.

The comparison then runs smaller instruction models and larger optimized variants under the same local setup. Its target is realistic developer behavior, rather than the theoretical limits of the hardware. Keeping that scope in view matters when interpreting the results: these are measurements of a serving workflow, not a universal ranking of model sizes or precision formats.

### What model size and precision do to completion speed

[5:34](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=334s)

The completion-throughput chart starts with the **1.5B Instruct** model at 61.73 tokens/s. The **14B NVFP4** model, despite having nearly ten times as many parameters, delivers 20.19 tokens/s in the same local test suite.

**NVFP4**, NVIDIA’s [four-bit floating-point quantization format](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/), is central to this operating point. Kabiri Chimeh describes the 14B result as retaining a sophisticated model at a generation rate faster than average human reading speed. Those are her practical assessments; the benchmark does not include a task-quality evaluation or a reading-speed study.

The **14B Base** model makes the precision comparison more concrete:

| Model variant | Completion throughput |
| --- | ---: |
| 1.5B Instruct | 61.73 tokens/s |
| 14B NVFP4 | 20.19 tokens/s |
| 14B Base | 8.40 tokens/s |

These are Kabiri Chimeh’s reported DGX Spark/vLLM results. The 14B base result shows why parameter count alone cannot predict usable speed: at the same nominal model size, the optimized variant produces substantially more completion tokens per second.

![Six-bar throughput chart: 1.5B Instruct 61.73, 8B FP8 23.88, 8B Base 14.60, 14B FP8 14.78, 14B NVFP4 20.19, and 14B Base 8.40 tokens per second. Yellow annotations highlight the first value and connect the final two bars.](https://ai.engineer/api/talk-essay-media/ebc27d6a1f9478393587290ed30bd063bede3b4fa43bec19850e8d79e13d32e6)

Throughput chart shows 20.19 tokens per second for 14B NVFP4 versus 8.40 for 14B Base.

For this experiment, precision format is a consequential part of the hardware choice. Spark makes it possible to test that trade-off locally and see how it affects the prototype. The gain should remain scoped to this setup: exact checkpoints, baseline precision, prompt and output lengths, concurrency, and the throughput denominator are not specified, so the chart is not a promise of the same rates in another application.

### Fitting a model is different from feeding it quickly

[7:31](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=451s)

Throughput describes how quickly the answer continues; first-token latency describes how quickly it starts. Larger models generally require more computation before producing their first token, and the next chart examines that initial wait.

The displayed TTFT p50 is 0.07 seconds for 14B NVFP4 and 0.24 seconds for 14B Base. Kabiri Chimeh reports the 14B NVFP4 model as 3.4 times faster to first token than the unoptimized 14B base model. These values describe the local experiment and its first-chunk timing approach.

![Bar chart labeled TTFT p50 in seconds compares six models, with values 0.03, 0.06, 0.08, 0.09, 0.07, and 0.24. A yellow arrow between the 14B Base and NVFP4 bars is labeled “3.4x faster.”](https://ai.engineer/api/talk-essay-media/f7c28f7007c6d09af1f2673f04b411a607904293c648a89d245639f2a0c72b8b)

The first-token chart highlights 0.07 seconds for 14B NVFP4 versus 0.24 for 14B Base.

**Memory capacity is not memory bandwidth.** Capacity determines whether the model can fit; performance also depends on how efficiently the system moves the data needed to run it. The large unified-memory pool therefore answers a different question from either of the performance charts.

Reducing the representation size with NVFP4 helps explain the observed improvement: a more compact representation reduces the data burden associated with model values. Kabiri Chimeh describes this as increasing intelligence per byte, making a 14B model feel more like a smaller one. That is an interpretation of responsiveness, rather than a separate measurement of equivalent model quality or identical latency.

### Keep iteration local, then scale the workflow

[9:23](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=563s)

The experiments lead Kabiri Chimeh to three recommended uses:

- **Steady-state workloads:** recurring work that can use dedicated local resources.
- **Privacy-sensitive data:** development where keeping data local matters.
- **Rapid prototyping:** frequent iterations that benefit from immediate access to the system.

The broader workflow includes building and fine-tuning locally with the software stack also used in DGX Cloud. The measurements here establish inference behavior, not fine-tuning performance.

The practical handoff is the [DGX Spark playbook collection](https://build.nvidia.com/spark), which Kabiri Chimeh points to for the software and workflows behind local development. Start with a local workload, measure its behavior, and iterate while the system is directly available. When the application is ready for more resources, carry the workflow to a data center or cloud.

## Resources

- [DGX Spark playbooks](https://build.nvidia.com/spark) — NVIDIA's practical guides for local inference, fine-tuning, and other AI workflows on DGX Spark.
- [Serve LLMs with vLLM](https://build.nvidia.com/spark/vllm) — Current instructions for serving models in an NVIDIA container with an OpenAI-compatible endpoint and hardware-specific launch recipes.
- [How NVFP4 represents model values](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/) — NVIDIA's explanation of four-bit floating-point values, micro-block scaling, and accuracy trade-offs.
- [DGX Spark hardware specifications](https://www.nvidia.com/en-gb/products/workstations/dgx-spark/) — Specifications for GB10, unified memory, memory bandwidth, and supported development workloads.
- [NVFP4 performance update for DGX Spark](https://forums.developer.nvidia.com/t/nvfp4-performance-update/367781) — An April 2026 account of GB10 NVFP4 support and ongoing improvements to inference kernels and frameworks.

## Slides

### Challenges Facing Local AI Development

Insufficient local memory or a missing software stack pushes AI work to the cloud or a datacenter, where competing workloads can delay development.

[0:36](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=36s) [Two monitor illustrations show insufficient memory and missing software, with workloads moving to the cloud or a datacenter.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae51ef8c22ae8cbde9ecc66daf34a671668c1c887f7cc8f8b9e4e990bd678db8.webp)

Challenges Facing Local AI Development

Local system does not provide enough memory for AI Workloads

Local system does not provide the required software stack

Work that can’t run locally must be moved to cloud or datacenter

NVIDIA

### DGX Spark

DGX Spark pairs local memory with NVIDIA’s AI software stack in a compact system. It can run standalone or serve as a network-connected compute resource.

[1:37](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=97s) [DGX Spark features include local memory, NVIDIA AI software support, a compact form factor, and standalone or network use.](https://assets.ai.engineer/sites/aiecode2025/sha256/8541c6aabe3ce2ca7b67addc88c77984517b7f092fbb080210f449ed40052f5f.webp)

DGX Spark
Designed to build and run AI

• Large local memory
• Support for NVIDIA AI software stack
• Compact, power efficient form factor
• Configure as standalone system or network connected compute resource

Interface text:
NVIDIA DGX Spark
jupyterlab
[illegible]

### NVIDIA GB10 Grace Blackwell Superchip

GB10 combines a Blackwell GPU and 20-core Arm CPU with 128 GB of shared memory. The speaker says that memory and FP4 support enable local work with models of up to around 200 billion parameters.

[1:46](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=106s) [A chip image accompanies specifications for the GPU, CPU, NVLink interface, and 128 GB of unified memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/bdc2b5253335fc8324385b69ddeedeff45c2297c5a29ee7c715b3d534e7c7ba4.webp)

NVIDIA GB10 Grace Blackwell Superchip

• NVIDIA Blackwell GPU
  • Support for FP4 data format, provides up to 1 petaFLOP of AI performance
• 20 Core Arm CPU
  • 10 high performance + 10 efficiency cores
• NVIDIA NVLink C2C Interface
  • ~5x PCIe bandwidth
• 128GB LPDDR5x Coherent Unified System Memory
  • GPU & CPU can share all memory

NVIDIA

### Methodology: The Reproducible Harness

The harness logs GPU metrics while running two benchmarks, then saves the model response, metadata and results in a timestamped run directory. The slide also shows commands to start vLLM and run the harness.

[2:38](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=158s) [Code excerpts illustrate benchmark execution, GPU logging, saved run artifacts, and example startup commands.](https://assets.ai.engineer/sites/aiecode2025/sha256/bcadb0fa48641f5d836bf61a14820250a7abd5845ee2bbfba52ad4c35a0387eb.webp)

Methodology: The Reproducible Harness

echo "Saving model info..."
curl -s "http://localhost:${PORT}/v1/models" > "${RUN_DIR}/models.json"

cat > "${RUN_DIR}/run_metadata.txt" <<EOF
model=${MODEL}
model_safe=${MODEL_SAFE}
port=${PORT}
max_model_len=${MAX_MODEL_LEN}
run_dir=${RUN_DIR}
timestamp=$(date +"%F %T")
host=$(hostname)
EOF

echo "Saving initial metrics..."
./05_collect_metrics.sh before
mv metrics_before.txt "$RUN_DIR/"

echo "Starting GPU logger..."
./02_log_gpu.sh "$RUN_DIR/gpu_metrics.log" >/dev/null 2>&1 &
GPU_LOG_PID=$!

cleanup() {
  echo "Stopping GPU logger..."
  kill "$GPU_LOG_PID" >/dev/null 2>&1 || true
}
trap cleanup EXIT

echo "Running basic benchmark..."
MODEL="$MODEL" python 03_basic_benchmark.py
mv bench_results.txt "$RUN_DIR/"

echo "Running TTFT benchmark..."
MODEL="$MODEL" python 04_ttft_benchmark.py
mv ttft_results.txt "$RUN_DIR/"

vllm_run_20260402_154351_Qwen_Qwen2.5-Math-1.5B-Instruct/  <-- ANNOTATED
├── models.json       <-- Saved /v1/models response
├── run_metadata.txt  <-- All model/hparams in a clean text file
├── gpu_metrics.log   <-- The long CSV from 02_log_gpu.sh
├── bench_results.txt <-- Text results from the E2E benchmark
├── ttft_results.txt  <-- Text results from the TTFT benchmark

export MODEL="Qwen/Qwen2.5-Math-1.5B-Instruct"
export MAX_MODEL_LEN=4096
./01_start_vllm.sh
./07_run_all.sh

NVIDIA

### Measuring UX: The TTFT Script

The script starts a timer before a streaming request and timestamps the first nonempty content chunk to calculate time to first token. If no content arrives, it uses the end time.

[4:11](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=251s) [A streaming response script highlights the start time, request, response content, and time-to-first-token calculation.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fdd0331ee7f2c9eaaea38b0e9eb2b6fa0e7272da09bdd8b19b43605f8f9a1ae.webp)

Measuring UX: The TTFT Script

def stream_once() -> Tuple[float, float, int]:
    start = time.perf_counter()
    first_token_time = None
    total_chars = 0

    with requests.post(URL, headers=HEADERS, json=PAYLOAD, stream=True, timeout=180) as r:
        r.raise_for_status()

        for raw_line in r.iter_lines(decode_unicode=True):
            if not raw_line:
                continue

            line = raw_line.strip()
            if not line.startswith("data: "):
                continue

            data = line[len("data: "):]

            if data == "[DONE]":
                break

            try:
                obj = json.loads(data)
            except json.JSONDecodeError:
                continue

            choices = obj.get("choices", [])
            if not choices:
                continue

            delta = choices[0].get("delta", {})
            content = delta.get("content", "")

            if content and first_token_time is None:
                first_token_time = time.perf_counter()

            total_chars += len(content)

    end = time.perf_counter()

    if first_token_time is None:
        first_token_time = end

    ttft = first_token_time - start
    e2e = end - start
    return ttft, e2e, total_chars

NVIDIA

### Performance Deep-Dive: Throughput

In local vLLM tests on DGX Spark, the 14B NVFP4 model reached 20.19 tokens per second, compared with 8.40 for 14B Base. The smaller 1.5B Instruct model reached 61.73.

[5:01](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=301s) [Bar chart compares throughput across six model variants, ranging from 8.40 to 61.73 tokens per second.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d80a64213be588112803fe1f9f8b834d764ceec0edca7bc1e44a9e077e02a99.webp)

Performance Deep-Dive: Throughput

Y-axis: Tokens / sec
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60
X-axis: Model
1.5B Instruct: 61.73
8B FP8: 23.88
8B Base: 14.60
14B FP8: 14.78
14B NVFP4: 20.19
14B Base: 8.40

NVIDIA

### The UX Reality: Latency & TTFT

Time to first token measures how quickly a model starts responding. The 14B NVFP4 model reaches its first token in a median 0.07 seconds, versus 0.24 seconds for 14B Base: 3.4x faster.

[7:28](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=448s) [Bar chart compares median time to first token across six models, highlighting 0.07 seconds for 14B NVFP4 versus 0.24 seconds for 14B Base.](https://assets.ai.engineer/sites/aiecode2025/sha256/6bf1ee6a56b202f86228733c29132a892d0094b5b46c1839cc6a56d3efbb9645.webp)

The UX Reality: Latency & TTFT

Y-axis: TTFT p50 (seconds)
Y-axis ticks: 0.00, 0.05, 0.10, 0.15, 0.20
X-axis: Model
1.5B Instruct: 0.03
8B FP8: 0.06
8B Base: 0.08
14B FP8: 0.09
14B NVFP4: 0.07
14B Base: 0.24

3.4x faster

NVIDIA

### Accelerate Your AI Developer Journey

DGX Spark supports local prototyping and fine-tuning with the same software stack used in DGX Cloud. The speaker points to build.nvidia.com/spark for playbooks and software before scaling to a data center or cloud.

[9:21](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=561s) [Slide promotes local prototyping and cloud scaling, with a screenshot of NVIDIA's DGX Spark developer resources and the build.nvidia.com/spark address.](https://assets.ai.engineer/sites/aiecode2025/sha256/dd75d760f3e2104890cb421cd90d78fcee3d4a15dd4ecff91ededdaa954da970.webp)

Accelerate Your AI Developer Journey
Prototype locally. Scale in the cloud

Start Building on DGX Spark
Find instructions and examples to customize and run AI workloads on DGX Spark

Connecting Remotely?
Get started with NVIDIA Sync
Set Up Remote Access to your DGX Spark
NVIDIA Sync helps set up and configure SSH access
10 MIN
Configure Now

First Time Here?
Try these developer quickstarts

Comfy UI — 45 MIN
Install and use ComfyUI to generate images

DGX Dashboard — 45 MIN
Monitor your DGX system and launch JupyterLab

Open WebUI with Ollama — 30 MIN
Install Open WebUI and use Ollama to chat with models on your Spark

VS Code — 5 MIN
Install and use VS Code locally or remotely

See More Playbooks Below

Explore Playbooks
Detailed instructions to set up and run popular AI workflows on DGX Spark

Optimized JAX — 2 HOURS
Optimize JAX to Run on Spark

Build and Deploy a Multi-Agent Chatbot — 30 MIN
Deploy a multi-agent chatbot system and chat with agents on yo...

NVFP4 Quantization — 1 HOUR
Optimize models to use NVFP4 to run on Spark

AI Workbench — 10 MIN
Install and use AI Workbench to clone and run a reproducible RAG...

TRT LLM — 1 HOUR
Install and configure TRT LLM to run on a single Spark or on two...

build.nvidia.com/spark
NVIDIA

Interface text:
NVIDIA
Explore
Models
Blueprints
GPUs
Docs
Search
?
Login

## Full transcript

[0:01](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=1s) Hello, everyone. I'm Mozhgan Kabiri Chimeh, Developer Relations Manager at NVIDIA, where I work closely with developers building and deploying AI systems. Today, we're looking at running LLMs locally, practical LLM performance on DGX Spark.

[0:20](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=20s) This isn't a theoretical talk, it's a data-backed journey through the trade-offs of modern AI infrastructure. Findings are based on hands-on experiments with the goal of understanding what's actually practical on a single system.

[0:38](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=38s) The evolution in AI puts greater demand on developer systems, creating two main challenges. You either run out of memory or you do not have access to the right software stack, then you end up on pushing everything to the cloud or data center.

[0:57](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=57s) As models move from experiments to production, concerns like cost predictability, data residency, and deterministic latency take center stage. And iteration speed often depends, often depends on access to shared infrastructure.

[1:16](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=76s) And since your work will be scheduled against other competing workloads, it causes delays in development work as well. So the question becomes: Can we bring some of that workflow closer to where development actually happens?

[1:31](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=91s) To maximize developer productivity, local solutions to these challenges are required.

[1:39](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=99s) DGX Spark is designed from the ground up to build and run AI and can be used as a standalone system.

[1:48](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=108s) It's powered by the GB10 Grace Blackwell Superchip, combining CPU and GPU with a unified memory architecture. With hundred and twenty-eight gigabyte of unified memory and FP4 support, it enables developers to work with models of up to around two hundred billion parameters locally on a system that fits under the, under a desk

[2:13](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=133s) or on top of your desk. It runs the same NVIDIA AI software stack used in production environments, meaning workflows can move from desktop to data center or cloud with minimal changes.

[2:29](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=149s) The key idea here is not replacing the cloud, but bringing powerful AI development closer to the developer.

[2:40](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=160s) This is my setup. Everything runs locally and is reproducible. To serve the models, I used vLLM.

[2:49](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=169s) With the set of Qwen models across different sizes and precision formats, I ran this inside an NVIDIA optimized container. This would ensure that our environment is identical to what you would deploy in a data center.

[3:04](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=184s) Instead of just showing raw results, I want to show you the how.

[3:09](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=189s) I built an automated benchmarking harness. Every model run from one point five billion to fourteen billion follows the same strict protocol: environment isolation via Docker, three mandatory warm-up runs, and background GPU metrics logging at one-second interval.

[3:28](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=208s) On the left, you are looking at the orchestrator script. For every execution, the script automatically generates a unique directory using a precise timestamp and a sanitized model ID.

[3:44](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=224s) For every model run, it captures the full model's endpoint response and necessary metrics. On the right, you see the result, a clean versioned artifact of the run. It contains everything to verify the findings, the metadata, and the test results from the benchmarks.

[4:06](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=246s) On the lower right, you can see an example command for getting things started.

[4:13](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=253s) Now, let's look at the actual measurement logic. In an AI application, end-to-end latency is important, but time to first token is the metric that defines the user's perceived performance.

[4:29](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=269s) If the first token arrives instantly, the application feels responsive, and the script here shows how we timestamp the very first chunk of a streaming response. In this script, we are not just calling an API and waiting for a result, we are explicitly handling the streaming response from the vLLM server.

[4:54](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=294s) If you look at the highlighted block in the stream_once function, you'll see the timestamping logic.

[5:03](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=303s) To explore this in practice, I ran a series of experiments using vLLM on DGX Spark. I tested different models from the smaller instruction models to larger optimized variants, all under the same setup.

[5:19](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=319s) Everything was served locally. The goal here wasn't to push theoretical limits, but to understand realistic behavior in a developer workflow. Let's dive into the raw performance data I captured.

[5:34](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=334s) This bar chart represents the completion token per second across the test suite.

[5:42](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=342s) At the far left, you see the one point five billion instruct model delivering a massive sixty-one point seventy-three tokens per second. But the most interesting data point for us is the fourteen billion NVFP4 model Despite being nearly ten times larger than

[6:07](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=367s) the one point five billion model, it still achieves twenty point nineteen tokens per second.

[6:16](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=376s) I would say this is a critical engineering sweet spot. By leveraging NVIDIA's NVFP4 four-bit floating-point quantization, we were-- we are able to maintain a sophisticated high-intelligent model at a throughput that is still faster than the average human reading speed.

[6:39](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=399s) What becomes very clear here is how aggressively throughput drops as model scale-- as models scale and how much quantization helps. Have a look at the fourteen billion base model.

[6:56](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=416s) It drops to just eight point forty tokens per second.

[7:01](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=421s) This proves that on Blackwell archite-- on Blackwell hardware, the choice of quantization format is just as important as the hardware itself. It's what allows the DGX Spark to bridge the gap between research toy and production prototyping engine.

[7:20](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=440s) The Spark allowed me to experiment with different precision formats locally and understand the trade-offs in the real-time.

[7:31](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=451s) While throughput is a measure of raw power, time to first token is the metric that defines user experience. It determines whether an application feels instant or broken. In other words, it reflects how quickly the model starts responding.

[7:51](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=471s) As we see in the results, the DGX Spark delivers exceptional responsiveness.

[7:58](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=478s) The increase for larger models is expected. More parameters mean more computation before the first token is generated.

[8:07](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=487s) Out of all from this chart, the interesting ones is a comparison between the fourteen billion parameter models, the base model, and the NVFP4 model. As we can see, as we can see, the fourteen billion NVFP4 is three point four times faster to first token than

[8:33](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=513s) the unoptimized fourteen base model. A key takeaway here from this data is that memory capacity is not the same as the memory bandwidth. While the DGX Spark's hundred and twenty-eight gigabytes of unified memory allows us to fit massive models up to two hundred billion parameters, our throughput

[8:58](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=538s) is still governed by how efficiently we can move data.

[9:04](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=544s) This is why NVFP4 is the hero here.

[9:10](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=550s) It effectively increases our intelligence per byte, allowing a fourteen billion model to feel as responsive as much smaller one.

[9:23](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=563s) After running all of these, three things stood out to me on when to use the DGX Spark. For steady-state workloads, privacy-sensitive data, and rapid prototyping. It allows you to build and fine-tune locally with the exact same software stack used in DGX Cloud.

[9:47](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=587s) Visit build.nvidia.com/spark to access the playbooks and software stack I used for these benchmarks. And in one line, run locally, iterate quickly, and when ready, scale to data center or cloud.

[10:04](https://www.youtube.com/watch?v=c5-kx2bwoCk&t=604s) That's the workflow DGX Spark enables.
