Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

What happens when an AI experiment outgrows your development machine? Often, one of two constraints forces the move: the model does not fit in memory, or the required software stack is unavailable locally. Mozhgan Kabiri Chimeh, NVIDIA Developer Relations Manager, approaches that problem through hands-on experiments aimed at finding what is practical on a single system.

Moving the workload to shared infrastructure introduces another dependency. Cost predictability, data residency, and deterministic latency become concerns as experiments approach production, while competing workloads can delay the next development run. The practical question is whether more of that iteration can happen where the developer works, without waiting for shared resources.

DGX Spark is a standalone system built around the GB10 Grace Blackwell Superchip, combining CPU and GPU with unified memory. Its 128 GB of unified memory and FP4 support underpin the advertised capacity to work with models of up to around 200 billion parameters. That is a model-capacity claim, not a measured generation rate or a guarantee for every context length and concurrency setting.

The other part of the proposition is software continuity. Spark runs the NVIDIA AI software stack used in production environments, allowing workflows to move from the desktop to a data center or cloud with minimal changes. Local hardware supplies a place to develop and iterate; cloud resources remain part of the eventual scaling path.

The experiment serves Qwen models locally through vLLM, using an NVIDIA-optimized container across model sizes and precision formats. The container establishes a consistent software environment intended to match the data-center serving stack. The linked serving guide is a living resource; its current recipes do not identify the exact container or software versions used for these measurements.

An automated harness applies the same protocol to models ranging from 1.5B to 14B parameters:

1. Isolate the environment with Docker.
2. Perform three mandatory warm-up runs before measurement.
3. Log GPU metrics every second in the background.
4. Create a unique run directory from a precise timestamp and a sanitized model ID.
5. Save the evidence: full model endpoint responses, metrics, metadata, and benchmark test results.

This makes each result traceable to a particular execution. A throughput number is useful only if the surrounding run information survives with it.

The methodology slide pairs the orchestration script with its resulting directory of artifacts and example startup commands. The directory is as important as the launcher: it preserves the information needed to inspect a run after the server has stopped.

End-to-end latency measures how long a request takes to finish. Time to first token addresses a different part of the experience: how long the user waits before the response begins. A fast first response can make an application feel responsive even while the rest of the answer is still being generated.

The demonstrated stream_once function explicitly consumes the vLLM streaming response and timestamps its first chunk. The essential Python timing pattern is:

from time import perf_counter
from openai import OpenAI


def stream_once(client: OpenAI, model: str, prompt: str) -> dict:
    started = perf_counter()
    first_chunk_at = None
    parts = []

    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    try:
        for chunk in stream:
            if first_chunk_at is None:
                first_chunk_at = perf_counter()
            for choice in chunk.choices:
                content = choice.delta.content
                if content:
                    parts.append(content)
    finally:
        stream.close()

    finished = perf_counter()
    return {
        "first_chunk_s": (
            None if first_chunk_at is None
            else first_chunk_at - started
        ),
        "end_to_end_s": finished - started,
        "text": "".join(parts),
    }

Here the timer records the first yielded chunk, even if it contains no visible text. That preserves the described first-chunk proxy; the talk does not specify token-event filtering. Current vLLM per-request metrics define server-side TTFT from scheduling to the first generated output token, so that metric is not interchangeable with this client-side measurement.

The comparison then runs smaller instruction models and larger optimized variants under the same local setup. Its target is realistic developer behavior, rather than the theoretical limits of the hardware. Keeping that scope in view matters when interpreting the results: these are measurements of a serving workflow, not a universal ranking of model sizes or precision formats.

The completion-throughput chart starts with the 1.5B Instruct model at 61.73 tokens/s. The 14B NVFP4 model, despite having nearly ten times as many parameters, delivers 20.19 tokens/s in the same local test suite.

NVFP4, NVIDIA’s four-bit floating-point quantization format, is central to this operating point. Kabiri Chimeh describes the 14B result as retaining a sophisticated model at a generation rate faster than average human reading speed. Those are her practical assessments; the benchmark does not include a task-quality evaluation or a reading-speed study.

The 14B Base model makes the precision comparison more concrete:

These are Kabiri Chimeh’s reported DGX Spark/vLLM results. The 14B base result shows why parameter count alone cannot predict usable speed: at the same nominal model size, the optimized variant produces substantially more completion tokens per second.

For this experiment, precision format is a consequential part of the hardware choice. Spark makes it possible to test that trade-off locally and see how it affects the prototype. The gain should remain scoped to this setup: exact checkpoints, baseline precision, prompt and output lengths, concurrency, and the throughput denominator are not specified, so the chart is not a promise of the same rates in another application.

Throughput describes how quickly the answer continues; first-token latency describes how quickly it starts. Larger models generally require more computation before producing their first token, and the next chart examines that initial wait.

The displayed TTFT p50 is 0.07 seconds for 14B NVFP4 and 0.24 seconds for 14B Base. Kabiri Chimeh reports the 14B NVFP4 model as 3.4 times faster to first token than the unoptimized 14B base model. These values describe the local experiment and its first-chunk timing approach.

Memory capacity is not memory bandwidth. Capacity determines whether the model can fit; performance also depends on how efficiently the system moves the data needed to run it. The large unified-memory pool therefore answers a different question from either of the performance charts.

Reducing the representation size with NVFP4 helps explain the observed improvement: a more compact representation reduces the data burden associated with model values. Kabiri Chimeh describes this as increasing intelligence per byte, making a 14B model feel more like a smaller one. That is an interpretation of responsiveness, rather than a separate measurement of equivalent model quality or identical latency.

The experiments lead Kabiri Chimeh to three recommended uses:

• Steady-state workloads: recurring work that can use dedicated local resources.
• Privacy-sensitive data: development where keeping data local matters.
• Rapid prototyping: frequent iterations that benefit from immediate access to the system.

The broader workflow includes building and fine-tuning locally with the software stack also used in DGX Cloud. The measurements here establish inference behavior, not fine-tuning performance.

The practical handoff is the DGX Spark playbook collection, which Kabiri Chimeh points to for the software and workflows behind local development. Start with a local workload, measure its behavior, and iterate while the system is directly available. When the application is ready for more resources, carry the workflow to a data center or cloud.