Introduction to LLM serving with SGLang

How do you get comfortable serving a model yourself—and then make it fast enough for your application? Philip Kiely and Yineng Zhang build this workshop around that progression: deploy a first model, adjust its performance, and find the implementation points where you can contribute changes. Zhang is introduced as an SGLang core maintainer working on inference; Kiely introduces himself with a joke about his B+ in linear algebra. The exercises are intended to accommodate both backgrounds.

SGLang is an open-source serving framework for language and vision models, in the same category as vLLM and TensorRT-LLM. Kiely emphasizes GPU performance, production readiness, rapid support for releases from Qwen and DeepSeek, and the ability to fix the implementation yourself. He identifies Baseten and xAI, serving Grok, among its users, alongside inference providers, cloud providers, researchers, universities, and product companies. The practical attraction is a configurable server whose internals remain accessible when configuration alone is insufficient.

The original SGLang paper appeared in December 2023. Kiely reports that the project approached 15,000 GitHub stars within 18 months. The accompanying timeline puts that growth alongside expanding adoption and community activity.

Zhang describes an engineering route into the project: optimizing click-through-rate ranking models and LLM inference at Meituan, connecting with SGLang creator Lianmin, and then working closely with Lianmin and Yin. That work also brought him into collaboration with Zihao on FlashInfer, which SGLang uses for attention and sampling kernels. This connection matters because serving performance depends on both runtime orchestration and the GPU operations underneath it.

Kiely’s funding aside concerns the adjacent Chatbot Arena ecosystem. The funding announcement identifies LMArena as the recipient, with a16z and UC Investments co-leading; it should not be read as an investment into SGLang.

The workshop’s QR code and displayed URL lead to the same setup materials. The deployment itself starts with a simple arrangement: an SGLang server command running inside a Docker container. Truss packages the dependencies and launch command in YAML, bundles them, and sends the deployment to a Baseten GPU. Baseten is the workshop host because Kiely could obtain free GPUs there; Truss is the packaging layer, not a requirement of SGLang itself.

The exercises use L4 GPUs for cost, availability, and FP8 support. Kiely also mentions H100 and H200 hardware, while describing Blackwell support as forthcoming at the time. In Truss, the accelerator setting selects the hardware. Beyond that selection, using SGLang means understanding its launch flags, their support matrix, and their interactions. Aggressive speculation and very large batches can work against each other; quantization can combine with other optimizations. The first exercise establishes a working deployment before adding those choices.

Once the server is up, the workshop calls it from call.ipynb, a Jupyter notebook, or an ordinary OpenAI SDK client. Each participant copies the deployment’s model ID from the Baseten workspace into the endpoint URL. That deployment ID identifies where the request goes; the client’s model field identifies the served model. A compact Python client keeps those settings separate:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["SGLANG_BASE_URL"],
    api_key=os.environ["SGLANG_API_KEY"],
)

response = client.chat.completions.create(
    model=os.environ["SGLANG_MODEL"],
    messages=[
        {"role": "user", "content": "Explain prefill and decode in two sentences."}
    ],
    max_tokens=128,
)

print(response.choices[0].message.content)

Set SGLANG_BASE_URL to the deployment’s OpenAI-compatible base URL, including the deployment ID where the hosting service requires it.

With the first endpoint available, the workshop moves to two performance exercises: CUDA graph capture coverage and EAGLE-3 speculative decoding. The repository and recording remain the follow-along materials for participants who are still completing deployment.

For direct access to the GPU host, Zhang switches to Runpod: the workshop’s Baseten environment does not expose SSH. He launches Llama-3.1-8B-Instruct on an L4 with SGLang already installed. He names FA3 as the attention backend and calls it the default, but that particular hardware/backend combination is uncertain: the current attention backend guide assigns FA3 to supported Hopper configurations and FlashInfer or Triton to Ampere/Ada. The tuning lesson here depends on the capture logs, not on resolving that spoken backend label.

The top terminal runs the model server; the bottom runs LM Eval to generate traffic. At startup, CUDA graphs are enabled, with captured batch sizes of 1, 2, 4, and 8. The logs distinguish prefill batches from decode batches. When the decode batch reaches ten running requests, it exceeds the capture ceiling of eight, and the log reports cuda graph: False. Zhang reports approximately 155 aggregate generation tokens per second with ten running requests, or roughly 15 tokens per second per user. The important diagnostic is that an optimization enabled at startup is no longer active for this observed batch.

Zhang stops the client and server, then raises the maximum captured batch size to 32. The historical launch flag discussed in the recording can be expressed as:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --cuda-graph-max-bs 32

After the restart and another LM Eval run, the observed decode batch has 13 running requests and reports cuda graph: True. Zhang estimates about 12 tokens per second per user in that later sample, then says the two samples are difficult to compare. The demonstrated result is coverage of a larger decode batch, not a controlled throughput improvement: concurrency changed between observations. He also points participants to a recorded cudaGraphMaxBatchSize demo.

An audience question brings the traffic generator into focus. LM Eval specifies the model and an OpenAI-compatible endpoint on localhost, port 8000, because it runs on the same Runpod node as the server. Its configured concurrency is 128; that client-side setting is distinct from the ten or thirteen requests observed in individual decode batches. A maximum generation length is also configured, though its value is not stated.

The evaluation uses GSM8K with the chat-completion interface, an applied chat template, and eight-shot prompting. Zhang describes a 0.15 limit on 1,319 prompts, producing approximately 200 evaluation prompts. He offers to add the command to the repository. Kiely then distinguishes their same-host SSH arrangement from remote HTTP testing, where he recommends a request-based benchmark. That is a deployment distinction rather than an LM Eval limitation: the Evaluation Harness API guide also supports configurable remote endpoints.

Why default to eight? Zhang explains that an unset maximum begins as None, then resolves internally according to hardware and model. He cites L4 VRAM and tensor parallelism of one, or TP1, for this demonstration’s default. When asked about choosing too large a value for a larger model, he gives an empirical procedure rather than a safe upper bound:

1. Inspect the startup parameters and captured batch sizes.
2. Generate traffic representative of the intended workload.
3. Inspect decode logs for batches where CUDA graphs are disabled.
4. Adjust the capture maximum and rerun the workload.

The default is a starting point shaped by the deployment, not a guarantee that every real batch will fit its capture coverage.

The next optimization changes how tokens are generated. EAGLE-3, released shortly before the workshop, proposes future tokens for the target model to verify. Higher acceptance makes more of the speculative work useful. The workshop includes a Truss example: enter its directory and run truss push, with separate paths for the target model and the draft model. Zhang verbally selects EAGLE as the algorithm and identifies an EAGLE-3 draft for a Llama 3 8B model.

Kiely contrasts this with pairing an independent small language model and a larger target, such as Llama 1B and Llama 8B. EAGLE-3 instead uses features from multiple target-model layers to support drafting. More precisely, its paper describes multi-layer feature fusion and direct token prediction; earlier EAGLE predicts top-layer features. When following the recording with a current installation, consult the speculative decoding configuration: current examples distinguish EAGLE3 from EAGLE and use --cuda-graph-max-bs-decode. The recording’s EAGLE selection and --cuda-graph-max-bs spelling belong to its historical configuration.

Once the target and compatible draft are selected, three parameters determine how much speculative work to attempt:

Zhang relates these values: depth three with top-k one uses four draft tokens in the example. They are connected choices, so changing one means reconsidering the others.

The repository’s playground includes bench_speculative.py for sweeping these choices. Zhang opens a single-GPU example with Llama 2 7B as the target and a draft model. It varies batch sizes 1, 2, 4, 8, and 16 alongside steps, top-k, and draft-token counts, running combinations rather than treating each setting independently.

The results expose speed and acceptance rate so that a suitable combination can become an online-serving configuration. Zhang offers batch size eight, three steps, top-k one, and four draft tokens as an illustrative candidate, not a measured winner shown in the walkthrough. Kiely adds the decisive constraint: replace the benchmark prompts with representative production inputs and outputs. Speculation guesses future tokens, so topic and content affect acceptance. A setting selected on unrelated prompts can be the wrong setting for live traffic; Zhang points out that the script’s prompts can be edited directly.

The route into contributing starts with using SGLang and reporting what breaks or what is missing. Kiely points to GitHub, the project’s social account, community Slack, and online and in-person meetups; the workshop slides retain the community links. He sketches three broad codebase areas—the runtime, domain-specific frontend, and optimized kernels—and points to DeepWiki and a contributor walkthrough for diagrams and orientation.

Zhang turns that map into concrete entry points. File an issue or feature request arising from actual use, browse the Good first issue and Help wanted labels, or look for a feature on the development roadmap. He points to VLM and VLA serving support as one possible contribution area. The roadmap helps distinguish an opportunity to join existing work from a missing feature that needs a new proposal.

The implementation tour separates responsibilities:

• SGL kernel library: Attention, normalization, activation, and GEMM implementations provide an entry point for CUDA programmers.
• SGL router: Cache-aware routing provides a separate area for work on request placement.
• Python SRT: The core inference runtime includes prefill/decode disaggregation, constrained decoding, function calling, an OpenAI-compatible server, and model implementations.

For a custom model with an architecture similar to Llama, Zhang suggests starting from the existing Llama implementation, making the necessary changes, and contributing them back. That gives contributors a concrete reference rather than requiring them to reconstruct the whole serving stack.

Before the audience questions, Kiely invites participants to an Oxen AI fine-tuning event. He recounts a demonstration in which SQL fine-tuning allegedly allowed Qwen 0.6B to beat GPT-4.1, and describes the small model as suitable for an older iPhone. Those details should remain an attributed anecdote: the closest located Oxen Text2SQL report compares against GPT-4o, and the small model’s advantage depends on the scoring metric. It does not establish the exact anecdote or measured iPhone performance. Kiely also invites people interested in CUDA, infrastructure, and model performance to consider Baseten’s engineering roles.

The first runtime-choice answer returns to extensibility. Baseten chooses engines by model and use case. Kiely characterizes basic out-of-box performance as broadly similar, then explains why SGLang appeals to his team: well-documented options enable deeper tuning, and accessible implementation paths let the team contribute fixes or model support. The operational benefit is the ability to unblock a deployment without waiting for someone else to implement the missing piece.

An audience member asks how to secure a deployment spanning multiple vendors, applications, endpoints, and subnets. Kiely places the main controls around the runtime: container packaging, least privilege, and isolation in the surrounding system. In that deployment framing, he sees no special SGLang-specific requirement compared with vLLM. Choosing an inference engine does not replace designing the security boundary around it.

The follow-up makes the boundary concrete: can development and inference remain inside an organization’s subnet, avoiding calls to external ChatGPT services? The questioner raises Defense and financial-application experience, endpoint controls, and CMMC certification. Kiely’s proposed arrangement is to place a GPU cluster in the workload’s VPC or physical data center, select an SGLang release, inspect its source, and pin that release. The model can then run directly on the GPUs and accept local requests, including over localhost. These are building blocks for an air-gapped-style deployment, not a certification procedure or compliance guarantee.

A further question asks about combining blockchain-based quantum-computing work and AI deliverables, contrasting decentralized protocols with a containerized service. Kiely offers no integration architecture: his SGLang experience is with traditional client-server applications. That leaves the boundary of the demonstrated system clear—a model-serving endpoint that another application calls.

The final technical question returns to why Baseten uses more than one engine. Its inference stack customizes vLLM, SGLang, and TensorRT-LLM; Kiely says TensorRT-LLM is probably the most heavily used of the three. vLLM often supplies broad compatibility. His concrete example is Baseten’s Gemma deployments, which used vLLM because it supported those models when they launched. The choice therefore includes an immediate operational question: which engine supports the model you need, alongside the tuning and extension work your team can perform?