What Is an Inference Engine, Anyway? — Charles Frye, Modal

Tokens go in, tokens come out. A basic implementation using PyTorch and Hugging Face Transformers can be small; making it fast enough to serve users at a workable cost is the hard part. Charles Frye, drawing on inference deployment work at Modal, uses SGLang, vLLM and TensorRT-LLM to explain what grows around that simple operation.

The initial architecture has communicating processes for network IO, input and output conversion, scheduling, and GPU execution. Two components receive most of the performance attention: the code doing expensive work on the accelerator, and the scheduler deciding which work gets there. The engine must both execute efficiently and keep execution supplied with work.

Frye’s economic framing is blunt: “training is a cost center and inference is a revenue center.” In the business pattern he describes, model producers create foundation models, while deployments sell useful responses to particular inputs. Inference also feeds post-training by generating samples. The engineering spans application architecture, linear algebra and hardware—an “engineer’s playground.”

A museum placard generator gives that machinery a visible purpose. Frye’s art piece, shown at the Legion of Honor, passes what it sees through a Qwen mixture-of-experts vision-language model, served by SGLang on Modal. The output describes the scene as an artwork. Pointing it at the entrance line produced a museum-style description. Frye supplies the joke himself: “you’re not art until somebody writes one of these little placards to describe what you’ve done.” The application sits between the strictest interactive latency demands and jobs that chiefly need high throughput.

Three application profiles help identify what a deployment should optimize:

• Chatbot plus. A person waits for the response, supplies more context, and may trigger interactions with external tools. ChatGPT and Claude Code are examples.
• Background agents. Similar interactions happen without a person waiting on each model call. The useful deadline might be completing a pull request in minutes rather than returning each token immediately.
• Data processors. A model converts unstructured material, such as a PDF, into a smaller structured result suitable for a database. These deployments often prioritize total throughput.

Inside the engine, the input first undergoes prefill: the model processes the prompt. Decode then repeatedly produces one or a few output tokens. A user experiences one response, but the engine handles two kinds of work. Prefill performs much more arithmetic per request; each decode step performs less arithmetic and repeats as generation continues. These phases can follow different code paths and need different scheduling choices.

A useful workload description includes requests per second per replica, input and output tokens per request, the opportunity to reuse cached prefixes, and a latency budget. Measure lengths rather than assuming them: users choose inputs, and the model helps determine when output ends. Time to first token measures how long the caller waits for generation to begin. Time per output token, or intertoken latency, describes how quickly the response continues.

The same API can conceal very different opportunities. Chatbots and background agents often reuse substantial prefixes as an interaction grows. The chatbot profile Frye describes produces tens to hundreds of output tokens and has a tight human-facing latency budget. A document processor sees different documents, shares relatively little beyond a system prompt, and produces a short extraction relative to its input. A database backfill can tolerate more waiting if the overall job finishes efficiently.

An inference server turns an HTTP or gRPC request into a response. The engine is the machinery underneath that interface. vLLM includes both a serving layer and an engine accessible through its software interface. At the per-replica request volumes discussed here, ordinary network serving is generally easier than the model work it wraps.

The transformations are concrete. Unicode text, images, video or streams arrive in representations useful to external systems. Preprocessing turns them into model inputs, including tokens and GPU tensors. In the simplest next-token picture, the model extends a sequence of K entries to K plus one. Postprocessing converts generated tokens back into a response people or tools can use.

The request interface can appear stateless while performance depends on retained state. Once the engine keeps previously computed information in a KV cache, earlier requests leave useful work behind. An individual replica manages its cache; the surrounding deployment layer generally defines sessions and how they relate to requests. A session and a replica’s cached computation are therefore separate concerns.

In the SGLang architecture described here, server IO communicates with preprocessing and postprocessing, which communicate with a scheduler. One or more model-runner processes perform GPU work. The scheduler manages queues and scarce resources through a single point of control. It performs far fewer operations than the accelerator, yet can bottleneck access to it: the “gatekeeper to the GPU.”

Parallel schedulers are not automatically an improvement. Coordinating concurrent access to GPU resources adds locks and other complexity. If the host already prepares work faster than the GPU consumes it, reducing scheduling time further may leave the next batch waiting for the previous GPU operation anyway. The immediate goal is to keep preparation ahead of execution.

A request arrives at the server, passes through tokenization, and reaches the scheduler. The scheduler asks how many resources it needs and which resources are available, then constructs a batch for the model runners. Accelerators benefit from processing multiple requests together. Frye compares this to a database scan: if loading the whole table is expensive, several queries can share that pass over the data.

Where does a request enter the repeating computation, and where do its output tokens leave? The diagram separates the scheduler–runner loop from the outward response path. Model runners return logits—the scores used to determine token probabilities—and generated tokens. Ready tokens pass to the detokenizer and back through the server while the loop continues generation. A response can therefore begin before all its GPU work is finished.

Tokenizer and detokenizer consistency comes from using the same mapping system and underlying libraries, generally Hugging Face tokenization libraries in this discussion. They do not inherently need a direct conversation for each request. Frye describes separate components in SGLang and a shared manager process in vLLM; the important relationship is the shared mapping, rather than identical process layouts.

Parallel tokenization does not remove the cost of a long prompt. Time to first token ends at the first generated output token, so it includes the model’s processing of the input. Longer inputs require more work in the core loop. When an engine divides a long input into chunks, congestion can also impose waiting at multiple scheduling opportunities. More tokenizer workers cannot eliminate either source of delay.

SGLang’s multiple tokenizer processes and single detokenizer reflect different rates of work. Prefill can consume input tokens much faster than decode produces output tokens. Detokenization still affects when users receive text, but it does not feed the expensive model computation. In Frye’s experience, prefill and especially decode usually dominate latency; smaller models with very large inputs can make tokenization a more significant bottleneck.

Multiprocessing gives server IO, tokenization, detokenization and model runners independent control flows. The model runner offloads work to the GPU, much as network software offloads work to a network interface. Separate event loops make it easier to tune each component’s latency, and the communication volume does not force all components into one thread.

For the Python implementations discussed, separate processes also avoid contention on the global interpreter lock. Frye acknowledges work on removing that restriction, with compatibility and performance still relevant concerns. Python remains useful because its model ecosystem and deep C/C++ interoperability make accelerator code accessible. Much of the expensive computation already runs in compiled, carefully optimized GPU kernels; rewriting the Python model worker does not automatically accelerate those kernels.

The scheduler is a more focused candidate for a faster host language. As GPU operations get faster, the host has less time to prepare the next batch. Scheduling manipulates resources without needing to reimplement the model ecosystem, so Frye expects it to be a useful target for a Rust rewrite. This is a proposed direction for engine development.

An engine generally consumes GPU kernels from backend libraries instead of implementing every operation itself. Its per-model forward-pass code selects and connects those operations. Reference implementations from Transformers provide a starting point, and the engines described offer paths to use them with automatic optimization. Commonly deployed architectures often have specially written execution code. Shared kernel libraries help explain why engine differentiation often comes from organizing work and getting out of the GPU’s way.

Batch construction is one of those distinctive engine responsibilities. A scheduler may separate prefill and decode, mix them in a batch, or switch between pure and mixed batches. Those choices determine how new prompts compete with ongoing generation. A queue is useful machinery, but a continually growing queue is an operating problem: the deployment is accumulating work faster than it clears it.

Multimodal preprocessing adds another place to spend time. A video may need FFmpeg to extract raw frames before those frames become model tensors. An image requires more preparation than an already-supported text tokenizer mapping Unicode input. Returning to the placard application, its visible simplicity does not mean its input path is as cheap as a text-only request.

Within the forward pass, the backend responsibilities become easier to distinguish:

• Cross-token computation. Attention combines information across the sequence. Backend choices discussed include FlashAttention, FlashInfer, CUTLASS, Triton implementations and kernels separated from TensorRT-LLM.
• Per-token computation. The MLP performs large matrix multiplications, often called GEMMs. A dense MLP uses the same network for every token; a sparse mixture of experts routes each token to selected smaller networks. DeepGEMM, Marlin and bitsandbytes appear among the backend examples.
• Expert routing and communication. Mixture-of-experts execution must send tokens to the right experts and combine results. DeepEP is an example of a separate communication library. Combining routing and matrix multiplication into a larger kernel can avoid penalties introduced by separating those stages.

Backend selection is configurable, although the configuration interface changes as engines develop. Frye describes SGLang as exposing choices at startup with smart defaults, and vLLM as emphasizing defaults with further changes available through environment variables. These are ways to choose implementations underneath the model, rather than changes to the application’s request format.

Under heavy concurrency, the scheduler must also allocate KV-cache capacity. Frye describes favoring requests that already have cached work, avoiding recomputation. Eviction or offloading can preserve availability, but he treats cache thrashing and spilling as symptoms of a poorly provisioned deployment. The eviction path depends on the engine; the operational analogy is swapping on a laptop—survival with painfully worse performance.

Cold starts are a separate deployment concern. The brief answer names memory snapshots, GPU-memory snapshots and a custom filesystem as pieces used to reduce startup time. Their implementation is not developed here. They address bringing a replica online, while scheduling and kernel optimization address its work once running.

KV caching trades memory for repeated computation. Retaining previously computed attention information avoids recalculating that information at every generation step. The retained state grows with the sequence, and GPU memory limits how much can remain useful at once. Moving it elsewhere is not automatically a win: fetching it back can cost more than recomputing it on a fast GPU.

Prefix reuse extends the same idea across requests. The commandments example shares the beginning “Thou shalt not”; computation for that common prefix can be reused until the token sequences diverge. After a differing token, matching later words do not restore a shared prefix. This produces a tree-shaped structure and makes a long common agent system prompt—potentially tens of thousands of tokens—valuable to cache across users.

Pages and radix structures organize cached state, and attention kernels can consume these layouts. That support takes over some work that previously belonged to the engine. The engine still manages cache capacity and layout: how much memory to allocate and which computed state can remain available. Faster attention kernels do not remove the capacity problem.

CUDA graphs address a different repetition: CPU work to launch GPU kernels. A model forward pass contains dependent operations over device memory. Graph capture records those operations and dependencies as a directed acyclic graph. Instead of repeatedly preparing every kernel launch on the CPU, the worker selects and launches the captured graph.

What changes when those launches are captured? The comparison shows that the GPU’s dependent operations remain; repeated host decisions shrink to a graph launch. This helps keep GPU work moving, although the CPU still has to schedule requests and return responses. Accidental host–device synchronization can still interrupt progress.

Ordinary decode repeatedly reads model parameters to produce the next token. For a mixture of experts, this means the active parameters. Producing only a little output per pass makes memory bandwidth a major constraint: the engine repeatedly moves a large amount of data for each increment of the response.

Speculative decoding uses another model, the speculator, to propose several upcoming tokens. The target model checks those candidates together. With the appropriate rejection-sampling procedure, the check preserves the target model’s sampling behavior, subject to numerical differences; it does not promise identical text from every separate stochastic run. Frye describes this as turning decodes into “tiny pre-fills”: the target performs useful work over several positions in one pass.

A better speculator gets more consecutive guesses accepted, so each expensive target pass advances generation further. This creates a direct route from training a better auxiliary model to faster serving. Frye contrasts it with optimizations won a few percentage points at a time and gives two-, four- and eightfold improvements as possible scales of gain. Actual speedup depends on accepted draft length and the cost of proposing and checking tokens; these examples are not a benchmark guarantee for an arbitrary deployment.

Splitting model work across accelerators introduces another responsibility. Kernels need to support the chosen parallelism, while the engine’s per-model execution code performs the surrounding communication. Moving and coordinating data is where many problems arise; adding accelerators does not remove the need to organize that work.

A fast engine also has to execute the model correctly. Distinguish application failures, such as an unwanted model response, from an engine optimization that changes model behavior. Tokenization and chat-template mistakes can alter what the model actually receives. Separately, performance may degrade only after a replica has run for a long time, or differ across machines in a heterogeneous cloud deployment.

The debugging advice connects each kind of problem to useful evidence:

• Model quality. Run evaluations against the actual deployment, retaining results and traces. Connect production user feedback to the requests that produced it.
• Tokenizer behavior. Log token IDs. They expose the representation the engine actually processes and generates.
• Performance. Retain enough metrics to compare behavior over time and across replicas. An unexpected correlation can make a difficult regression much easier to locate.

The museum application returns as a load example. Frye describes a setup with three replicas and an endpoint dashboard showing a traffic spike. Time to first token rises as requests wait to enter prefill. Intertoken latency rises too: ongoing decodes first slow a little, then much more as work contends for the GPU. The observable change is both a later start to each placard and a slower continuation once text begins.

Additional replicas relieve that congestion. Increased load is detected, new replicas come online, and they take some of the incoming requests. Queued work falls back to baseline. The causal sequence matters: more serving capacity spreads the demand and reduces contention, allowing requests to enter computation sooner.

For deeper diagnosis, Nsight Systems and Torch Profiler can expose activity across processes and GPU operations together. Frye gives a case where one GPU ran everything more slowly than its peers and the cause was NUMA awareness. A whole-engine trace helps distinguish a resource-placement problem from a slow model operation instead of treating every latency increase as the same failure.

miniSGLang and nano vLLM offer smaller implementations of the architecture to study. Their size makes the request loop, scheduling and execution code easier to hold in mind—and easier to discuss with a coding agent that can fit the implementation into its context. Follow one request through the code: where does it wait, what computed work survives, and what launches the next GPU operation?

The closing recommendations include a vLLM walkthrough by Aleksa Gordic, notebook exploration, and DeepWiki’s repository explanations and question-answering interface. Modal endpoints are presented as a way to begin with a reasonably optimized deployment baseline. These starting points serve different purposes: compact code explains the machinery, while evaluations, dashboards and traces reveal what limits the actual deployment.