Large clusters for small models — Daniel Svonava, Superlinked

A model that fits on one GPU removes a major serving problem: distributing its weights and computation across several devices. Daniel Svonava of Superlinked defines small models operationally—models that fit on NVIDIA hardware two or three generations old. That makes the hardware easier to obtain and more affordable, while creating room for lower latency and higher throughput. 1:19

The quality claim has a specific scope. A small model can reach or exceed frontier performance for a particular task. Svonava reads the Artificial Analysis Intelligence Index over time as a convergence: the frontier shows diminishing returns while smaller open models continue to improve. That motivates testing a smaller replacement for an existing workflow; aggregate benchmark proximity alone does not establish that the replacement will handle every task in that workflow.

The move to small models begins by splitting the workload into tasks. For each task, choose an open model, run evaluations, and adapt it if necessary. A single 27-billion-parameter model is not automatically the right destination for every prompt previously sent to a general-purpose endpoint. 3:49

The contract-review example makes the infrastructure change concrete. The agent uses nine different models. What previously looked like many requests to one API becomes traffic for a fleet, with different architectures and serving requirements. Add several agents across a company and the operational question grows: how can infrastructure support all these models without making each one a separate deployment project?

Specialization can come from training data as much as model size. Vietnamese receipt OCR is Svonava’s example: a project built around those receipts may have gathered the data most relevant to recognizing their contents. The practical reason to evaluate that model is its exposure to the task. His prediction that it will outperform alternatives remains a task-specific hypothesis to test.

The available tasks span OCR, document question answering, image labeling, SQL generation, and code review. Svonava’s diagnosis is that the models already exist; serving them is the bottleneck. He criticizes AWS Bedrock as a substitute for owning this fleet, citing restricted model selection and fine-tuned artifacts that remain tied to its infrastructure. Those are his descriptions of the managed-service constraints, rather than a demonstrated comparison across its offerings.

Open serving software supplies building blocks, but a wide fleet exposes three different kinds of work:

• Tuning each model and GPU combination: vLLM and SGLang do not arrive optimized for every model, hardware choice, and traffic pattern. Parameter sweeps and workload-specific tuning can turn adoption into an open-ended research project.
• Keeping GPUs busy: A router designed to assign requests from above must track worker state, including KV caches and local queues. With many short requests, that information becomes stale quickly enough to produce poorly sized batches.
• Deploying adaptations: An AI engineer’s ten LoRAs or overnight fine-tune creates another handoff to infrastructure. Repeating that conversation for each new artifact slows the rate at which model improvements reach production.

The routing failure follows from where the decision happens. A large-model router chooses a worker or group of workers using its view of their state. A small request may finish before that view catches up. The router still has to distribute enough work to fill each GPU’s next batch, and separate local queues make a mistaken assignment difficult to correct. In Superlinked’s experiments with vLLM and SGLang routers on this traffic, GPU utilization was difficult to push beyond 20–30% under constant load; that observation concerns their tested small-model workload. 7:48

The organizational goal is equally concrete: infrastructure engineers should operate the serving system while AI engineers improve models, without either group blocking the other’s daily work. Superlinked arrived at its cluster design through search, document-processing, and agent deployments in environments with differing hardware availability. Small models helped because obtaining L4s or a modest GPU quota was easier than finding capacity for a much larger deployment.

Superlinked’s design moves the assignment decision toward the workers. A gateway inspects part of a request, attaches metadata, and inserts it into a shared queue and side channels. Workers pull work when they can use it. The stack is described as Apache 2.0 open source from the control plane through the code running on the GPU. 11:17

The request format also matters when inference is fast. The system uses MessagePack, a binary format, throughout instead of carrying binary inputs as Base64-encoded JSON. Images and videos can enter through the same API as the rest of the request, so the client does not have to send a storage location and separately give the cluster access to its cloud storage.

Consider the larger multimodal request described in the talk. The client sends all its bytes through the gateway. For a request over roughly a megabyte, the gateway can separate heavy payload pieces and place them in backend cloud storage while the request waits in the queue. The observable change is internal: the shared queue no longer carries all the bulky data, while the client keeps one upload interface. This protects the scheduling path from becoming a bulk-data path.

What stays on the scheduling path, and where does model execution begin? The diagram separates the gateway’s light annotation work from the workers’ pull decisions and local runtime connection. Backend storage handles heavy payloads alongside that path; it is not a second upload destination the client must manage.

Both gateway and worker expose REST interfaces; the worker connects to a runtime over a local socket. The gateway avoids parsing the whole request because doing too much work at the single entry point would recreate the bottleneck. NATS JetStream supplies the shared queue, with Svonava citing capacity of a million requests per second. Avoiding repeated serialization and deserialization across components keeps more of the request’s time available for useful work.

A shared queue changes who must predict the right batch. The router no longer tries to fill every worker’s local queue correctly in advance. Each worker picks up requests and estimates the cost of the batch it is building. Work remains shared until a worker takes it, giving workers more control over keeping themselves busy. 14:47

That does not make batch sizing easy. A worker can pull too much and discover that some requests should go back. Returning them through the centralized queue would add a network hop—milliseconds that matter for short inference calls. On machines with multiple GPUs, Superlinked adds a machine-local queuing element so colocated worker processes can negotiate work back and forth locally. This optimization applies to workers sharing a machine, rather than performing that negotiation across the network.

The reported result is double the cluster throughput after centralizing the queue. Svonava stresses that this is a substantial change, rather than a five-percent tuning win. The claim is a result from Superlinked’s experiments, not a universal multiplier for every cluster topology or traffic mix.

The worker layer has to accommodate different model architectures without accumulating conflicting Python requirements. Three runtimes serve different purposes:

• PyTorch: Superlinked writes and optimizes execution code for models such as encoder-only architectures, using an autoresearch loop to improve performance.
• Candle: A statically linked worker binary reduces the deployment payload. Svonava contrasts a roughly 12-gigabyte PyTorch Docker image with a Candle package around ten percent of that size. Moving less data helps when bringing up cold workers across machines, but their Candle implementation still performs well below PyTorch.
• SGLang: Optimally tuned SGLang provides a performance baseline that the surrounding system should at least match.

The Rust sidecar and local socket put shared serving behavior outside any one runtime. Superlinked reports improving on bare SGLang through additional batching logic. The same logic could potentially be implemented inside SGLang with custom plugins, but those plugins would tie the work to that runtime. With on the order of 50 parameterized model adapters, keeping the common batching layer separate avoids making one execution engine the organizing principle for the entire fleet. 17:46

The useful benchmarking target is the knee. As offered traffic rises, completed throughput initially rises with it. Eventually throughput flattens while latency climbs: the server has reached saturation, and additional demand mostly creates waiting. The knee identifies the region of maximum throughput before latency suffers. 18:48

The performance examples presented here were measured on an RTX PRO 6000. For embedding models with up to hundreds of millions of parameters, Svonava describes hundreds of thousands of input tokens encoded per second, reaching roughly half a million tokens per second on one GPU, with call latency in the low tens of milliseconds. He contrasts that with hundreds of milliseconds and orders-of-magnitude higher costs for managed embedding endpoints. These are workload-dependent comparisons; the talk does not establish a matched model-quality and total-cost comparison for an arbitrary deployment.

For a search system, the change is straightforward: input text enters a local GPU and embedding vectors come back at high throughput. That is why embeddings are Svonava’s “no-brainer” starting point for self-hosting small models. The serving architecture supports the economics by making it easier to keep the GPU supplied with useful batches instead of paying for hardware that sits partly idle.

Other candidates include named entity recognition, multi-vector search, and task-specific generation of text or structured output. Synthetic data and annotations for fine-tuning or evaluations are particularly attractive in Svonava’s account because the task is controlled and the team can inspect output quality. Encoding throughput counts input tokens; generative throughput counts output tokens, so the two rates describe different work. Scaling across more GPUs also depends on the surrounding infrastructure continuing to feed them efficiently.

A dedicated, permanently preloaded worker pool per model carries over assumptions from serving enormous models that take tens of minutes to load. Small models allow a different approach: pack multiple models on the same GPU, pin some in memory, and combine that with lazy loading and eviction as memory pressure changes. The decision becomes which models should stay resident and which should load on demand. The ending introduces this combination without developing a specific placement or eviction algorithm. 22:52

Autoresearch addresses the earlier tuning problem. Superlinked builds measurement tooling that feeds loops for adding model support and improving execution performance. When support for a model ships, it includes an end-to-end cluster configuration with the tuning already done. Users should not have to begin by running another parameter sweep.

The research setup includes a meta loop that builds the harness, the loop that runs experiments, and a dashboard for understanding the results. One output was a LoRA that cost 80 cents to train and improved retrieval quality on German legal text by 18% as a proof of concept. The retrieval metric and whether the gain is relative or absolute are unspecified, so the result demonstrates a promising inexpensive adaptation rather than a transferable accuracy guarantee.

The closing invitation is to try RCluster, the implementation described throughout the talk. The closing slide at 24:23 provides the GitHub-repository QR code. “Happy self-hosting” rests on the work developed along the way: evaluate a model for each task, keep requests shared until workers can batch them, accommodate several runtimes, and distribute tuned configurations with model support. Small models reduce the size of each inference problem; the cluster makes a large collection of those problems manageable.