The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

What is missing between understanding how a model computes and making it run efficiently in production? That is the problem behind Filip Makraduli’s work on small-model inference. He introduces it with a second, visual question: the patterned background of his slides represents something from machine learning. He offers a reward for identifying it, leaving the answer until the end.

The infrastructure question began with a Substack article. Makraduli had explained FlashAttention, model execution, and the difference between memory-bound and compute-bound work. Feedback exposed something the explanation had overlooked: inference as an operational system. His experience with vLLM, training, fine-tuning, applied ML, and academic research had not covered the whole path to production. GPU scheduling, routing, and automation are separate engineering problems from understanding the model.

To learn that part, he joined Superlinked’s infrastructure team and helped build the open-source Superlinked Inference Engine, or SIE. The soft launch presented here targets small models for AI search and document processing. Makraduli reports testing with Chroma, Qdrant, Weaviate, and LanceDB. The project connects model execution to the machinery needed to operate it.

The first question is why search and document-processing inference matters to agents. The next is what serving a model leaves unfinished. Those lead to the two halves of SIE’s design: model support and infrastructure, which Makraduli calls the yin and yang of inference.

The agent problem starts with context. Chroma’s Context Rot research shows that longer inputs can degrade performance under tested conditions; the effect varies across models and tasks. Small models provide a way to manage what reaches the agent: preprocess documents, select useful information, or expose a focused operation as a tool call. This also answers the objection that Claude Code and grep can already search files. Preprocessing can improve the files and information those tools search.

Several approaches fit this pattern:

• Structured knowledge: Makraduli points to Andrej Karpathy’s graph-based knowledge-base work. Named entity recognition can supply entities for ontology and knowledge-graph construction, or help organize a filesystem.
• Input filtering: He cites a Chroma model that preprocesses and filters input to manage context, alongside community tools that reduce token volume.
• Model-backed tools: In a production e-commerce use case, Makraduli reports using SIE for taxonomy classification. Small models become tools that retrieve and process data during the workflow, rather than only preparing it beforehand.

The common mechanism is to give the agent a more focused representation of the data it needs.

Adding compute is an incomplete answer when each model uses only a small part of a GPU. Makraduli describes Stella embedding models, rerankers, and the GLiNER named entity recognition model as occupying only a few gigabytes each, without specifying variants, precision, or workload memory. Giving every model its own GPU can leave both memory capacity and compute underused.

SIE instead supports swapping models on a shared GPU. A workflow can switch from one model-backed tool to another without reserving a separate GPU for each. Its least recently used eviction policy chooses which resident model to remove when capacity is needed: a model that has gone unused longer is evicted before a more recently used one. Makraduli reports higher utilization, lower costs, and quick switching, but supplies no measured utilization, cost, or switching-latency comparison.

Sharing a GPU still leaves the operational work around the server. TEI, vLLM, or an API wrapper can provide a starting point for inference, while the surrounding system needs routing, autoscaling, queues, GPU provisioning, and monitoring through Prometheus metrics and Grafana. Makraduli describes a gap in the open-source options available for connecting these pieces at the intended scale. SIE’s response is to supply the production infrastructure together with model serving.

Infrastructure is useful only if it supports the models a workload needs. Makraduli therefore starts the first half of the design with model breadth. He cites millions of models on Hugging Face in a March snapshot and estimates that the count could be approaching three million by the talk. That is an ecosystem-size argument: the available choices keep expanding, and users want access to them.

Makraduli cites MTEB and other benchmarks as evidence that open models can outperform managed services on narrow tasks. He also points to small Gemma models achieving higher Elo scores than much larger models. The presentation does not identify the tasks, model versions, baselines, or scores for these comparisons. Their role here is to motivate support for specialized models: choosing a smaller open model can be about suitability and quality as well as deployment control.

Supporting hundreds of models is not simply a matter of loading different weights. BERT, Qwen, and ModernBERT differ in their architectures, attention implementations, and positional representations. SIE’s approach is to reimplement forward passes with the adaptations each architecture needs: attention handling, padding, variable-length execution, and query/key/value projection fusion where appropriate.

The output contract also changes what the runtime must preserve. ColBERT is a late-interaction model that produces multiple vectors, retaining token-level representations for subsequent matching. Cross-encoders and rerankers instead return scores rather than embedding vectors. A common serving layer therefore has to accommodate different result structures, not just different tensor sizes.

Makraduli’s comparison works through these differences across BERT, Qwen, and ColBERT. One qualification matters: he attributes a Qwen projection-fusion limitation to grouped-query attention, but grouped-query attention does not inherently prevent combined QKV projections. The Qwen2 implementation in vLLM v0.8.5 uses QKVParallelLinear. The relevant engineering task is adapting fusion to the model’s head layout and runtime, rather than assuming one fusion strategy works unchanged everywhere.

These differences require a consistent implementation process. SIE uses both agents and humans to implement the adapted forward passes, turning architectural knowledge into efficient model execution. Broad model support depends on preserving each model’s semantics while changing how its computation runs.

Variable-length FlashAttention addresses a concrete batching problem. Put a short request and a long request into a rectangular batch, and the short one may be padded to match the long one. Computation then includes empty positions introduced for the batch layout. Token-based batching needs an execution path that respects the actual sequence lengths, so padding does not consume unnecessary attention work.

For a small illustrative batch, take request A with token IDs [11, 12] and request B with [21, 22, 23, 24]. A rectangular representation gives A two padding slots. A packed representation retains [11, 12, 21, 22, 23, 24] and sequence boundaries [0, 2, 6]. Those boundaries are essential: removing padding must not allow one request’s tokens to attend to another request. The token contents stay the same; their storage and execution layout change.

The second half of the system is the cluster. Makraduli introduces a detailed architecture diagram but concentrates on the path through it rather than explaining every component. At this soft launch, the API is organized around three primitives: encode, score, and extract. Current SIE documentation includes additional generation capabilities; the three-operation interface is the one presented in the recording.

Requests pass through a router and a load-dependent queuing mechanism, which distribute work across pools of GPUs and other resources. Those pools can include spot instances and larger GPUs. KEDA autoscaling, driven by Prometheus metrics, connects observed demand to capacity changes. Hardware provisioning and model switching work together to keep available resources useful rather than leaving GPUs idle.

The integration is the product: users receive model support and cluster infrastructure together, instead of having to connect an inference engine to a separately assembled production stack. In the deployment workflow Makraduli describes, the operator selects models through configuration and applies the infrastructure:

terraform apply

Published Helm charts and Docker images provide additional deployment artifacts. The command is the final application step for the configured infrastructure, not a replacement for selecting and configuring the deployment.

The release includes both sides of SIE: the adapted model execution code, including attention and forward-pass work, and the cluster machinery for provisioning and operating GPUs. This closes the gap that started with the FlashAttention article. Understanding how computation becomes fast inside a model now connects to controlling where that computation runs and how requests reach it.

With the infrastructure explanation complete, Makraduli returns to the background puzzle. His hints point toward embeddings, attention, and transformer foundations. An audience member tentatively recalls a visualization in which embeddings clustered in part of the space because of positional encoding. Makraduli accepts the positional-encoding connection as the answer.

The reveal identifies the background as a vector visualization of sinusoidal positional encodings. The final slide pairs a green-and-blue heatmap with the orange striped pattern seen in the background. Makraduli connects the pattern to the way transformers represent token positions during training. In the original Transformer formulation, these sinusoidal encodings are fixed functions used during training, distinct from learned positional embeddings. The visual puzzle ends on the same architectural detail that mattered in model serving: how a model represents position is part of what its implementation must preserve.