# The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

Filip Makraduli

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/qdh_x-uRs9g-small-model-infrastructure-nobody-built-so) · [Video](https://www.youtube.com/watch?v=qdh_x-uRs9g)

## The Small Model Infrastructure Nobody Built (So We Did)

Small models can make agent context more useful, but serving them efficiently requires architecture-aware execution, shared GPU capacity, and a production cluster that works with both.

Familiarity with embeddings, transformer attention, and basic GPU serving will help with the runtime and infrastructure details.

### What makes a model fast in production?

[0:16](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=16s)

What is missing between understanding how a model computes and making it run efficiently in production? That is the problem behind Filip Makraduli’s work on small-model inference. He introduces it with a second, visual question: the patterned background of his slides represents something from machine learning. He offers a reward for identifying it, leaving the answer until the end.

The infrastructure question began with a Substack article. Makraduli had explained FlashAttention, model execution, and the difference between memory-bound and compute-bound work. Feedback exposed something the explanation had overlooked: inference as an operational system. His experience with vLLM, training, fine-tuning, applied ML, and academic research had not covered the whole path to production. **GPU scheduling, routing, and automation are separate engineering problems from understanding the model.**

![Two boxes contrast training, evaluation and AI research with production inference, GPU scheduling, routing and deployment. Below: “Join the team building it.”](https://ai.engineer/api/talk-essay-media/e1cf6852042e7590492208398ec88d1e3f93981ef074b46e2f36ffe4280e5951)

Model expertise versus the infrastructure blind spot.

To learn that part, he joined Superlinked’s infrastructure team and helped build the open-source [Superlinked Inference Engine](https://github.com/superlinked/sie), or SIE. The soft launch presented here targets small models for AI search and document processing. Makraduli reports testing with Chroma, Qdrant, Weaviate, and LanceDB. The project connects model execution to the machinery needed to operate it.

### Small models before and inside the agent

[3:42](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=222s)

The first question is why search and document-processing inference matters to agents. The next is what serving a model leaves unfinished. Those lead to the two halves of SIE’s design: model support and infrastructure, which Makraduli calls the yin and yang of inference.

The agent problem starts with context. Chroma’s [Context Rot](https://www.trychroma.com/research/context-rot) research shows that longer inputs can degrade performance under tested conditions; the effect varies across models and tasks. Small models provide a way to manage what reaches the agent: preprocess documents, select useful information, or expose a focused operation as a tool call. This also answers the objection that Claude Code and `grep` can already search files. Preprocessing can improve the files and information those tools search.

![Slide titled “Why it matters for your agents” shows declining performance curves alongside boxes for context management, tool calling and “Why not just Opus 4.6?”](https://ai.engineer/api/talk-essay-media/b60404ebd4d22a97d6970ac87d95d288dea58a2c3d2891d62c7640e1c08b3a97)

Context rot motivates context management and small-model tool calling.

Several approaches fit this pattern:

- **Structured knowledge:** Makraduli points to Andrej Karpathy’s graph-based knowledge-base work. Named entity recognition can supply entities for ontology and knowledge-graph construction, or help organize a filesystem.
- **Input filtering:** He cites a Chroma model that preprocesses and filters input to manage context, alongside community tools that reduce token volume.
- **Model-backed tools:** In a production e-commerce use case, Makraduli reports using SIE for taxonomy classification. Small models become tools that retrieve and process data during the workflow, rather than only preparing it beforehand.

The common mechanism is to give the agent a more focused representation of the data it needs.

### More GPUs do not solve idle capacity

[7:03](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=423s)

Adding compute is an incomplete answer when each model uses only a small part of a GPU. Makraduli describes **Stella** embedding models, rerankers, and the **GLiNER** named entity recognition model as occupying only a few gigabytes each, without specifying variants, precision, or workload memory. Giving every model its own GPU can leave both memory capacity and compute underused.

SIE instead supports swapping models on a shared GPU. A workflow can switch from one model-backed tool to another without reserving a separate GPU for each. Its **least recently used eviction policy** chooses which resident model to remove when capacity is needed: a model that has gone unused longer is evicted before a more recently used one. Makraduli reports higher utilization, lower costs, and quick switching, but supplies no measured utilization, cost, or switching-latency comparison.

Sharing a GPU still leaves the operational work around the server. TEI, vLLM, or an API wrapper can provide a starting point for inference, while the surrounding system needs routing, autoscaling, queues, GPU provisioning, and monitoring through Prometheus metrics and Grafana. Makraduli describes a gap in the open-source options available for connecting these pieces at the intended scale. SIE’s response is to supply the production infrastructure together with model serving.

### The yin: models worth serving

[9:24](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=564s)

Infrastructure is useful only if it supports the models a workload needs. Makraduli therefore starts the first half of the design with model breadth. He cites millions of models on Hugging Face in a March snapshot and estimates that the count could be approaching three million by the talk. That is an ecosystem-size argument: the available choices keep expanding, and users want access to them.

Makraduli cites MTEB and other benchmarks as evidence that open models can outperform managed services on narrow tasks. He also points to small **Gemma** models achieving higher Elo scores than much larger models. The presentation does not identify the tasks, model versions, baselines, or scores for these comparisons. Their role here is to motivate support for specialized models: choosing a smaller open model can be about suitability and quality as well as deployment control.

### A shared API still needs different forward passes

[11:10](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=670s)

Supporting hundreds of models is not simply a matter of loading different weights. **BERT**, **Qwen**, and **ModernBERT** differ in their architectures, attention implementations, and positional representations. SIE’s approach is to reimplement forward passes with the adaptations each architecture needs: attention handling, padding, variable-length execution, and query/key/value projection fusion where appropriate.

The output contract also changes what the runtime must preserve. **ColBERT** is a late-interaction model that produces multiple vectors, retaining token-level representations for subsequent matching. Cross-encoders and rerankers instead return scores rather than embedding vectors. A common serving layer therefore has to accommodate different result structures, not just different tensor sizes.

| Implementation concern | Difference the runtime must handle |
| --- | --- |
| Normalization | Model-specific normalization behavior |
| Q/K/V projections | Fusion choices and attention-head layout |
| Position | BERT absolute lookup; Qwen rotary embeddings |
| Attention | Architecture-specific execution and masking |
| Output | Embedding vectors, per-token vectors, or scores |

Makraduli’s comparison works through these differences across BERT, Qwen, and ColBERT. One qualification matters: he attributes a Qwen projection-fusion limitation to grouped-query attention, but grouped-query attention does not inherently prevent combined QKV projections. The [Qwen2 implementation in vLLM v0.8.5](https://github.com/vllm-project/vllm/blob/v0.8.5/vllm/model_executor/models/qwen2.py) uses `QKVParallelLinear`. The relevant engineering task is adapting fusion to the model’s head layout and runtime, rather than assuming one fusion strategy works unchanged everywhere.

![Comparison table covers normalization, Q/K/V, position, attention and output for BERT, Qwen2 and ColBERT, highlighting architectural differences and ColBERT’s per-token vectors.](https://ai.engineer/api/talk-essay-media/e18ce0208f7122b3e8578149edbbc633597dd291500a317eaefca652cd85fedb)

BERT, Qwen2 and ColBERT differ across five implementation steps.

These differences require a consistent implementation process. SIE uses both agents and humans to implement the adapted forward passes, turning architectural knowledge into efficient model execution. **Broad model support depends on preserving each model’s semantics while changing how its computation runs.**

### Stop spending attention work on padding

[13:48](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=828s)

Variable-length FlashAttention addresses a concrete batching problem. Put a short request and a long request into a rectangular batch, and the short one may be padded to match the long one. Computation then includes empty positions introduced for the batch layout. Token-based batching needs an execution path that respects the actual sequence lengths, so padding does not consume unnecessary attention work.

For a small illustrative batch, take request A with token IDs `[11, 12]` and request B with `[21, 22, 23, 24]`. A rectangular representation gives A two padding slots. A packed representation retains `[11, 12, 21, 22, 23, 24]` and sequence boundaries `[0, 2, 6]`. Those boundaries are essential: removing padding must not allow one request’s tokens to attend to another request. The token contents stay the same; their storage and execution layout change.

Preserve request tokens while removing padding
Packing removes padding while preserving each request’s tokens and separate sequence boundaries.
Constructed example: Request names, token IDs, lengths, PAD notation, and boundary offsets are teaching values. Packed storage and boundaries illustrate the variable-length mechanism; they are not a captured SIE execution.
Two requests — unchanged: A = [11, 12]; B = [21, 22, 23, 24]
Operation: Replace the padded rectangular representation with packed tokens and sequence boundaries.
- Request A tokens
  - Before: Padded batch: [11, 12]
  - After: Packed variable-length batch: [11, 12]
- Request B tokens
  - Before: Padded batch: [21, 22, 23, 24]
  - After: Packed variable-length batch: [21, 22, 23, 24]
- Token storage
  - Before: Padded batch: [[11, 12, PAD, PAD], [21, 22, 23, 24]]
  - After: Packed variable-length batch: [11, 12, 21, 22, 23, 24]
- Padding positions
  - Before: Padded batch: Request A: slots 2 and 3
  - After: Packed variable-length batch: Not present
- Packed sequence boundaries
  - Before: Padded batch: Not present
  - After: Packed variable-length batch: [0, 2, 6]
Packing removes padding while preserving each request’s tokens and separate sequence boundaries.

### The yang: a cluster that can operate the models

[14:23](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=863s)

The second half of the system is the cluster. Makraduli introduces a detailed architecture diagram but concentrates on the path through it rather than explaining every component. At this soft launch, the API is organized around three primitives: `encode`, `score`, and `extract`. Current SIE documentation includes additional generation capabilities; the three-operation interface is the one presented in the recording.

Requests pass through a router and a load-dependent queuing mechanism, which distribute work across pools of GPUs and other resources. Those pools can include spot instances and larger GPUs. **KEDA autoscaling**, driven by **Prometheus metrics**, connects observed demand to capacity changes. Hardware provisioning and model switching work together to keep available resources useful rather than leaving GPUs idle.

The integration is the product: users receive model support and cluster infrastructure together, instead of having to connect an inference engine to a separately assembled production stack. In the deployment workflow Makraduli describes, the operator selects models through configuration and applies the infrastructure:

```bash
terraform apply
```

Published Helm charts and Docker images provide additional deployment artifacts. The command is the final application step for the configured infrastructure, not a replacement for selecting and configuring the deployment.

The release includes both sides of SIE: the adapted model execution code, including attention and forward-pass work, and the cluster machinery for provisioning and operating GPUs. This closes the gap that started with the FlashAttention article. Understanding how computation becomes fast inside a model now connects to controlling where that computation runs and how requests reach it.

### The pattern behind the slides

[16:47](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1007s)

With the infrastructure explanation complete, Makraduli returns to the background puzzle. His hints point toward embeddings, attention, and transformer foundations. An audience member tentatively recalls a visualization in which embeddings clustered in part of the space because of positional encoding. Makraduli accepts the positional-encoding connection as the answer.

The reveal identifies the background as a vector visualization of **sinusoidal positional encodings**. The final slide pairs a green-and-blue heatmap with the orange striped pattern seen in the background. Makraduli connects the pattern to the way transformers represent token positions during training. In the original [Transformer formulation](https://arxiv.org/html/1706.03762v7), these sinusoidal encodings are fixed functions used during training, distinct from learned positional embeddings. The visual puzzle ends on the same architectural detail that mattered in model serving: how a model represents position is part of what its implementation must preserve.

![Projected slide shows a green-and-blue sinusoidal positional encoding heatmap above an orange striped pattern, with a QR code at right and the speaker at the lectern.](https://ai.engineer/api/talk-essay-media/470d1481f1eee98729153a52db7814e17d05ec09a5bd911517718c118eb40ad0)

The closing reveal pairs a sinusoidal positional encoding heatmap with the patterned background.

## Resources

- [Superlinked Inference Engine](https://github.com/superlinked/sie) — Source code and current setup instructions for shared-model inference and its production cluster.
- [What Actually Makes Embedding Model Inference Fast?](https://filipmakraduli.substack.com/p/what-actually-makes-embedding-model) — Makraduli's explanation of memory traffic, FlashAttention, and embedding inference bottlenecks.
- [Beyond Flash Attention](https://filipmakraduli.substack.com/p/beyond-flash-attention-the-inference) — A companion discussion of forward-pass costs, variable-length attention, and padding, scoped to dense embedding models on H100-class GPUs.
- [Context Rot](https://www.trychroma.com/research/context-rot) — Chroma's experiments on how input length, semantic ambiguity, and distractors affect model performance.
- [Gemma 3 Technical Report](https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf) — Architecture and evaluations for Gemma 3, including a dated preliminary Chatbot Arena comparison with larger models.
- [Qwen2 attention implementation in vLLM](https://github.com/vllm-project/vllm/blob/v0.8.5/vllm/model_executor/models/qwen2.py) — A version-pinned implementation showing combined QKV projections in Qwen2 attention.
- [Attention Is All You Need](https://arxiv.org/html/1706.03762v7) — The original Transformer architecture, including its sinusoidal positional encoding formulation.
- [State of Open Source on Hugging Face: Spring 2026](https://huggingface.co/blog/huggingface/state-of-os-hf-spring-2026) — A historical account of public model repositories, derivative models, and adoption across Hugging Face.

## Slides

### AI Engineer Europe

The recording opens with the AI Engineer Europe logo.

[0:00](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=0s) [AI Engineer Europe logo in white on black.](https://assets.ai.engineer/sites/aiecode2025/sha256/099222ed3a40508baa117281b8765de70d74bc7dbd514791aa20e7c7d38d1c4f.webp)

AI Engineer
EUROPE

### Presenting sponsor

Google DeepMind is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=5s) [Presenting sponsor Google DeepMind on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/94b8a83e75af54cd32eb742cc8be01b92271ca2d2903fa413c8fdc796bc24862.webp)

PRESENTING SPONSOR

Google DeepMind

### Platinum sponsors

[0:10](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=10s) [Braintrust, WorkOS and OpenAI logos under the heading Platinum Sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/04946fdb4b4052b0a41167b1d4ca474e205397256d6e70ecf209da364042601e.webp)

PLATINUM SPONSORS

Braintrust
WorkOS
OpenAI

### What Actually Makes Embedding Model Inference Fast?

After writing about FlashAttention and compute- and memory-bound processes, Makraduli learned he had overlooked inference as a key part of real-world model speed.

[0:52](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=52s) [An article titled What Actually Makes Embedding Model Inference Fast? beside the words Not quite; the speaker appears below the slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/4b7abf1e7739a6e4159077321e147a1f31274be5c7fe9faaf12477e11c7941d3.webp)

A WRITER'S DIARY ON AI

What Actually Makes Embedding Model Inference Fast?

From Flash Attention to Quantization, where is the inference bottleneck? Is it the architecture, the maths, or will writing everything in Rust solve all my problems? (Hint: It's Not Rust)

FILIP MAKRADULI
JAN 22, 2026

Not quite.

AIE
Google DeepMind
AI Engineer EUROPE 2026

[1:22](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=82s) [An article about embedding model inference appears beside the words “Not quite.”](https://assets.ai.engineer/sites/aiecode2025/sha256/e75436b8dc454f91eb1609fb53bc3325da081ee2a42bae18538c387999e51215.webp)

A WRITER'S DIARY ON AI

What Actually Makes Embedding Model Inference Fast?

From Flash Attention to Quantization, where is the inference bottleneck? Is it the architecture, the maths, or will writing everything in Rust solve all my problems? (Hint: It's Not Rust)

FILIP MAKRADULI
JAN 22, 2026

Not quite.

AIE
Braintrust
WorkOS
OpenAI
AI Engineer EUROPE 2026

### The Overlooked Aspect: Inference

Makraduli says others pointed out that he had overlooked inference as a key part of what makes models fast in the real world.

[1:37](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=97s) [The slide says the overlooked aspect was inference.](https://assets.ai.engineer/sites/aiecode2025/sha256/830b905f91c538b0ce31f963465114c4a1cbfb06d4217f94e2db6229662a69a1.webp)

The aspect I overlooked was...

Inference

AIE
AI Engineer EUROPE

### I Had to Learn

After overlooking inference's role in making models fast in the real world, Makraduli realized he needed to learn it. A cat reading a newspaper illustrates the moment.

[1:40](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=100s) [The words “I realised ... I had to learn” appear above a cat reading a newspaper.](https://assets.ai.engineer/sites/aiecode2025/sha256/51db506767a19b20664d93452ab251239b6fb8ed4335e573cb88d481d1ff00f3.webp)

I realised ... I had to learn

AIE
AI Engineer
EUROPE

### Learning From First Principles

Makraduli knew model training and evaluation, but production inference—including GPU scheduling and routing—was a blind spot. He chose to learn by joining a team building it.

[2:00](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=120s) [A comparison contrasts experience with models against overlooked production infrastructure, followed by a suggestion to join the team building it.](https://assets.ai.engineer/sites/aiecode2025/sha256/2420ab17776980bb4d03fc02c65c797b8dfcf9e93cf5609d345c60ce7dd7d479.webp)

LEARNING FROM FIRST PRINCIPLES

I knew models well
Training & fine-tuning
Evaluation & benchmarking
Applied AI research
vLLM, Flash Attention

years of experience

I had overlooked infrastructure
How models run in production
GPU utilization & scheduling
Routing & autoscaling
Deployment & operations

blind spot

Best way to learn?
Join the team building it.

AIE
AI Engineer EUROPE

### Joining Superlinked

To learn production infrastructure, Filip joined Superlinked and helped build the open-source Superlinked Inference Engine, introduced here in a soft launch.

[2:41](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=161s) [The speaker introduces Superlinked and its inference engine, with a screenshot of tool integrations.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b5c7a6e444cd2008d7cc017e191fab10a8f7953a4f3722bcbc071fd69a6cb4c.webp)

So I joined the world class infrastructure team at:

Superlinked

GitHub: github.com/superlinked/sie

VC - backed AI infrastructure company building the model stack for AI search and document processing

Works with your favorite tools
BROWSE INTEGRATIONS →

Chroma — DOCS →
“Chroma makes context engineering simple. SIE adds instruction-following rerankers and relationship extractors for even more precise retrieval.”
Jeff Huber
CEO & Founder

LanceDB — DOCS →
“LanceDB centralizes multi-modal training datasets and with SIE you can self-host inference for all the required data transformations.”
Chang She
CEO & Co-founder

Qdrant — DOCS →
“Modern search systems compose the best indexing, scoring, filtering and ranking models. With SIE you can self-host them all in one cluster.”
Andre Zayarni
CEO & Co-founder

Weaviate — DOCS →
“Weaviate's Query Agent unlocks natural language search and with SIE you can pre-process your query and data for better latency.”
Bob Van Luijt
CEO & Co-founder

AIE
AI Engineer EUROPE
2026

### Key points

The talk connects inference for AI search and document processing to agent workflows, then examines misconceptions and the yin and yang of model inference.

[3:41](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=221s) [Three key points: relevance to agents, misconceptions about small-model inference, and the yin and yang of model inference.](https://assets.ai.engineer/sites/aiecode2025/sha256/41809825f8b6fe082ce967ae42fe2d78e50f78e1dff0e1da0887c42a1cfcc42d.webp)

KEY POINTS

1. Why this matters for your agents, even if you're not doing AI search
2. What small-model inference is not about
3. The yin and yang of model inference

AIE
Engineering the future of AI
AI Engineer EUROPE 2026

### Why this matters for agents and workflow

Makraduli introduces context rot as a reason this matters for agentic workflows.

[4:33](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=273s) [Section slide asks why this matters for agents and workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/f73c9e7845d19f1c52f2250891077eac00eabb6ff86279565b32714f225202eb.webp)

THE ANTITHESIS

Why this matters for your
Agents and workflow ?

AIE
AI Engineer
EUROPE

### Context rot is real

Chroma’s repeated-words chart shows scores falling as input length grows across four models. Makraduli suggests using small models to preprocess data and manage what enters an agent’s context.

[4:38](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=278s) [A chart shows performance declining as input length increases, alongside points about context management and tool calling.](https://assets.ai.engineer/sites/aiecode2025/sha256/49ece8b190ba6dec47ea77c1d81dd2d35b32d777945d3e31f4cd0f98f38047da.webp)

WHY IT MATTERS FOR YOUR AGENTS

Context rot is real

Performance degrades as context fills up.
More tokens, worse answers.

Repeated Words - Performance by Input Length (Tokens)
Y-axis: Average Normalized Levenshtein Score
Y-axis ticks: 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
X-axis: Input Length (Tokens)
X-axis ticks: 10², 10³, 10⁴
Legend: Claude Sonnet 4; GPT-4.1; Qwen3-32B; Gemini 2.5 Flash

Source: Chroma Research

Context management
Manages what goes INTO context,

Tool calling
Embeddings, reranking, extraction

“Why not just Opus 4.6?”
Works until context rot kills quality

AI Engineer EUROPE

### Context management requires preprocessing

The slide shows accuracy dipping when relevant information sits in the middle of context, and points to knowledge bases and Chroma’s self-editing search agent as ways to manage it.

[5:42](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=342s) [A bar chart shows lower accuracy when relevant information is in the middle of context, beside examples of knowledge-base and context-management work.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0da56eeac32fb9929ee120c5bac83f258d440e43e7bdf1856055254b8ff08cd.webp)

WHY IT MATTERS FOR YOUR AGENTS

Context management requires a level of pre-processing

Accuracy drop when relevant info lands in the middle of context,

Y-axis: Accuracy
Position in context: Start, Middle, End
High; Low; High

Liu et al. 2023 · 18 models tested by Chroma · 194K LLM calls

Andrej Karpathy
@karpathy
LLM Knowledge Bases

Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into

Chroma Context-1: Training a Self-Editing Search Agent

r/ClaudeCode
u/captainkink07 · 18h · github.com
71.5x token reduction by compiling your raw folder into a knowledge graph instead of reading files. Built from Karpathy's workflow
Showcase
shamsi/graphify

Interface text:
Join
AIE
AI Engineer EUROPE 2026
Engineering the future of AI

[5:50](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=350s) [A chart shows lower accuracy for information in the middle of context, alongside examples of knowledge-base and context-management work.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d7aec7a27c6c9a3930d4e175768dd391fabcfdf3db89f57a959dea2f00fb52a.webp)

WHY IT MATTERS FOR YOUR AGENTS

Context management requires a level of pre-processing

Accuracy drop when relevant info lands in the middle of context,

Y-axis: Accuracy
Position in context: Start, Middle, End
High; Low; High

Liu et al. 2023 · 18 models tested by Chroma · 194K LLM calls

Andrej Karpathy
@karpathy
LLM Knowledge Bases

Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into

Chroma Context-1: Training a Self-Editing Search Agent

r/ClaudeCode
u/captainkink07 · 18h · github.com
71.5x token reduction by compiling your raw folder into a knowledge graph instead of reading files. Built from Karpathy's workflow
Showcase
shamsi/graphify

Interface text:
Join
AIE
AI Engineer EUROPE

### A production story for tool calling

A production example classifies a Shopify product catalogue across 10K categories using extraction, text-and-image embeddings and reranking. Switching model type takes one parameter.

[6:30](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=390s) [A product classification diagram uses extraction, encoding, and scoring to assign a taxonomy category.](https://assets.ai.engineer/sites/aiecode2025/sha256/3b24bb54c401ff12e87a511b74438885bd3af5d6d6776088d4a7af13aec8f8c3.webp)

WHY IT MATTERS FOR YOUR AGENTS

A production story for tool calling

Taxonomy classification
10K categories Shopify product catalogue text + images

Product
title + image

sie.extract()
NLI classification

sie.encode()
embed text + images

sie.score()
rerank candidates

Electronics
› Computers
› Laptops

Switching model type = changing one parameter, not rebuilding infra

Context management
Manages what goes INTO context,

Tool calling
Embeddings, reranking, extraction

“Why not just Opus 4.6?”
Works until context rot kills quality

Event branding:
AIE
AI Engineer EUROPE

### What inference doesn't look like

Makraduli introduces a common view of inference: adding GPUs and compute solves the problem.

[7:02](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=422s) [Section slide titled What Inference Doesn't Look Like.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c84f14afdd67333ed03a404a2c8c5fd9984fa803c05b065f8dcbe36e223eea0.webp)

THE ANTITHESIS

What Inference
Doesn't Look Like

Event branding:
AIE
Engineering the future of AI
AI Engineer
EUROPE

### Multi-model GPU sharing

The example leaves 18–22GB idle on each of five 24GB GPUs. It shows five models sharing one GPU at 80%+ utilization, with LRU eviction swapping out the coldest model when memory is tight.

[7:10](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=430s) [A diagram compares five mostly idle dedicated GPUs with models sharing one GPU through hot-swapping and least-recently-used eviction.](https://assets.ai.engineer/sites/aiecode2025/sha256/b083139f77befb77103b9dfb9239573219ffe55a3fd529cc0688a63780da5fad.webp)

IT'S NOT ABOUT FASTER MODELS OR BIGGER GPUS

One model per GPU — most capacity wasted

GPU 1 · 24GB
stella
20GB idle

GPU 2 · 24GB
bge-m3
18GB idle

GPU 3 · 24GB
reranker
21GB idle

GPU 4 · 24GB
gliner
22GB idle

GPU 5 · 24GB
clip
21GB idle

Multi-model GPU sharing — hot-swap with LRU eviction

1 GPU · 24GB · 80%+ utilized
stella
bge-m3
reranker
gliner
clip

LRU eviction: coldest model swaps out when memory is tight

AIE
Engineering the future of AI
AI Engineer Europe

### It’s Not About the Server

A model server handles one model in one container, while production also needs routing, autoscaling, monitoring and deployment. Makraduli says teams often have to build that layer themselves.

[8:21](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=501s) [Diagram contrasting a solved model server with production needs including routing, autoscaling, monitoring, and deployment.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7a655f608ed0219fa9981105e0ac969f1ffccfd62d8e4f82b54f967ca466c44.webp)

IT'S NOT ABOUT THE SERVER

Model Server
Solved
TEI · vLLM · FastAPI wrapper
one model, one container

?
MIND THE GAP

Production
Routing — per-model, pool isolation
Autoscaling — KEDA + spot instances
Scale-to-zero — $0 when idle
Monitoring — Prometheus + Grafana
Terraform · Helm — AWS · GCP

Every team builds this layer alone. It barely exists in open source.

Interface text:
AIE
AI Engineer EUROPE
Braintrust
WorkOS
OpenAI

### What Is Inference About?

Makraduli shifts from what inference isn't to a holistic approach that combines two key elements.

[9:22](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=562s) [Slide asks, “So what is inference about?”](https://assets.ai.engineer/sites/aiecode2025/sha256/2a1607f20056a32320ab079e15f1276dec581db1fc6d7e5585987647cbd2b9d6.webp)

THE ANTITHESIS

So what is inference about?

AIE
AI Engineer
EUROPE
2026

### The Yin and Yang of Inference

Makraduli frames inference as a holistic approach combining two key things. The first is model support: offering users the right models and enough options.

[9:35](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=575s) [A yin-yang symbol above the title, “The Yin and Yang of Inference.”](https://assets.ai.engineer/sites/aiecode2025/sha256/22978748d739cbd83ebd6d99784c793b970684bbed70d5b7271794c72687dd25.webp)

The Yin and Yang of Inference

Event branding:
AIE
AI Engineer EUROPE
AI Engineer EUROPE 2026

### Model Support and Open Source Growth

Hugging Face Hub’s model count climbs to 2.7M by March 2026. Makraduli argues that inference needs broad model support as open-source models proliferate and improve.

[9:48](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=588s) [Line chart of Hugging Face Hub model counts rising from 2022 to 2.7 million by March 2026, with model milestones labeled.](https://assets.ai.engineer/sites/aiecode2025/sha256/8dd1777f9fd3873afef1e48ffc64e49546ff9692dd12aa3600285fc7415c51e8.webp)

YIN — MODEL SUPPORT IS KEY, OPEN SOURCE IS NOT STOPPING

2.7M
Y-axis ticks: 0, 500K, 1M, 1.5M, 2.5M+
Timeline: 2022, 2023, 2024, 2025, Mar 26
Model labels: BLOOM, LLaMA 2, Qwen2, Flux 1.0, DeepSeek-R1

Source: Hugging Face Hub statistics

AIE
Engineering the future of AI
AI Engineer EUROPE

### Model Performance vs. Size

The chart places two Gemma 4 models above several much larger models on Elo score, illustrating why infrastructure needs to keep up with smaller open-source models.

[10:42](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=642s) [Scatter plot comparing model Elo scores and parameter counts, highlighting two Gemma 4 models.](https://assets.ai.engineer/sites/aiecode2025/sha256/4bccae2a13ab8e4c1cf711e67270e5a12ade0206fe4262f7b7f486034e2f9719.webp)

YIN — MODEL SUPPORT IS KEY, OPEN SOURCE IS NOT STOPPING

Model Performance VS Size

Y-axis: Elo Score
Y-axis ticks: 1350, 1360, 1370, 1380, 1390, 1400, 1410, 1420, 1430, 1440, 1450, 1460
X-axis: Total Model Size (Billion Parameters)
X-axis ticks: 10, 20, 30, 40, 50, 100, 200, 400, 500, 600, 1000

Model labels:
gemma-4-31B-thinking
gemma-4-26B-A4B-thinking
qwen3.5-27b
qwen3.5-122b-a10b
qwen3.5-397b-a17b
deepseek-v3.2-exp-thinking
mistral-large-3
glm-5
kimi-k2.5-thinking
gpt-oss-120b

Gemma 4 model performance — your infra needs to keep up

Interface text:
AIE
AI Engineer EUROPE
Engineering the future of AI

[10:48](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=648s) [Scatter plot comparing model Elo scores and parameter counts, highlighting two Gemma 4 models.](https://assets.ai.engineer/sites/aiecode2025/sha256/a0c0b41ecd7580bb95fa8ca01b768fa5548376a7849ac76cd73e29ce1c4616c2.webp)

YIN — MODEL SUPPORT IS KEY, OPEN SOURCE IS NOT STOPPING

Model Performance VS Size

Y-axis: Elo Score
Y-axis ticks: 1350, 1360, 1370, 1380, 1390, 1400, 1410, 1420, 1430, 1440, 1450, 1460
X-axis: Total Model Size (Billion Parameters)
X-axis ticks: 10, 20, 30, 40, 50, 100, 200, 400, 500, 600, 1000

Model labels:
gemma-4-31B-thinking
gemma-4-26B-A4B-thinking
qwen3.5-27b
qwen3.5-122b-a10b
qwen3.5-397b-a17b
deepseek-v3.2-exp-thinking
mistral-large-3
glm-5
kimi-k2.5-thinking
gpt-oss-120b

Gemma 4 model performance — your infra needs to keep up

Interface text:
AIE
AI Engineer EUROPE

### Same five steps, different implementation

BERT and Qwen2 differ in normalization, position and attention; ColBERT outputs a vector per token. Packed variable-length attention avoids padding waste across sequences of different lengths.

[12:38](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=758s) [Table comparing normalization, Q/K/V, position, attention and output for BERT, Qwen2 and ColBERT.](https://assets.ai.engineer/sites/aiecode2025/sha256/e145d7252a299a97aaea71cda595bbaa4ad103769675a1cfe70539aba9af2e0f.webp)

SAME 5 STEPS, DIFFERENT IMPLEMENTATION

BERT | Qwen2 | ColBERT
1. Norm: Post-norm (after) | Pre-norm RMSNorm | From base arch
2. Q/K/V: 3 → we fuse into 1 | Separate (GQA) | From base arch
3. Position: Absolute lookup | RoPE rotation | From base arch
4. Attention: MHA, all same | GQA (fewer KV) | Bidirectional
5. Output: 1 pooled vector | 1 pooled vector | N vectors (per-token)

Different from BERT — different forward pass
Different output type entirely

All on packed sequences with flash_attn_varlen = zero padding waste

AIE
AI Engineer EUROPE 2025
Engineering the future of AI

## Full transcript

[0:00](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=0s) [on-hold electronic music] Hello, everyone.

[0:16](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=16s) Um, welcome to this talk. I'll be speaking about small model inference and a gap that we've recognized in the market, and what we did about it, and why we kind of made this approach.

[0:33](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=33s) And as you can see, this background slide here, um, this is no accident. So if you can guess what this is, I'll prompt you at the end of the slides.

[0:42](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=42s) You win a little reward, so you can catch me at the break afterwards. So think about this, but also listen to me, so don't think too hard.

[0:53](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=53s) So the story starts with me posting an article a few months ago on Substack that got a bit of traction, got a few people interested, and I explained FlashAttention, I explained how models worked, how processes can be memory-bound, compute-bound.

[1:11](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=71s) And I felt really good 'cause I kind of went deep into this, and as a person who's been in AI for a few years, I felt very confident. And that was true.

[1:23](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=83s) But then some people pointed out that actually I had overlooked one key aspect around what, uh, makes these models fast in the real world.

[1:36](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=96s) And that aspect that I overlooked was inference. So as someone who kind of wants to understand things in first principles and work to understand the problems and the solutions deep, I realized, okay, I need to figure this out.

[1:53](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=113s) I need to know kind of where I've made my mistake, and as a AI researcher and engineer, I need to find more about inference. So I've done a lot of work with vLLM, training models, fine-tuning, kind of doing applied, uh, ML and AI.

[2:12](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=132s) Did a bit of also research in academia as well. But this part around kind of how models run in production, scheduling GPUs, routing, and automation, I guess, was a bit of a b-blind spot for me.

[2:26](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=146s) So I realized, okay, this is the time I have to now figure this out, learn and make it work. And what better way to do this than to actually build stuff?

[2:38](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=158s) So I decided to join a team, um, a team at Superlinked, uh, comprised of very good, uh, infrastructure engineers and actually work with them and build something around inference.

[2:54](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=174s) And that something is, um, this, uh, repo, the Superlinked Inference Engine that we have open-sourced, and this is kind of like the soft launch that I'm doing today, so you can have a look at that later.

[3:06](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=186s) So basically, it's inference for small models around AI search and document processing. And we've tested this out, as you can see, with some of our partners. So we've tested it out with Chroma, Qdrant, Weaviate, so a lot of the vector DBs, uh, as well as LanceDB.

[3:23](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=203s) And as you can see, they've tried it out a bit. Sounds fun, sounds interesting, so this is working. And, uh, it was the right step for me to kind of figure this out and learn where inference, um, kind of truly is and how that combines with my ML experience.

[3:42](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=222s) So the three key points I want you to, uh, go away with from this talk are these. So first, I want to tell you why this matters, so why doing inference for AI search and document processing actually matters when you're building agents or when you're building workflows that involve, uh, agents.

[4:04](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=244s) So this is very important, the why. Then I want to talk to you about the second thing, which is what inference is not about. So there are some misconceptions or ideas of how inference looks like, but it's not all about those things.

[4:20](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=260s) And the third thing is how we see inference, and I call this as like the yin and yang of model inference, and it's a way of combining a few things around model support and infrastructure.

[4:35](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=275s) So why this matters for your agentic workflow. Well, what you have encountered for sure is context rot. And as we probably all know, this research paper from Chroma from some time ago, um, showcases this effect that no matter what you do, there is this effect of context rot.

[4:55](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=295s) So quality degrades as context increases. So being able to manage this context and do some context management is very important and a useful way to solve this. So using small models that can pre-process your data so that then you can actually use your agents and build your workflows is a very powerful technique.

[5:18](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=318s) You can also use the small models for tool calling to, again, do similar things and tackle this, uh, problem of context management. And you might say, "Okay, why would I not use like Claude Code and grepping?"

[5:31](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=331s) And that's a valid point. However, you can still do that, and having your data being pre-processed is actually making the grepping and file systems that you build even better.

[5:44](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=344s) And it's not just me saying this, so this is how the community has responded to this problem. So Andrej Karpathy is building knowledge bases, graph-based. So for example, you can use named entity recognition models, um, to generate ontologies and then build knowledge graphs.

[6:00](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=360s) Or- You generate your file systems in that way. Chroma have also, uh, shipped their own model that actually does this. It kind of pre-processes and filters the input, manages the context in order to tackle and achieve the same problem.

[6:17](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=377s) And there's also people in the community building a lot of solutions that, um, lower this, um, context and kind of reduce the token amount so that the agents can be even more effective.

[6:32](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=392s) And we've done this in production as well. We have a use case where this repo that I showcased, the, um, Superlinked Inference Engine, um, we've used it as a tool calling basically, uh, solution where it was around taxonomy classification for like an e-commerce store.

[6:50](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=410s) So going directly, uh, with tool calling, where the small models are tools that can retrieve and go through your data is also a powerful way to approach this.

[7:03](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=423s) So now I'll go about inference and what inference doesn't look like. So the traditional maybe perspective is, okay, I will just chuck in more GPUs, get more compute, all good, I've solved inference.

[7:19](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=439s) However, um, with small models, where actually each model, um, takes up only a small space in memory. So you can see, for example, Stella, which is an embedding model, then other rerankers, um, the named entity recognition GLiNER model, they only occupy like a few gigabytes of memory, and if you provision a GPU for each model, you're wasting

[7:43](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=463s) a lot of, um, idle space. Your GPU stays idle, and it's not used. So in this case of small model inference, it's very important to be able to hot-swap models.

[7:54](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=474s) So what we've done is we've built the ability for you to swap all of these models in one GPU so that per GPU, you get much higher utilization. So this lowers your costs, but also enables you to hot-swap and switch around between models quickly.

[8:10](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=490s) If you want to have one tool that's one reranker or another tool that's another model, you can switch them around quickly, and we have this least recently used eviction policy that we've built in.

[8:23](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=503s) And also, what inference is not about is, um, only the server or only kind of the production situation. So we have solutions for both the server and the production, but building something, so for example, like using TEI or vLLM or having even, uh, some API wrapper, in order to get that in production where you actually do routing,

[8:48](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=528s) autoscaling, um, you do kind of monitoring with Prometheus metrics and Grafana, you have to write that code yourself. And on the market currently, there is no open source solution that leads you from creating the model inference to actually productionizing this at a scale, um, of this size.

[9:09](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=549s) So that's why we've kind of wanted to fill this gap as well. So, um, we've included a lot of stuff around ri- uh, routing, autoscaling, queuing mechanisms, and provisioning GPUs.

[9:24](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=564s) So I've talked about what inference isn't, why this is important, and let me tell you now about, um, what actually inference is about. And I call this the yin and yang of inference because I feel like this is a holistic approach that has to combine two key things.

[9:45](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=585s) So first, the yin is model support. So your inference is worthless if you're not supporting the right models or you're not offering enough breadth of, um, options for your users.

[10:01](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=601s) And open source models, so on Hugging Face currently there are millions of models. Um, this is from March. Now there might be even more like close to three million models.

[10:12](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=612s) And open source is moving very quickly, both in size but also in accuracy. So you need to support these models because people want to use them, people want to work with open source, and the performance is also getting better and better.

[10:26](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=626s) So it's not even a sacrifice anymore. It's actually you're being able for specific tasks. So if you look at MTEB or different benchmarks, you can see that for very narrow tasks, um, open source models are beating managed services.

[10:43](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=643s) And we see this even with more general purpose models like, uh, what Gemma have done. So they've released very low parameter models that have Elo scores that are higher than, um, much, much bigger models.

[10:58](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=658s) So open source small models are very relevant, and you need an infra, um, to support, to support them.

[11:10](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=670s) And we decided to do exactly this. Okay, let's do it. We'll build and support hundreds of models. However, uh, that's not as straightforward as it might sound because all of these models have different runtimes.

[11:27](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=687s) So for example, with if you use BERT and Qwen as, um, models, they have different implementation of FlashAttention, they have different positional embeddings, and you need to adjust this in order to make, um, make this kind of applicable to your case and, uh, make it more general.

[11:47](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=707s) So there is no universal engine that can handle BERT, Qwen, and ModernBERT as well because the architectures are different. So what, what we've done is we've set up, um, a way of re-reimplementing this forward pass in order to adapt attention, in order to do padding where needed, so have variable length attention as well.

[12:09](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=729s) Uh, work with kind of fuse- fusion of the, uh, query key and value, um, and this is kind of an underrated component, especially if, um, you look at supporting models like ColBERT that has multiple vectors as output, so late interaction models.

[12:27](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=747s) Also, cross encoders and re-rankers that do not output a vector at all, but they output kind of scores. So it's very important to, um, figure this out, and one example of this is, for example, this, which is, um, a comparison of a few models that are totally different in this five aspects.

[12:49](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=769s) So normalization is done differently in BERT and Qwen, so that needs to be accounted for. In ColBERT, um, it's also totally different. Uh, the q- uh, the query key and values are also can be fused somewhere.

[13:06](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=786s) In others, um, for example, in Qwen, it cannot be because there is, um, grouped query attention, so that's another problem. The positioning- positional embeddings are also different. You can do in BERT absolute lookup where at w- with Qwen, there is, uh, rotary positional embeddings.

[13:24](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=804s) So these are problems that maybe, uh, people don't think about straight away, but if you want to support a lot of models, there needs to be a consistent way of doing this.

[13:34](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=814s) So we've set up this with kind of working with agents as well as with humans to be able to support this and re-implement this forward pass in order to make, um, the, the inference of this model efficient.

[13:48](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=828s) And we've also worked, uh, to do FlashAttention, um, where it's variable-length FlashAttention so that you can do, uh, padding and that there is no waste. Because what happens is if you wanna do token-based batching, you can have tokens that are kind of, um, lower number of tokens, one request, and then another is a higher amount, and then

[14:11](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=851s) if both are padded at the higher amount, you basically are wasting compute on empty tokens. So you need to adapt this in order to make the models, uh, work even better and quicker.

[14:23](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=863s) Um, and that's a key differentiator in our way of approaching inference. The other part, the yang, is around

[14:36](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=876s) infrastructure. So this is the complex plot. Um, it's kind of this is the deep dive. We have this in our website and in our repo. I won't go into each component here.

[14:46](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=886s) But basically, what's up there is the three primitives of our A- API, so encode, score, and extract. And then the infrastructure layer happens, and this is kind of the summary of it.

[14:57](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=897s) So basically, we have the three primitives of the API, and then we have a router and also a queuing mechanism depending on, uh, the load, so we're adjusting between, uh, these and also different pools of different GPUs and resources that, um, we use in order to kind of distribute the workload.

[15:18](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=918s) And we, we use spot instances but also bigger GPUs. So it's about being able to provision the hardware and also have the metrics to autoscale this. So we're doing KEDA autoscaling with Prometheus metrics in order to be able to kind of switch models around, not keep GPUs idle and kind of waste resources.

[15:41](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=941s) So the key driver here is that we want you to have the cluster as well as the model support. So not just one or the other, and then you have to merge them together and write all that code.

[15:54](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=954s) But we want to give you the whole end-to-end thing so that you can work with these models quickly, and the models are basically just a config that you can switch around and then do terraform apply, and that's it, and we also have published Helm charts and Docker images as well.

[16:11](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=971s) And that's how I learned my lesson. Um, uh, we've built SIE, so this is kind of our soft launch in a way. And we've open sourced both the model inference that I talked about with kind of adapting this forward pass, working with attention, re-implementing different aspects on the model, but also the cluster so that you can actually

[16:31](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=991s) use this straight away without having to think about hardware, provisioning GPUs, and all of those problems that arise in, in the real world. Um, you can scan the QR code to look at the repo, and it's called SIE, so S-I-E.

[16:47](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1007s) Um, and the company is Superlinked. And also now we have the, the background as well, so a quick reminder on that. Um, does anyone have any ideas maybe from the audience?

[17:01](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1021s) I'll reveal it in the next slide, but yeah, I don't want to maybe do it too quickly. If anyone knows what this is, think of it machine learning a bit, kind of, uh, foundational knowl- I was talking about attention and all that, so it's a bit around embeddings, but also how transformers work.

[17:21](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1041s) Okay. No, no, no reward, I guess, for you, sadly. Or yeah, yeah. Sorry?

[17:28](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1048s) Can't remember exactly, but there was one with, like, all the embeddings were clustering on, like, one area and wasn't using most of the embedding space-

[17:36](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1056s) Okay

[17:36](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1056s) ... because of the way we were doing the positional encoding. As much as I remember.

[17:43](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1063s) So I will take that as a correct answer. So it was around positional, uh, encodings, and yes, this is basically... Yeah. Congrats. [audience applauding] So yes, this is a- around vector visualization of positional encoding.

[17:57](https://www.youtube.com/watch?v=qdh_x-uRs9g&t=1077s) So this is done when transformers are trained. This is how positions are encoded, and it's sinusoidal. That's why you get this pattern because it- they're embeddings. And yeah. Thank you very much. [audience applauding] [upbeat music]
