# Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub

Arek Borucki, Hugging Face

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/lyL5QhgIOxc-serving-2-million-models-without-melting-scaling) · [Video](https://www.youtube.com/watch?v=lyL5QhgIOxc)

## Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub

How Hugging Face separates metadata from artifacts, prepares names for search, distributes database work, and scales Kubernetes capacity as its model catalog grows.

Familiarity with database queries, replication, and Kubernetes pods will help; the article explains the search and scaling mechanisms as they appear.

### What happens when the model catalog outgrows search?

[0:33](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=33s)

How do you keep a model hub responsive while its catalog grows into the millions? Developers still need to publish, discover, and download models without thinking about the infrastructure behind those actions. [Arek Borucki](https://arekborucki.cloud/), a machine learning platform and database engineer at Hugging Face, describes the decisions behind that experience. Although the recording’s title says two million models, his account uses a spoken figure of three million.

At the time of the talk, Borucki reports more than 14 million users, three million public models, one million datasets, and 50,000 organizations. He also reports that more than 30% of Fortune 500 companies use Hugging Face in their AI workflows. The opening slide labels the public-model count more precisely as 2.9M+.

![Slide lists 14M+ users, 2.9M+ public models, 1M+ datasets, 50K+ organizations, and adoption by 30%+ of the Fortune 500.](https://ai.engineer/api/talk-essay-media/7d255af04266e3bee47fe0d7371a3b2a8743c97f70dcb247e95caeabe21c625b)

Hugging Face’s scale: users, public models, datasets, and organizations.

Borucki puts the model catalog’s growth at roughly 150×, from 20,000 models three years earlier to three million. Major releases such as **Llama** and **DeepSeek** generate thousands of derivative models, so a successful release also creates a wave of new repositories to manage.

Datasets follow the same trajectory: Borucki reports 10,000 in 2022, 100,000 in 2024, 500,000 less than a year before the talk, and one million at the time of the talk. Every addition must be stored, indexed, and made searchable. Storage growth therefore becomes a discovery problem, not just a capacity problem.

### The slow end of search matters

[4:09](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=249s)

At 20,000 models, Borucki recalls, even queries without an index seemed fast enough. At three million, that approach broke down. Users expect search results immediately; a slow catalog gives them a reason to leave. His illustrative calculation is that 1% of 14 million users would mean 140,000 people experiencing slow search. That is a hypothetical affected population, not a measured incident or a conversion from request percentiles into distinct users.

**Tail latency becomes a design priority.** P50 describes the median request; P99 describes the latency boundary below which 99% of requests fall. Borucki emphasizes P99 because a healthy median can conceal a painful slow tail.

![Three slide panels describe expectations for instant results, growth from 20K to 3M models, and the importance of p99 over p50 for 14+M users.](https://ai.engineer/api/talk-essay-media/4c3b71b64b502b95641394030e39d083626bb6666496efaa576b50997799d2fd)

At three million models, search latency and p99 matter.

The response spans several layers: precomputed search tokens, a denormalized MongoDB collection optimized for reads, Lucene-backed search, and Kubernetes autoscaling. Database sharding is the next planned step. Each addresses a different source of work or contention rather than expecting one larger machine to solve everything.

### Separate compute, metadata, and model files

[5:41](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=341s)

A request travels from the frontend to the Hub API, which runs on Kubernetes. The [Horizontal Pod Autoscaler](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/) adds application pods during traffic spikes and removes them as demand falls. The request then reaches MongoDB Atlas, the source of truth for Hub metadata.

**MongoDB stores information about models, not the model binaries themselves.** Its responsibilities include users, repositories, model and dataset records, buckets, Spaces information, configuration data, billing, and access control. Model artifacts, tokenizer files, card assets, and configuration files live separately in cloud object storage such as AWS S3.

| Layer | Responsibility |
| --- | --- |
| Kubernetes | Run and scale Hub application compute |
| MongoDB Atlas | Store and query metadata |
| Cloud object storage | Store model files and other artifacts |

This separation lets the team scale metadata independently of binary storage, and compute independently of both. Each component can be optimized for its own workload.

### Prepare model names before anyone searches

[7:55](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=475s)

Consider someone typing `llama` into the Hub search bar. The request reaches a dedicated MongoDB read collection, not the main repository collection containing all repository data. This separate, denormalized copy is built for reads and listings.

The key optimization happens earlier: **model names are tokenized at insertion time rather than query time**. For the example identifier `meta-llama/llama3.1-8b`, the stored array includes tokens such as `meta`, `llama`, `3.1`, and `8b`. Atlas Search uses Lucene-backed autocomplete to match against that prepared representation. The query does not have to start by decomposing every model name.

Borucki inspects the stored representation with `findOne`, selecting that model identifier and projecting only its search-token array. The useful distinction is between the repository’s full identifier and the smaller searchable pieces stored alongside it: the readable model name remains intact while search gets a representation suited to matching.

### Replace regex matching while keeping trending order

[10:03](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=603s)

The earlier search implementation used MongoDB’s `find` method with a regex against `searchTokens`, then sorted matches by descending `trendingScore`. The query’s structure is compact:

```javascript
db.models
  .find({ searchTokens: { $regex: "llama" } })
  .sort({ trendingScore: -1 });
```

Borucki says the trending score is recalculated every five minutes using downloads and likes; he recalls a seven-day activity window. Matching determines which models qualify, while the trending score determines their order.

![MongoDB query with arrows highlighting find, the regex search for llama in searchTokens, and descending trendingScore sorting.](https://ai.engineer/api/talk-essay-media/35d90b72e101164cc05742351316a7244502e369d58771e0b64d16174a97409e)

The existing regex search sorts results by trending score.

That approach worked while the dataset was small. As the catalog grew, regex search developed latency problems, prompting the move to Atlas Search and Apache Lucene. In the architecture Borucki describes, a separate process, `mongot`, wraps Lucene while exposing search through MongoDB’s query interface. This is distinct from MongoDB’s native [text indexes and `$text` queries](https://www.mongodb.com/docs/manual/core/indexes/index-types/index-text/); core MongoDB does have text-search capabilities, but those are not the Lucene-backed autocomplete path used here.

The replacement uses an aggregation pipeline with `$search`, names the search index, and applies `autocomplete` to the token field. It still sorts by trending score. Expressed as a JavaScript helper taking the configured index name, the query shape is:

```javascript
function searchLlamaModels(models, indexName) {
  return models.aggregate([
    {
      $search: {
        index: indexName,
        autocomplete: {
          query: "llama",
          path: "searchTokens"
        }
      }
    },
    { $sort: { trendingScore: -1 } }
  ]);
}
```

The change is in the matching engine, while the product’s popularity-based ordering remains in place.

Borucki reports that the replacement had scaled well so far and resolved the team’s search-bar latency issues; he provides no latency distribution or benchmark workload. The two displayed results have trending scores of 33 and 14, illustrating the retained descending order.

### Keep unnecessary work off the primary

[12:56](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=776s)

Search is only one database consumer. Borucki describes hundreds of Hugging Face services using MongoDB and a seven-node cluster handling millions of queries, without specifying a time interval. Distributing queries across machines prevents one node from becoming the read bottleneck. All inserts, updates, and deletions still go to the single primary; reads can be spread across other members.

One member is a **hidden analytics node**. It replicates data from the primary but is invisible to ordinary application routing: the MongoDB driver does not send normal application queries there. The team connects directly to it for reporting and heavy queries. Like the other secondaries, it stays synchronized by following the primary’s operation log, or `oplog`.

Routing then follows the workload’s requirements:

- **Freshness-sensitive reads:** In the described policy, queries requiring strong consistency stay on the primary. Reads that do not require the latest data go to secondaries.
- **Complex aggregations:** Pipelines that scan large amounts of data, sort, group, or transform records run on secondaries rather than competing with primary-only work.
- **Change streams:** Cache invalidation, synchronization to systems such as AWS Redshift, and event-driven workloads react to database changes. This work also belongs on secondaries where possible.
- **Ad hoc analysis:** Reporting, experiments, and exploratory queries go to the hidden member, isolated from ordinary production traffic.

**The primary should concentrate on work only the primary can do.** Moving the rest elsewhere preserves its capacity for writes and the reads that need it.

### The next database step is partitioning

[16:38](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=998s)

Read distribution extends the useful life of a replica set, but it does not remove the limits of keeping the entire dataset on each member and sending writes to one primary. Borucki presents **sharding as a planned next step**, not an already completed migration. The dataset would be divided among shards, each retaining its own primary and secondaries.

| Property | Single replica set | Planned sharded cluster |
| --- | --- | --- |
| Data placement | Full dataset on each member | A portion on each shard |
| Replication | One set of replicas | Replication within each shard |
| Write destination | One primary | The primary of the relevant shard |
| Horizontal expansion | More read-serving members | More shards sharing the dataset |

Adding shards introduces more places to store and process data. MongoDB’s balancer redistributes data across them, but the distribution depends on a consequential design choice: the shard key. Borucki explicitly leaves that nontrivial choice outside the talk’s scope. The intended expansion covers CPU, memory, storage, reads, and writes.

### Scale application replicas and the capacity beneath them

[18:19](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1099s)

At the application layer, the Hub’s HPA configuration responds when CPU or memory rises above its target. Borucki reports that the Hub deployment can scale from 10 to 500 pods depending on traffic. It scales back down as demand drops, reducing the need for manual intervention and permanent overprovisioning.

More desired pods do not guarantee more usable capacity. If the Kubernetes cluster has no room for them, the scheduler leaves them pending. [CAST AI](https://docs.cast.ai/docs/business-continuity) supplies the second scaling layer:

1. HPA increases the deployment’s requested pod count.
2. Pods remain pending when existing nodes cannot accommodate them.
3. CAST AI adds nodes.
4. The Kubernetes scheduler places the pending pods on the new capacity.

Pod scaling changes how many application replicas should run. Node scaling supplies the infrastructure on which those replicas can run.

The next planned change is adoption of **KEDA**, Kubernetes Event-Driven Autoscaling, to make scaling decisions from application signals such as requests per second and event-loop utilization. A pod can have low CPU utilization while requests accumulate in a queue; resource utilization alone can therefore miss demand that matters to users. The CPU/memory limitation here describes the Hub’s presented HPA configuration, not HPA in general: HPA also supports custom and external metrics, and KEDA integrates with it rather than replacing its underlying control loop.

![Kubernetes diagram connects users, a Hub Deployment labeled 10 to 500 pods, KEDA, and horizontal scaling, with panels for application-aware autoscaling and workload signals.](https://ai.engineer/api/talk-essay-media/531d257c30d5ab7156b537b4ce7e84bf51602bc57c5ec7316483058b7a218f73)

KEDA autoscaling driven by requests and event-loop signals.

The user-facing test of all this machinery is simple: pushing a model, searching the Hub, or downloading a model should work without requiring the user to understand it. The internal architecture can become more elaborate as the catalog grows, while those everyday actions remain straightforward.

## Resources

- [Kubernetes Horizontal Pod Autoscaling](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/) — Configure pod autoscaling using resource, custom, or external metrics, including scaling policies and stabilization.
- [MongoDB Text Indexes](https://www.mongodb.com/docs/manual/core/indexes/index-types/index-text/) — Native MongoDB text indexes, $text queries, and their storage and performance constraints.
- [Cast AI Node Autoscaling and Business Continuity](https://docs.cast.ai/docs/business-continuity) — How pending pods trigger node provisioning, with guidance for detecting and handling capacity interruptions.
- [State of Open Models: Summer 2026 Observations](https://github.com/huggingface/blog/blob/main/state-of-open-models-summer-2026.md) — Hugging Face's summer 2026 analysis of public repository growth, downloads, likes, and derivative models.

## Slides

### Serving 3 Million Models

Arek Borucki introduces the architectural decisions and challenges behind scaling Hugging Face’s infrastructure to serve three million models to developers worldwide.

[0:27](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=27s) [Title slide introducing Arek Borucki’s talk on scaling the Hugging Face model hub.](https://assets.ai.engineer/sites/aiecode2025/sha256/24c8172ab919e4c8adcb4cbc71e97c001a98d3fdb91d8da368241829cbcf5efd.webp)

Serving 3 Million Models
How Hugging Face Scaled the World’s Largest Model Hub

27K
35K
41K
TA
50K
40K
50K

Arek Borucki
ML Platform & Database Engineer @ Hugging Face
MongoDB Champion | Author of MongoDB 8.0 in Action

### Hugging Face: The Scale

Hugging Face reports 14M+ users, 2.9M+ public models, 1M+ datasets and 50K+ organizations; more than 30% of the Fortune 500 use it in AI workflows.

[1:34](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=94s) [Scale figures for users, public models, datasets and organizations, with a Fortune 500 usage claim.](https://assets.ai.engineer/sites/aiecode2025/sha256/5703a4c9417c7ec24c7cabada721da9c406f90daab783ad4cccc618d63c3ba64.webp)

Hugging Face: The Scale

14M+ USERS
2.9M+ PUBLIC MODELS
1M+ DATASETS
50K+ ORGANIZATIONS

USERS
AI
PUBLIC MODELS
DATA
DATA
DATA
DATASETS
ORGANIZATIONS

30%+ of the Fortune 500 use Hugging Face

### Growth of Models on Hugging Face

The chart shows model counts rising to 3M by mid-2026. The speaker says major releases such as Llama and DeepSeek generate thousands of models built on top, adding pressure on Hub infrastructure.

[3:00](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=180s) [Chart showing cumulative models rising to 3 million by the first half of 2026, with milestones and model releases annotated.](https://assets.ai.engineer/sites/aiecode2025/sha256/4a2b8de287d0669482567a11b52a98e0096efcbc9ce844b42fd71e1800d23990.webp)

Cumulative Number of AI Models on Hugging Face

Y-axis ticks: 0, 500K, 1M, 1.5M, 2M, 2.5M, 3M
Timeline: 2022, 2023, 2024, 2025, 2026 (H1)

100K Models
500K Models
1M Models
2M Models
Halfway through 2026
3M Models

BLOOM
LLaMA 2
Qwen2
Flux 1.0
DeepSeek-R1
GPT-OSS

### Hugging Face Crosses 1 Million Datasets

Hugging Face’s datasets grew from 10K in October 2022 to 1M by 2026, across task categories. The speaker says making the growing collection searchable required rethinking search.

[3:27](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=207s) [Stacked area chart showing cumulative datasets by task category through 2026, with 10,000, 100,000 and 500,000 milestones.](https://assets.ai.engineer/sites/aiecode2025/sha256/e60710dce9f968e0d43f4aeb539aed071509151bd1e3b1d68cdd94174560dba0.webp)

Hugging Face just crossed 1M datasets
Cumulative datasets published, by task category

Y-axis: Cumulative datasets
Y-axis ticks: 0, 200k, 400k, 600k, 800k, 1.0M
Timeline: Apr, Jul, Oct 2022; Jan, Apr, Jul, Oct 2023; Jan, Apr, Jul, Oct 2024; Jan, Apr, Jul, Oct 2025; Jan, Apr 2026

Oct 2022: 10K datasets
Feb 2024: 100K datasets
Sep 2025: 500K datasets

Legend:
Other
Tabular
Reinforcement Learning
Audio
Multimodal
Computer Vision
Natural Language Processing

### When 20K Become 3 Million: Latency Matters

A search approach that was fast at 20K models breaks at 3M. With 14 million users, even 1% experiencing slow search means 140,000 people, so the team prioritizes p99 over p50.

[4:05](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=245s) [Slide explains that search approaches must change as the model count grows and highlights the effect of p99 latency on users.](https://assets.ai.engineer/sites/aiecode2025/sha256/16f3ce0a7174557d37725e468bb26b28ccdc902911452b6527b71b3a7add46f8.webp)

When 20k become 3 Million. Latency Matters

Users expect instant results.
Slow search = they leave.

At 20K models any query was fast.
At 3M the same approach breaks.

14+M users, p50 lies.
What hurts UX is p99.

### Hugging Face

[5:32](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=332s) [Hugging Face mascot centered on an orange background.](https://assets.ai.engineer/sites/aiecode2025/sha256/b40aba13febf4aa4aec8713cf0fbfa2b7bc089dfbc905055b5490764735f8762.webp)

### Hub architecture and storage

Requests pass through the Hub frontend and API to MongoDB Atlas for metadata. Model artifacts live in cloud storage, allowing metadata, binary storage, and compute to scale independently.

[6:27](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=387s) [Diagram showing requests through the Hub to MongoDB Atlas, with model artifacts in cloud storage.](https://assets.ai.engineer/sites/aiecode2025/sha256/d2bb94fc7d7a3df3f6e5c297a8c13c43cb3ec64e98805647881bf336ac63845f.webp)

High-level Architecture

Users
Hub Frontend
Hub API

MongoDB Atlas
(Source of Truth for Metadata)
(Holds metadata, card text, configuration data, user info)

MongoDB doesn’t store the models.
It stores everything about the models

Cloud Storage
(S3 / GCS - Binary Data & Models)
Holds all model weights, tokenizer files, card assets, and configuration files

### Searching three million models

The Hub searches a separate read collection using model-name tokens computed at publish time, returning ranked results with a stated p99 under 15 ms and no external cache.

[7:54](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=474s) [Diagram of model search using precomputed tokens and an optimized read collection, with a stated p99 under 15 milliseconds.](https://assets.ai.engineer/sites/aiecode2025/sha256/645cd03cd68d54f9f47cf763e373f663b88bdbfec7dc57665fd8633f59184a01.webp)

How we search 3M Models under 15ms

Users → “Llama” → Hub → Optimized Read Collection → results → Ranked Results

Write time: meta-llama/Llama-3.1-8B → meta | llama | 3.1 | 8b
Pre-computed search tokens

Tokens at Write Time
Tokenized once at publish, searched millions of times

p99 Under 15 ms
One query, one collection, zero additional lookups

No External Cache
Right index, right data model.

One collection. One query. Pre-computed tokens in, ranked results out.

### Precomputed search tokens

A findOne query projects a model’s precomputed search tokens, showing both individual components and combined forms of its name in one array.

[9:28](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=568s) [A findOne query projects a model's search token array, followed by example tokens.](https://assets.ai.engineer/sites/aiecode2025/sha256/52a2074a216ec7e22bc44016c647eb575590bc7e0d33ab718191bf09efe1d297.webp)

db.model.findOne(
  { id: "meta-llama/Llama-3.1-8B"},
  { searchTokens: 1, _id: 0 }
)

{
  searchTokens: [
    'metallamallama3.18b',
    'metallama',
    'llama3.18b',
    'meta',
    'llama',
    '3.1',
    '8b'
  ]
}

### Regex model search query

The former MongoDB query searched precomputed tokens for “llama” and sorted results by trending score. As the dataset grew, regex search caused latency problems, prompting a switch to Atlas Search.

[10:06](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=606s) [A MongoDB find query searches the token array with a regular expression and sorts by trending score.](https://assets.ai.engineer/sites/aiecode2025/sha256/42d1ddc8c2ea5fbeec3c8f3b87d08ec254dbc626bca0dc1017be36efa398c79a.webp)

db.model.find(
  {searchTokens:
       {$regex: "llama"}
  },
  {
    _id: 0,
    id: 1,
    downloads: 1,
    trendingScore: 1,
    pipeline_tag: 1
  }
).sort({
  trendingScore: -1
})

### MongoDB and Lucene architecture

MongoDB sends queries from mongod to mongot, a separate process wrapping Apache Lucene. The diagram labels the reverse arrow “Change streams.”

[11:15](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=675s) [Diagram showing mongod and mongot processes with arrows labeled Query Lucene and Change streams.](https://assets.ai.engineer/sites/aiecode2025/sha256/0f1cdc03357b46b6d556740366a6ea680f3b2c16e1970ccdc1808865eded6189.webp)

High-level Architecture

MongoDB
{}
mongod process

Query Lucene →
← Change streams

Apache Lucene
{}
mongot process

### Autocomplete search query

The aggregation searches the model_search index for “llama” in searchTokens, sorts matches by descending trending score, and returns at most two results.

[11:46](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=706s) [MongoDB aggregation query using autocomplete and sorting by trending score.](https://assets.ai.engineer/sites/aiecode2025/sha256/91359a57d3e4b9798885ce70e386d2ec16e01b7feae0e1a5d5d72d725174ab70.webp)

db.model.aggregate([
  {
    $search: {
      index: "model_search",
      compound: {
        should: [{
          autocomplete: {
            path: "searchTokens",
            query: "llama",
            tokenOrder: "sequential"
          }
        }]
      },
      sort: { trendingScore: -1 }
    }
  },
  { $limit: 2 },
  {
    $project: {
      _id: 0
    }
  }
])

### Autocomplete search results

The previous autocomplete query returns two Meta Llama models, ordered by trending score: 33 for Llama 3.1 8B and 14 for Llama 3.2 1B.

[12:38](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=758s) [Two model search results with trending scores of 33 and 14.](https://assets.ai.engineer/sites/aiecode2025/sha256/700ec6d7ecd940516d0cb73a7864c3dbe5453674b47c1290534aefe85ad9ef1b.webp)

{
  id: 'meta-llama/Llama-3.1-8B-Instruct',
  author: 'meta-llama',
  likes: 5899,
  trendingScore: 33,
  downloads: 10561150,
  pipeline_tag: 'text-generation'
},
{
  id: 'meta-llama/Llama-3.2-1B-Instruct',
  author: 'meta-llama',
  likes: 1431,
  trendingScore: 14,
  downloads: 8119763,
  pipeline_tag: 'text-generation'
}

### MongoDB replica-set architecture

Hugging Face sends writes to one primary and distributes reads across nodes. A hidden analytics node replicates primary data and serves reporting and heavy queries through direct connections.

[12:54](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=774s) [Client application connected to primary, secondary, read-only and analytics nodes, with replication from the primary.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ab624aef5b0435d52ac844ba6eba02c22a8e9b54067ea84814dacada2479993.webp)

CLIENT APPLICATION

READS
WRITES
READS

SECONDARY NODE
(Electable)

PRIMARY NODE

SECONDARY NODE
(Electable)

DATA
DATA

READ-ONLY NODE
READ-ONLY NODE
DATA REPLICATION
READ-ONLY NODE
ANALYTICS NODE
Hidden

### What works on secondaries

Hugging Face sends reads that don't need the latest data, heavy aggregations, and change streams to secondaries. Ad hoc and reporting queries go to a hidden member isolated from production traffic.

[14:44](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=884s) [Four secondary workloads: offloaded tasks, aggregation pipelines, change streams, and reporting and analysis.](https://assets.ai.engineer/sites/aiecode2025/sha256/3cd39d26e459d616d7bd7a1897e8e12c295781e470c5d1ed5ef2450ef3897920.webp)

What works on secondaries

OFFLOAD WORKLOAD
primary → secondary
Offloaded Tasks

COMPLEX AGGREGATIONS
FILTER
GROUP BY
+ T L
Aggregation Pipelines

REAL-TIME DATA FLOW
CHANGE
CHANGE
DATASET
MONITOR
Change Streams

AD HOC + REPORTING
Reporting & Analysis

### MongoDB sharded cluster

The proposed cluster splits data across replica-set shards. Mongos routers direct client requests, while a balancer moves chunks between shards.

[16:36](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=996s) [Client requests pass through mongos routers to two replica-set shards; config servers and a balancer support the cluster.](https://assets.ai.engineer/sites/aiecode2025/sha256/8aa8b435178bef525ce5585d7fe05d14631509af1999641cf994b1f80a10d38b.webp)

CLIENT APPLICATION
WRITES/READS

MONGOS
(QUERY ROUTER)

MONGOS
(QUERY ROUTER)

MONGOS
(QUERY ROUTER)

METADATA & CONFIG
CONFIG SERVERS
(REPLICA SET)

SHARD A
(REPLICA SET)
PRIMARY NODE
(SHARD A)
DATA
DATA
SECONDARY NODE
(ELECTABLE)
SECONDARY NODE
(ELECTABLE)

BALANCER
CHUNK MIGRATION

SHARD B
(REPLICA SET)
PRIMARY NODE
(SHARD B)
DATA
DATA
SECONDARY NODE
(ELECTABLE)
SECONDARY NODE
(ELECTABLE)

### Scaling the Hub

The Hub’s Kubernetes HPA scales its deployment from 10 to 500 pods based on CPU and memory utilization. The speaker says it adds pods during spikes and scales back down when traffic drops.

[18:13](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1093s) [Kubernetes HPA adjusts the Hub deployment from 10 to 500 pods based on CPU and memory utilization.](https://assets.ai.engineer/sites/aiecode2025/sha256/33df690ad6759ed01a8f466a50454f2e9fe28db2c09b3f8d3a97d537ba525bf5.webp)

Scaling the Hub

Users

Hub Deployment
10 → 500 Pods

HPA
(CPU/memory > target)

Horizontal Scaling

Kubernetes HPA
Automatically adjusts replicas based on CPU and memory utilization.

Up to 500 Pods
Critical Hub services can scale from 10 to 500 pods.

Resource-Based Scaling
Scaling is driven by CPU and memory exceeding target thresholds.

### Multi-Layer Autoscaling

HPA adds pods as demand rises. When pending pods cannot be scheduled for lack of capacity, CAST AI adds nodes so Kubernetes can schedule them.

[19:03](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1143s) [Diagram showing increased demand triggering pod scaling, followed by CAST AI node scaling when pods are pending.](https://assets.ai.engineer/sites/aiecode2025/sha256/74040a75595a3789846b704655dff5ad858ba9636502c8e80c712ced18d2c838.webp)

Multi-Layer Autoscaling

Increased Demand
More Pods
(HPA scales)
Pending Pods
More Nodes
(CAST AI scales)
Stable Performance

CAST AI
Adds capacity when pods cannot be scheduled

Pod Autoscaling
Application workloads scale automatically.

Node Autoscaling
Infrastructure provisioning on demand.

Elastic Capacity
Application and infrastructure scale together.

### Kubernetes Event-driven Autoscaling

The team plans to migrate from HPA to KEDA, scaling the Hub from 10 to 500 pods using application metrics such as requests per second and event loop utilization.

[19:53](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1193s) [Diagram showing KEDA scaling the Hub deployment from 10 to 500 pods based on application metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/8cd9e05a38ae57546dffd3a8725f6d01ac666aafba788484ad592cb8c3b192e4.webp)

Kubernetes Event-driven Autoscaling

Users

Hub Deployment
10 → 500 Pods

KEDA
(RPS/Event Loop > target)

Horizontal Scaling

KEDA
Application-aware autoscaling

Scale on Demand
Scale from 10 to 500 pods

RPS & Event Loop
Scaling driven by real workload

### Thank You

Arek Borucki closes the talk with thanks.

[21:09](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1269s) [Thank-you slide naming Arek Borucki and his roles, with a Hugging Face mascot and illustrated cap.](https://assets.ai.engineer/sites/aiecode2025/sha256/266d207a1fccc24a712b678c72f85d92c1129af67c864c0fe3b10caa9b7aa3ad.webp)

Thank You!
Arek Borucki
ML Platform & Database Engineer @ Hugging Face
MongoDB in Action 8.0 Author

## Full transcript

[0:00](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=0s) [on hold music] Good afternoon, everyone. I have a question.

[0:19](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=19s) How many of you knows Hugging Face? Nice.

[0:27](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=27s) How many of you already use Hugging Face?

[0:33](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=33s) Amazing. Almost everyone. But I think we still have opportunity to grow our usage. My name is Arek Borucki. I work as machine learning platform and database engineer at Hugging Face.

[0:49](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=49s) Today, I would like to walk you through how Hugging Face scaled

[0:57](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=57s) infrastructure, and how we ended up serving three million models to developers around the world.

[1:09](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=69s) I would like to share architectural decisions we made, challenges we faced, and lessons we learned while scaling one of the fastest growing open source AI communities in the world.

[1:29](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=89s) I hope you will enjoy it, and let's get started.

[1:37](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=97s) Before I dive into technical details, let's talk about scale.

[1:44](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=104s) Today, Hugging Face serves more than fourteen million users, and this number is growing very fast, especially in the last couple of months.

[1:57](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=117s) We host three million public models, one million datasets,

[2:07](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=127s) ten-- fifty thousand organizations. And not only hobbyists or scientists.

[2:16](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=136s) More than thirty percent of Fortune five hundred use Hugging Face as a part of AI workflows.

[2:27](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=147s) Just to give you some perspective, three years ago, we had twenty thousand models. Today, three million.

[2:39](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=159s) It is around one hundred fifty x increase in just last couple of years.

[2:48](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=168s) And this growth is exactly why I'm here today talking about infrastructure decisions that keep the hub healthy at scale.

[3:03](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=183s) This is how fast the number of public models is growing on the hub. Every big release like Llama or DeepSeek generated thousands of new models on top,

[3:20](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=200s) and our infrastructure needs to handle that. And it is not only models,

[3:28](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=208s) also datasets. In twenty twenty-two, we had ten K.

[3:36](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=216s) In twenty twenty-four, one hundred K. Less than a year ago, we had five hundred K. Today, one million.

[3:49](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=229s) All this data must be stored, indexed, and also must be searchable. And that's the hardest part.

[4:02](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=242s) And this is also the reason why we had to rethink our search.

[4:09](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=249s) At twenty thousand models, any query is fast, even without an index. Trust me, no one would notice. At three million, same approach breaks.

[4:23](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=263s) Imagine what would you do if the hub search would be slow.

[4:30](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=270s) You would just leave and go somewhere else, and this is also what user are, are doing. They expect fast, instant results.

[4:39](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=279s) With fourteen million users, even one percent is a not small number. It is one hundred and forty thousand of people hitting slow search.

[4:56](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=296s) At scale, P99 is much more important than P50, and we are paying a lot of attention to P99.

[5:09](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=309s) And that's the reason why we invest in precomputed tokens,

[5:16](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=316s) denormalized, optimized-for-read collection in MongoDB, full-text search based on Apache Lucene, Kubernetes autoscaling,

[5:30](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=330s) and soon in database sharding. The next slides will show you how.

[5:41](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=341s) High-level architecture. When user interact with the Hugging Face Hub, his request flows from the front end to the Hub API.

[5:54](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=354s) The hub is running on Kubernetes. Currently, we are using Horizontal Pod Autoscaler.

[6:03](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=363s) During spikes, new pods scale up automatically to handle the load and scale back down when traffic drops.

[6:14](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=374s) This help us to keep the hub healthy without manual intervention. Next, the request

[6:26](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=386s) goes to MongoDB Atlas, which is source of truth for our metadata. And there is one point that sometimes surprise people. MongoDB does not store the models themselves.

[6:44](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=404s) It stores everything about the models. What does it mean in practice?

[6:52](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=412s) In MongoDB, we hold all the metadata, users, repositories, models, data sets, buckets, Spaces information, configuration data, billing data, access control, and more.

[7:17](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=437s) The actual models artifacts, tokenizer files, card assets, and configuration files are stored separately in cloud object storage, such as AWS S3.

[7:34](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=454s) This separation of concern let us scale metadata independently from binary storage and compute independently from both.

[7:46](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=466s) We can optimize each component individually for specific workload.

[7:55](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=475s) Now let's check how search works in details. For example, someone wants to search the model on the hub, and let's say that's Llama. So someone type Llama into Hugging Face search bar.

[8:12](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=492s) His request flows through the hub to an optimized read collection on MongoDB. And this is not our main repo collection when we keep all the data. It's a separate denormalized copy

[8:30](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=510s) only for reads and listings. The key information is on the left.

[8:40](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=520s) We tokenize model names on insert time, not at query time.

[8:47](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=527s) For example, someone wants to publish model meta-llama/llama3.1-8b.

[8:58](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=538s) We split the long model name into small tokens like meta, llama, three dot one, eight b, and we store them in an array in MongoDB document. Next, Atlas Search, which is using Apache Lucene under the hood, use autocomplete to find matching models

[9:23](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=563s) instantly. This is example of single document from our model collection. In this example, I'm using findOne method. I want to find model ID meta-llama/llama3.1-8b.

[9:43](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=583s) So that's the model from the previous slide.

[9:48](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=588s) And I'm projecting only searchToken array. And we see that all those precomputed tokens are part of this array. So we have meta, llama, three dot one, meta, llama, et cetera.

[10:03](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=603s) Next, there must be a query. In the past, we were using classical MongoDB find method on models collection with regex operator.

[10:20](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=620s) And this regex operator were searching in search tokens arrays models which are equal to llama.

[10:30](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=630s) And then we were set-- sorting results by trending score, which is calculated every five minutes. This is number of downloads and number of, of likes.

[10:43](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=643s) It is, as far as I remember, from the last seven days.

[10:48](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=648s) This solution was working well as long as data set was small.

[10:55](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=655s) Regex doesn't scale well, so when our data set started to grow very quickly, we started to having problems with latency. So we decided to switch to Atlas Search.

[11:11](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=671s) That's a feature which is using Apache Lucene under the hood.

[11:18](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=678s) So MongoDB doesn't provide in core MongoDB server full-text search. There is additional process, mongot.

[11:29](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=689s) Uh, this MongoDB process is a wrapper around Apache Lucene.

[11:37](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=697s) For end user, the-- users, this is transparent. You are just using unified MongoDB query API. And

[11:47](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=707s) when you use aggregation pi-pipeline together with $search operator, MongoDB will know that you would like to search Apache Lucene index.

[12:02](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=722s) Obviously, you need to put the name of this index, which is in this scenario, model search.

[12:11](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=731s) autocomplete model equal to Llama, path search tokens,

[12:20](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=740s) and we still sort results by trending score. And this solution is much more efficient,

[12:28](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=748s) and is so far scale well. So we don't have any more latency issues in our search bar.

[12:41](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=761s) First two results returned by previous query. First Meta Llama has trending score thirty-three, second one fourteen.

[12:56](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=776s) But Hugging Face Hub is not only search. We have hundreds of different services in Hugging Face which are utilizing, which are using MongoDB.

[13:08](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=788s) To handle million of queries, we use seven nodes MongoDB clusters, cluster.

[13:17](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=797s) With multiple machines, we can distribute queries across multiple nodes, so no single node become read bottleneck.

[13:29](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=809s) This is how it works. Application talk to the MongoDB cluster. All inserts, deletions or updates goes to single primary because only primary can handle them.

[13:43](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=823s) However, we are distributing reads across multiple machines.

[13:51](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=831s) We also have one analytic hidden node. What does it mean? This me- this node is invisible from application. Mong- MongoDB driver is not routing any queries to this hidden node.

[14:10](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=850s) This node is still replicating data from primary, but it's not interacting with, interacting with production traffic. We are connecting directly to this node, and we use him for any kind of

[14:28](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=868s) reporting traffic on re- or any kind of really heavy queries.

[14:35](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=875s) All secondaries continuously tail the oplog from primary, keeping cluster in sync.

[14:45](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=885s) Now let's have a look what actually is running on secondaries. First, all queries which doesn't require the latest data go to secondaries.

[14:58](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=898s) Only queries that must have strong consistency stay on primary, and we are paying a lots of attention to this. We are paying lots of attention to the queries which must run on primary.

[15:15](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=915s) Second, complex aggregations. Aggregations pipelines that scan large amount of data, sort, group, or transform the data

[15:31](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=931s) should not go on primary. They are heavy. Secondaries are better place for them.

[15:38](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=938s) Third, change streams. We react to data changes in real-time for several reaso- reasons. For example, cache invalidation, sync to different data store technologies like, for example, AWS Redshift, or for event-driven workloads.

[16:00](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=960s) Those kind of operations are also not very light, and they should stay if possible on secondaries. Fourth,

[16:10](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=970s) all ad-hoc queries, reporting queries, maybe some experimental queries go to hidden MongoDB replica set member which is isolated from production traffic.

[16:23](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=983s) The pattern is simple. Primary should focus on what only primary can do. Anything else can be pushed to different machines.

[16:38](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=998s) However, with fourteen million users, three million models and our grow, soon single MongoDB replica set will not be enough.

[16:52](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1012s) The next step is sharding. Sharding means scaling your database horizontally. Instead of putting full dataset on one repli- on one replica set cluster, we are going to cut data into pieces

[17:13](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1033s) and put each piece on separate shard. Each shard will have his own replication, primary and secondary. So we will keep replication just multiplied.

[17:26](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1046s) The key difference between replica set cluster and sharded cluster is replica set keep full dataset on each node. Sharded cluster keep only part of the data on each shard.

[17:39](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1059s) And then if you want to scale horizontally more, you are just adding more shards, and then MongoDB balancer will balance data across all those shards. There is also shard key which must be selected.

[17:53](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1073s) This is not trivial operation, but this talk is not about choosing shard key.

[18:00](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1080s) This way we are going to scale everything,

[18:04](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1084s) CPU, memory, storage, reads and writes Now let's have a look what is going on the hub level.

[18:19](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1099s) The hub is running on Kubernetes. S- currently, we are using Horizontal Pod Autoscaler. When CPU or memory threshold goes above target, Kubernetes adds new pod automatically to handle the spike and scale them back down when traffic drops.

[18:41](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1121s) Our deployment, hub deployment can scale from ten to five hundred pods, depends on, on the traffic. This is how we keep the hub healthy without manual interventions and without infrastructure overprovisioning.

[18:59](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1139s) So this is also cost-effective solution. However, what happens if Horizontal Pod Autoscaler want to add new pods but Kubernetes does not have free nodes anymore?

[19:17](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1157s) This is where second layer comes in. We are using CAST AI for Kubernetes node autoscaling. When pods are pending because there is no capacity and Kubernetes sched- scheduler is not able to schedule them, CAST AI is adding new nodes, and then scheduler is able to schedule those

[19:41](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1181s) pods. So we have two layers of scaling. First one is at deployment level, second one is at infrastructure level via CAST AI.

[19:54](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1194s) But we are going to migrate Horizontal Pod Autoscaler to KEDA, Kubernetes Event-Driven Autoscaling. The difference,

[20:07](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1207s) HPA scale only based on CPU and memory. KEDA scale on real application metrics like request per second or

[20:19](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1219s) event loop utilization. It means scaling is driven by actual workload, not by resource utilization only. For example, pod can have low CPU but high request queue, KEDA can see it, HPA not.

[20:43](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1243s) The best part of this architecture, you never have to think about it. When you pop-- When you push the model, search the hub, or download the model, it just works.

[20:58](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1258s) This is what scaling million models is really about, keeping the user experience simple no matter how complex it gets under the hood.

[21:10](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1270s) Thank you very much. It was pleasure for me to be a here today, and I wish nice day for all of you. [audience applauding]

[21:21](https://www.youtube.com/watch?v=lyL5QhgIOxc&t=1281s) Thank you. [outro music]
