# RAG and the MongoDB Document Model

Ben Flast, MongoDB

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/2Ey275TX4ZU-rag-mongodb-document-model) · [Video](https://www.youtube.com/watch?v=2Ey275TX4ZU)

## RAG and the MongoDB Document Model

A useful AI application needs more than similar documents: MongoDB combines vector retrieval with the operational records, conversation history, and cache entries that supply application context.

Familiarity with JSON objects, database queries, and LLM prompts is helpful; the article introduces the retrieval and vector-search concepts it uses.

### Why an LLM cannot know your bank balance

[1:21](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=81s)

Ask an unconnected LLM how much money is in your bank account, and it cannot answer. The missing ingredient is access to your private data. Ben Flast opens with this example because it exposes a basic application boundary: a model can generate language without possessing the information needed to make that language useful to a particular user.

![Slide showing the prompt “How much money is in my bank account?” and response “Uh what?” on a dark green background.](https://ai.engineer/api/talk-essay-media/54c93bf51fcd98832aed0f218abe63862ac957a71dd5c3df2b7c357e910f8dc1)

A bank-account question receives the response “Uh what?”

**Retrieval-augmented generation supplies context at prompting time.** A model has a training cutoff and lacks your private, personalized information. An application can retrieve company records, product information, or order history and include that material in its prompt. The aim is more relevant, consistent, and accurate answers; supplying context is not itself a guarantee of correctness. MongoDB’s role in this architecture is to make data already powering an application available to its AI features.

### The standard RAG request path

[2:47](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=167s)

The baseline architecture turns a user’s question into a retrieval request before asking the LLM to answer:

1. Receive the user’s question.
2. Send it to an embedding model, which produces a vector representation.
3. Use that vector to search [MongoDB Atlas Vector Search](https://www.mongodb.com/docs/vector-search/) for similar documents.
4. Combine the retrieved documents with the original question in the LLM’s input.
5. Return the generated answer to the user.

This is the familiar chatbot or copilot pattern: semantic retrieval selects context, and generation turns that context into a response.

### Documents hold richer application context

[3:26](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=206s)

More capable applications need context beyond a set of semantically similar passages. The [MongoDB document model](https://www.mongodb.com/docs/manual/core/document/) provides a way to keep richer application objects together. Developers work with JSON-like documents, stored internally as **BSON**, or binary JSON, containing fields such as a name and a nested profile.

Flast compares this with reconstructing a customer or contact object from relational tables. With a document, the application can store and retrieve the object in a form closer to the one it already uses.

| Representation | How the application obtains an object |
| --- | --- |
| Multiple relational tables | Assemble fields from related records |
| A document containing the object | Retrieve the stored object structure |

The benefit in this comparison is less reconstruction between the application’s data structures and its database representation.

Documents can represent tabular, key-value, geospatial, and graph-shaped data as well as conventional JSON objects. Flast associates this flexibility with greater efficiency and developer productivity, and points to **sharding** as MongoDB’s mechanism for horizontal scaling. These are architectural benefits he argues for, rather than measured comparisons presented in the talk.

![“Documents are universal” slide listing JSON documents, tabular, key-value, text, geospatial, graph, and time series data beside More Efficient, More Productive, and More Scalable columns.](https://ai.engineer/api/talk-essay-media/713f20d781e1d158c2a43119ca338f1804ebf756a7d97a6cafdba97eca3bc592)

Documents support multiple data models, with efficiency, productivity, and scalability benefits.

### Add embeddings to existing documents

[5:06](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=306s)

Vector retrieval adds another access path to those documents. Atlas uses **HNSW indexes** to support approximate nearest-neighbor search over stored vectors. In Flast’s example, a document already contains `symbol`, `quarter`, and `content`; adding `content_embedding` puts the vector alongside the data it describes.

The embedding can represent the whole document, a selected part of it, or external data mapped back to that record. At the time of the talk, Flast states a maximum of 4,096 dimensions per vector. That historical limit has since increased: the linked current Vector Search overview lists 8,192 dimensions. The mechanism remains the same—store the embedding as a field and index that field.

The vector index definition identifies the field’s type and path, its dimension count, and the similarity function used to compare vectors. For example, an embedding model producing 1,536 values could use this definition for the `content_embedding` field:

```json
{
  "fields": [
    {
      "type": "vector",
      "path": "content_embedding",
      "numDimensions": 1536,
      "similarity": "cosine"
    }
  ]
}
```

The dimension count must match the embedding’s length. The similarity function determines how the index judges closeness between the query vector and stored vectors.

### Query, tune, and filter retrieval

[6:18](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=378s)

Atlas builds the vector index and maintains it as database records change. That removes the need to manage a separate index-update path in the application, but should not be read as a promise that every write is immediately visible to search. The **`$vectorSearch` aggregation stage** names the index, supplies the query vector, and identifies the document field to search.

Two parameters control different parts of retrieval:

| Parameter | Role |
| --- | --- |
| `limit` | Maximum number of returned results |
| `numCandidates` | Number of nearest-neighbor candidates considered during search |

Increasing candidate exploration gives approximate search more opportunity to find relevant neighbors. Flast describes this using HNSW graph entry points; the [documented query terminology](https://www.mongodb.com/docs/languages/c/c-driver/v1.x/mongodb-vector-search/) is candidates considered, rather than a literal count of graph entry points.

A **pre-filter** restricts which documents are eligible during retrieval. This differs from retrieving a final set of neighbors and then discarding unwanted results. The displayed query uses `numCandidates: 100`, `limit: 10`, and a filter requiring `year` to be greater than `1995`.

The same structure can be expressed as a JavaScript function, with the query vector supplied by the application’s embedding step:

```javascript
function vectorSearchPipeline(indexName, queryVector) {
  return [
    {
      $vectorSearch: {
        index: indexName,
        queryVector,
        path: "content_embedding",
        numCandidates: 100,
        limit: 10,
        filter: { year: { $gt: 1995 } }
      }
    }
  ];
}
```

Here `content_embedding` retains the field used in the preceding document example; the filter adds a structured eligibility condition to semantic retrieval.

![Sample Query slide showing $vectorSearch with index, queryVector, path, numCandidates set to 100, limit set to 10, and a year filter greater than 1995.](https://ai.engineer/api/talk-essay-media/30ed2177e838d67aefd7b32f0bcde40dccacaec41a840a63b697dbb6d8cfea2f)

A sample vector search query includes candidate count, result limit, and a year filter.

### Scale search separately from transactions

[7:10](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=430s)

A unified database interface does not require every workload to share the same resource allocation. Flast’s transactional topology has a primary and two secondaries, providing the replication structure needed for durability and availability. Search can put a different kind of pressure on infrastructure than transactional reads and writes.

**[Search Nodes](https://www.mongodb.com/docs/vector-search/deployment/deployment-options/) separate search capacity from transactional capacity.** They hold the vector indexes and can scale independently of the infrastructure storing transactional data. An application keeps a consistent document model and query interface while allocating resources according to each workload’s needs. The separation is in deployment and scaling, without requiring the application to abandon its shared data model.

### Framework integrations beyond a vector store

[8:16](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=496s)

The integration ecosystem connects these database capabilities to application frameworks. Flast names LlamaIndex, LangChain, Microsoft Semantic Kernel, AWS Bedrock, and Haystack among [MongoDB’s AI integrations](https://www.mongodb.com/docs/atlas/ai-integrations/). Their supported primitives differ; the list does not imply feature parity.

For example, the [LangChain integration](https://www.mongodb.com/docs/atlas/ai-integrations/langchain/) includes both a vector-store abstraction and chat message history. These give an application different ways to use the same database: semantic retrieval supplies relevant material, while stored operational data supplies additional context for the prompt. This is the extension beyond the baseline RAG loop that the following examples illustrate.

### Reuse answers with a semantic cache

[9:11](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=551s)

A **semantic cache** can avoid a new model call when an existing answer matches the meaning of a request. MongoDB can serve as the cache backend for LangChain. In the flow shown here, the retriever first gathers additional context and combines it with the user’s prompt. That augmented prompt then goes to the cache for a semantic-similarity check.

The lookup creates two paths:

- **Cache hit:** Fetch the cached answer and return it without calling the LLM again.
- **Cache miss:** Send the prompt to the LLM and return its generated answer.

The saving comes from bypassing generation on a hit. Retrieval and caching can use the same MongoDB database, with LangChain connecting the pieces. Flast supplies no measured hit rate or quantified resource savings.

### Persist conversation history for continuity

[10:06](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=606s)

Conversation history solves a different problem: keeping a later request connected to what the user and assistant already discussed. LangChain’s chat message history abstraction can persist those messages in MongoDB, supporting the continuing conversation experience familiar from **ChatGPT**.

On a subsequent request, the application fetches prior messages, includes them alongside context from vector search, and sends the assembled prompt to the LLM. The returned answer continues that conversation. Semantic retrieval finds relevant data; message history preserves the interaction that gives the current question its meaning.

### 4149 combines retrieval with operational memory

[10:46](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=646s)

Flast presents [4149](https://www.mongodb.com/company/blog/building-ai-mongodb-boosting-productivity-efficiency-with-assistants-agents) as a customer example of an AI teammate beyond coding. In his account, the system listens to meetings, tracks activity, retrieves additional information, and surfaces context that could help a user complete a task, write an email, or schedule a project. This is a reported application use case, not a live demonstration of autonomous task completion.

The database holds more than embeddings. User records, meeting information, and conversation history sit alongside the data used for semantic search. That combination lets an application draw on both similarity-based retrieval and ordinary operational records. Flast’s broader point about agents follows from this combination: **a transactional database gives agents more ways to store and interact with application state than vector retrieval alone.**

### Deploying the application on Atlas

[12:04](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=724s)

The closing layer is operational. Atlas supplies security and privacy controls around the database. Flast also emphasizes availability and performance automation, without presenting an SLA or performance benchmark.

Flast states that Atlas is deployable in more than 100 regions across major cloud providers. He also highlights Search Nodes as a way to optimize resource placement, but Atlas’s overall regional footprint should not be treated as identical to dedicated Search Node availability; the deployment documentation specifies supported regions for that offering.

For an entry point, Flast advertises an Atlas free tier that includes vector search. The path through the talk builds from that initial retrieval capability: keep embeddings with application documents, add the context needed for useful responses, and separate search resources from transactional resources as the workload grows.

## Resources

- [MongoDB document structure](https://www.mongodb.com/docs/manual/core/document/) — Reference for MongoDB documents and their field-and-value structure.
- [MongoDB Vector Search](https://www.mongodb.com/docs/vector-search/) — Current overview of embedding storage and vector retrieval, including today's dimensionality limit.
- [Deploy and scale Search Nodes](https://www.mongodb.com/docs/vector-search/deployment/deployment-options/) — Deployment guidance for separating search resources from database nodes, with tier and regional requirements.
- [MongoDB's LangChain integration](https://www.mongodb.com/docs/atlas/ai-integrations/langchain/) — Implementation examples for vector retrieval, semantic caching and persistent chat history.
- [Semantic caching and the dedicated LangChain package](https://www.mongodb.com/company/blog/innovation/introducing-semantic-caching-dedicated-mongodb-lang-chain-package-gen-ai-apps) — Explains the package's vector store, conversation-history and semantic-cache components and links to examples.
- [4149.AI and operational context](https://www.mongodb.com/company/blog/building-ai-mongodb-boosting-productivity-efficiency-with-assistants-agents) — MongoDB's account of AI assistants, including how 4149.AI combines summaries, conversation history and embeddings.

## Slides

### Presenting sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=5s) [Microsoft presenting sponsor slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/b64f94bc5eebb077bf6965bd4a32bbfc5c29c465df8458520e86faa2a537a7da.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and platinum sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/f87f77f5af91389461d6d26c03755c47e50d3b3d450b5269da28936b3cd34608.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Agenda

The talk moves from RAG basics to MongoDB’s document model, vector search, AI integrations, and use cases.

[0:26](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=26s) [Agenda covering retrieval augmented generation, the document model, vector search, AI integrations and use cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/382f061a772736653b58d7784a1dfc082a8b8e33579df7a77c3f8eab8b7fea1c.webp)

Agenda

RETRIEVAL AUGMENTED GENERATION
THE DOCUMENT MODEL
SEARCHING THROUGH VECTORS
AI INTEGRATIONS
USE CASES

AIE
Microsoft
smol ai
aws
AI Engineer World’s Fair

### Bank account prompt and response

A vanilla LLM connected to nothing cannot know a user's bank balance. The slide illustrates that missing context with a confused response.

[1:26](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=86s) [A hypothetical bank account question receives the response “Uh what?”](https://assets.ai.engineer/sites/aiecode2025/sha256/9e99d04ba91c4a304671f56b4fdf3dc967f109f829d3eb622a90707591848201.webp)

prompt: “How much money is in my bank account?”
response: “Uh what?”

### Without context

An unconnected LLM cannot answer a question about your bank balance. The speaker introduces RAG as a way to provide context for useful LLM applications.

[1:34](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=94s) [Slide reads: “Without context, there’s only so much you can do.”](https://assets.ai.engineer/sites/aiecode2025/sha256/813e3a391f9a375a30022f7b28cd4e69c4dc1fc5a43a938135deb5a526195d35.webp)

Without context, there’s only so much you can do

### Retrieval Augmented Generation (RAG): making an LLM useful

RAG adds proprietary data, such as product information and order history, at prompt time to give a generic model context. The goal is more consistent and accurate responses.

[2:35](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=155s) [Generic model limitations, proprietary business data for context, and an AI-powered application with refined, consistent and accurate responses.](https://assets.ai.engineer/sites/aiecode2025/sha256/e142fdfd0503b11b799a81fae1c2b1a35e06912b552832292c3494d2680f79d1.webp)

Retrieval Augmented Generation (RAG): making an LLM useful

From Generic AI/ML models
General information
Training Cut-off
Missing private data
Unfocused / Hallucinations
Not personalized

How? With Data
Context and focus with proprietary business data
Company specific data
Product info
Order history
Secure User Data

A transformative AI-Powered application
Refined, consistent, and accurate responses

Interface text:
AIE
Microsoft
smol ai
aws
AI Engineer
World’s Fair

### Your standard RAG

The question is embedded and used to search a vector database. Similar documents join the original prompt in the LLM, which returns an answer to the user.

[2:45](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=165s) [A question is embedded and used to search a vector database; similar documents and the prompt are supplied to an LLM, which returns an answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/9bac62bddfb78a414b51a71edcc6b126fb0170c8fd742caea17d8101d5f269b5.webp)

Your standard RAG

User
Prompt
Question
Embedding model
Embedding
Semantic Search
Vector DB
Similar documents
LLM
Answer

Event and sponsor text:
AIE
AI Engineer World's Fair
Microsoft
smol ai
aws

### Standard RAG needs more context

The speaker argues that future applications will need more context than standard RAG provides, setting up the MongoDB document model.

[3:38](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=218s) [Slide stating that standard RAG is not enough and future applications need more context.](https://assets.ai.engineer/sites/aiecode2025/sha256/3dd0c824c682e274d96902aca49e6502fb3c5225113e29562f44ae2c884c08b9.webp)

But your standard RAG is not enough

the applications of tomorrow are hungry for context

Interface text:
AIE
Microsoft
smol ai
aws
AI Engineer
World’s Fair

### MongoDB Document Model

MongoDB stores documents as BSON, or binary JSON. The speaker introduces the document model as a way to provide more context for RAG applications.

[3:46](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=226s) [Title slide introducing the MongoDB document model.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc459438a243ba791c234af1d5472098777c15b722edd25004cee438e9d69c7c.webp)

MongoDB Document Model

Interface text:
AIE
Microsoft
smol ai
aws
AI Engineer World's Fair

### Objects mapped to relational tables

Application objects connect through an object-relational mapping layer to tables. The speaker describes stitching those tables together to serve objects such as customers and contacts.

[4:08](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=248s) [Customer, opportunity, contact and lead objects connect through a mapping layer to multiple tables.](https://assets.ai.engineer/sites/aiecode2025/sha256/27fa05de769a83548b2e5671d88b0e6d93b6f28aa7307b9794cf3de250eab3b2.webp)

With the Document model you can go from this...

Objects
Customer
Opportunity
Contact
Lead

Object Relational Mapping Layer

Tables
Customer Detail
ARR
Summary
Address
Open Activities
Contact Roles
Activity History
Opportunity Team
Name
Phone
Name
Phone

Interface text:
AIE
Microsoft
smol ai
aws

### Store objects directly

Each application object connects directly to a matching database object, illustrating how the document model stores objects without stitching together relational tables.

[4:27](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=267s) [Customer, opportunity, contact and lead objects each connect directly to a corresponding database object.](https://assets.ai.engineer/sites/aiecode2025/sha256/3061c904b2f4ff431471129574396552c3c62f057fd8546da80fa45788316a35.webp)

...to this

Store objects directly...

Objects
Customer | Opportunity | Contact | Lead

Database
Customer | Opportunity | Contact | Lead

AIE
Microsoft
smol ai
aws
AI Engineer World's Fair

### Now, with Vectors

The talk moves from MongoDB’s document model to adding vectors.

[5:05](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=305s) [Section slide introducing vectors.](https://assets.ai.engineer/sites/aiecode2025/sha256/79c940ef6907ef5af3767ebb4318278eb24346d67305250037252f5395fa777a.webp)

Now, with
Vectors ...

AIE
Microsoft
smol ai
aws
AI Engineer World’s Fair

### Approximate Nearest Neighbor via HNSW

MongoDB Atlas uses HNSW indexes for approximate nearest-neighbor vector search. The diagram follows connections from a sparse upper graph down through progressively denser layers.

[5:14](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=314s) [A layered graph illustrates approximate nearest neighbor retrieval using HNSW.](https://assets.ai.engineer/sites/aiecode2025/sha256/73509779b7c8a3bab34dccf38c38882f548d9719688f701ac42e66fc61b25f00.webp)

Approximate Nearest Neighbor via
HNSW for Fast Retrieval

Interface text:
AIE
Microsoft
smol ai
aws
AI Engineer World's Fair

### Vectors in Document

The document stores an embedding alongside its symbol, quarter and content. The embedding is an array of floats; more dimensions mean more floating-point values.

[5:28](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=328s) [A sample document sits beside an array of embedding values, with a note explaining that embeddings are stored as floats.](https://assets.ai.engineer/sites/aiecode2025/sha256/e25e04c588636ac2a2f64fc91ff7f7d544398d4a38893deffa65c8ffb554203d.webp)

Vectors in Document

_id: ObjectId('62f13a3f7aecb216')
symbol: "ABMD"
quarter: 4
year: 2021
date: 2021-04-29T20:10:40.000+00:00
content: "Operator: Ladies and gentlemen, thank you for standing by, and welcome..."

content_embeddings: Array
  0.03898080065846443
  -0.05879044905304909
  0.04323238879442215
  -0.021337900310754776
  -0.036346953362226486
  0.028689613565802574
  -0.03514527902007103
  -0.07414846867322922
  -0.00993054173886776
  0.007234036456793547
  -0.03197460621595383

Embeddings are stored as an array of floats
More dimensions = more floating points

Event branding: AIE; AI Engineer World's Fair; Microsoft; smol ai; aws

### Sample Query

The $vectorSearch stage runs approximate nearest-neighbor search using an index and query vector. This example considers 100 candidates, limits results to 10, and filters for years after 1995.

[5:51](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=351s) [A sample vector search query shows an index, query vector, path, candidate count, result limit and year filter.](https://assets.ai.engineer/sites/aiecode2025/sha256/78d91bf09b843e3335deea9856e7e8b470e7af9637c39b29a2ea98ee2c5a974c.webp)

Sample Query

{
    “$vectorSearch”: {
        "index": "content_embedding_index",
        “queryVector”: [ 0.03898080065846443, -0.05879044905304909, ... ],
        “path”: “content_embedding”,
        “numCandidates”: 100,
        “limit”: 10,
        “filter”: {
            “year”: {“$gt”: 1995}
        }
    }
[illegible: lower code obscured by presenter inset]

AIE
Microsoft
smol ai
aws
AI Engineer
World’s Fair

### Decoupled Scaling

Dedicated search nodes store vector indexes and scale independently from the Atlas nodes holding transactional data. The example cluster shows 2 to 32 search nodes.

[7:09](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=429s) [Diagram shows dedicated vector search nodes alongside Atlas database nodes in an example cluster.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7a5e47c5c40d2e2ccc6b43d9f2460f961f972d66cb77ccf3f22380396d6afe8.webp)

Decoupled Scaling

Search Nodes
Atlas Vector Search (mongot) is located on nodes dedicated to vector search, separate from Atlas nodes.

Cluster (Example)
P
S
S
Atlas Nodes
mongod
SN₂
SN₃
...
Search Nodes
(Min 2, max 32 nodes)
mongot

Event branding: AIE; Microsoft; smol ai; aws; AI Engineer World's Fair

### AI Integrations

The talk turns from scaling vector search to AI integrations.

[7:57](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=477s) [AI Integrations section title on a dark green background.](https://assets.ai.engineer/sites/aiecode2025/sha256/711549a1dff71004bec5104943a82ba5802179299339d5227468d39d1bcc9638.webp)

AI Integrations

AIE
AI Engineer World's Fair
Microsoft
smol ai
aws

### LLM Application Framework Integrations

MongoDB integrates with five LLM application frameworks. In LangChain, its integrations include a vector store and a chat message history abstraction.

[8:16](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=496s) [Framework overview naming LangChain, LlamaIndex, Microsoft Semantic Kernel, Amazon Bedrock and Haystack.](https://assets.ai.engineer/sites/aiecode2025/sha256/dd2f7105e436617110940f6647d4012e25b47724c97929ef0135ad19544c01f7.webp)

Integrated in the most popular
LLM Application frameworks

LangChain
LlamaIndex
Microsoft Semantic Kernel
Amazon Bedrock
haystack by deepset

AIE
Microsoft
smol ai
aws
AI Engineer World’s Fair

### RAG with Semantic Caching

Retrieved documents augment the prompt before a MongoDB semantic cache check. A hit returns a cached answer; a miss sends the prompt to the LLM, reducing repeat calls.

[9:06](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=546s) [RAG flow retrieves similar documents and checks a MongoDB semantic cache, returning a cached answer on a hit or calling an LLM on a miss.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4e709ec8d910afe6ec83444abf45337aef3417f3fe85189173777cf5323f2a8.webp)

RAG with Semantic Caching

Retriever
(MongoDB Vector Store)
Similar Documents
User
Question
Prompt
Semantic Cache
(MongoDB)
Hit
Yes
No
LLM
Answer
Cached Answer

Interface text:
AIE
Microsoft
smol ai
aws
AI Engineer
World’s Fair

### RAG Chain with Chat History

MongoDB chat history helps an LLM turn the user's question into a standalone question for retrieval. The final prompt combines the original question, chat history and similar documents to generate an answer.

[10:02](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=602s) [RAG flow uses MongoDB chat history to form a standalone question, retrieve similar documents and generate an answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/09c65723ba5f70f6ea9cb17bd059411e725e8610354b553ccce31208c79e843b.webp)

RAG chain with Chat History

User
Question
Prompt
LLM
Standalone Question
Chat Message History (MongoDB)
Retriever (MongoDB Vector Store)
Similar Documents
Prompt
LLM
Answer

Interface text:
AIE
Microsoft
smol ai
aws

### Available to Run Anywhere

MongoDB Atlas combines a document model and unified interface with transactional, search and analytical workloads. The slide says it deploys in 100+ regions across three cloud providers.

[12:03](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=723s) [Slide describes security, automation and deployment across cloud regions beside a layered platform diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d4ce2e4192e0676475ae3ab2db8f1778ead29c723380697e9ab985df771c95d.webp)

Available to run anywhere

Comprehensive controls to ensure data security and privacy

Continuous uptime with advanced automation to ensure performance

Deploy in 100+ regions across 3 cloud providers to get the best from each provider with no lock-in.

Diagram:
Document Model
Key value pairs
Graphs
Relationships
Geospatial
Time series
Objects
Unified interface
Transactional
Search
Analytical
Distributed applications
Mobile applications
Serverless applications
Secure
Resilient & Elastic
Global & Multi-cloud

Interface text:
AIE
Microsoft
smol ai
aws
AI Engineer World's Fair

### Thank you for attending the session

Scan the left QR code to try MongoDB Atlas free, or the right to learn about its AI capabilities. The speaker says the forever-free tier includes vector search.

[12:45](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=765s) [Closing slide with QR codes to try MongoDB Atlas free and learn about its AI capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/7083e1d59c545f6904f9f7f5d78ea2b7b7f54d057f6866cc2ef3de5f804959a0.webp)

Thank you for attending the session

Try MongoDB Atlas free

Learn more about our AI capabilities

## Full transcript

[0:00](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=0s) [upbeat music] Great to see everyone, and that was a great talk.

[0:16](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=16s) Uh, I'm very interested in this RAG group, um, though I have some, some concerns. Uh, so I- I'm here from MongoDB. I'm gonna be talking about RAG and specifically what's unique about doing RAG with the MongoDB document model and, and MongoDB Atlas, the, the platform.

[0:31](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=31s) I'm gonna start by talking a bit about just retrieval-augmented generation in general. I'm sure a lot of us are very familiar with it already, but I think it'll be good to cover some of the basic concepts.

[0:41](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=41s) Then I'm gonna talk about the document model, and so for those of you who are not so familiar with MongoDB, this, this will be kind of a, a nice little brief intro to, to what it means to use MongoDB and, and why we're a unique database.

[0:52](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=52s) Uh, then I'm gonna talk about vector search and a capability that exists inside of MongoDB now. Uh, and then I'll talk about some of our AI integrations, uh, and then some use cases to, to kind of help, you know, stimulate some ideas for all of you.

[1:04](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=64s) I'm gonna do all this in a quick 15. Um, so obviously, LLM's super exciting. It's been, been crazy, right, over the past year and a half. Um, but, but there has been a question, right, around kind of, you know, what all can they do, and, and when do you need to use RAG and when do you not?

[1:21](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=81s) And if you took kind of a vanilla LLM connected to nothing and asked it how much money is in your bank account, it wouldn't know. And I think we can all understand why that's the case and, you know, hope for the foreseeable future it continues to be the case. [laughs]

[1:35](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=95s) Um, but, uh, all that said, if we wanna make, you know, useful applications with these LLMs, then the reality is that without context, there's only so much you can do with the LLM, and so that's where RAG comes in.

[1:49](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=109s) So RAG stands for retrieval-augmented generation. Um, I'm sure this is old hat to most of you, but we're just gonna go through quickly. What this means is that you take a generic AI or ML model, um, that, you know, today we're generally talking about LLMs, but it has a training cutoff.

[2:06](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=126s) It, you know, it's, um, missing your private data. Maybe it hallucinates, maybe it doesn't. But overall, it's not personalized. And you take your data, right, and you augment it at the, the time of, of prompting to give it the context that it needs to answer the questions that you want it to for the use cases that you're

[2:24](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=144s) bringing it to bear for. And so that could be company-specific data, could be product info, could be order history, anything that you're storing inside of your application database that's already powering kind of your in-app experiences.

[2:36](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=156s) And with that, you get a transformative AI-powered application, right, that's gonna be refined and consistent and accurate in the responses that it gives, uh, when you're prompting the models.

[2:47](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=167s) So the typical RAG that you've all probably seen and, and in most cases have probably implemented is... will look something like this, right? So you have a user that enters a prompt.

[2:58](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=178s) The, the question that they, they enter will get sent, uh, to an embedding model. It'll be embedded. It'll then do a search, a semantic search on a vector database, in this case, MongoDB Atlas Vector Search, obviously, which will pull back similar documents.

[3:11](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=191s) So then those documents, along with the original prompt for the most cases, will go into the large language model, and that'll give an answer which goes back to the user.

[3:19](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=199s) And this is kind of what, you know, most people are doing for, for all, you know, chatbot and Copilot and other types of use cases, right?

[3:26](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=206s) But what's really interesting is that when you use MongoDB, you can go quite a bit farther than this and do things that are, you know, in many cases a bit different.

[3:34](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=214s) So, um, with, you know, RAG, the standard RAG is really not gonna be enough. The applications of tomorrow are gonna need more context, right? And that's where the MongoDB document model comes in.

[3:50](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=230s) So the document model is really just JSON, and it gets stored inside of MongoDB in something called BSON, which stands for binary JSON. But you have things like a name, a profile.

[4:00](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=240s) Um, you know, y- you can include whatever you want as long as it's JSON, uh, and that is actually what you store inside of your database and what you fetch from the database.

[4:09](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=249s) So with the document model, if you're comparing it to something that you would do in kind of a relational system where you have objects that your applications are interfacing with, right, like a customer object or a contact object, and you're, you know, stitching together different tables inside of a relational database.

[4:24](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=264s) Instead of having to kind of go through all of this pain and hassle, you get to go to something like this, right, where you just store the objects that your application is using directly inside of the database, and there's not all of this kind of reconfiguring and, and reconnoitering.

[4:37](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=277s) The way we look at this is that, you know, documents are universal, right? In many cases, they're kind of the superset of all, you know, data types that you might want to model.

[4:46](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=286s) And so you can have JSON, you can have tabular data, key-value stores, geospatial graph, it goes on. And what this translates to is, you know, it's more efficient in many places.

[4:57](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=297s) It is more productive for developers who are building systems, and in many cases it can be more scalable since MongoDB is just naturally very horizontally scalable through, uh, sharding.

[5:06](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=306s) So that's documents, and that's kind of just the, the core benefit of MongoDB. But now when we add on vectors is where things get, you know, really interesting, right?

[5:16](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=316s) So what we've done is we've added in HNSW indexes into MongoDB Atlas, which allows you to do approximate nearest neighbor vector search over data that's stored in your database.

[5:27](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=327s) And so what you do is you take your embeddings and you add them directly into the documents that you're already storing in your database. And so if you had this JSON that had symbol, quarter, and content fields, you could add a content_embedding field, which would just be the vectorization of, you know, either your entire document, some piece

[5:46](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=346s) of data in your document, or some piece of data that's living elsewhere that you're gonna map back to. And you can store all of that inside of your documents, and you can store vectors that are up to four thousand and ninety-six dimensions.

[5:57](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=357s) Once that's done, you add in an index definition. Uh, in this case, you know, the type of index is a vector search index. You would say the type of field that you're indexing is a vector.

[6:08](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=368s) You would say the... where the path is, where it's located, the number of dimensions, and the similarity function. So how do you want to determine the distance between the vectors that you're searching for and the ones that you're gonna find?

[6:18](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=378s) So once that's done, behind the scenes, the vector index is immediately built and kept in sync with data as it's updated inside of the database. And then you can use our $vectorSearch aggregation stage to, to go ahead and compute an approximate nearest neighbor search.

[6:31](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=391s) And so you have your index, you have the query vector, which is the vectorization of the data that you're searching for. Uh, you have the, the path where the data lives inside of your documents, and then you have numCandidates and limits.

[6:43](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=403s) Uh, and so the limit is how many results you wanna get back from this stage, and the numCandidates is how many entry points into your HNSW graph do you wanna make, uh, which is allows you to kind of tune the accuracy of your results.

[6:55](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=415s) Um, and then finally, uh, you can use a filter. And this filter is basically a pre-filter. So as we traverse this graph, we'll allow you to kind of fetch the, the documents, um, and, and filter out the ones that are less relevant for your specific query.

[7:10](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=430s) So that is vector search capability, but there's one other kind of core thing that's really important to just call out that we've also introduced alongside vector search, which is something called search nodes.

[7:20](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=440s) And this allows you to decouple your approach to scaling. So with a transactional database, right, you have a primary and two secondaries, and this allows you to have, you know, durability, high availability, and all of these guarantees that you would want for a transactional database.

[7:33](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=453s) But when you're adding search to it, the profile of resource usage may be a bit different. And so what we've done is we've added in a new type of node into the platform that allows you to store your vector indexes on those nodes and scale them independently from the infrastructure that's storing your transactional data.

[7:51](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=471s) And this allows you to really tune the amount of resources that you bring to bear to perfectly serve, uh, your workload. And so w- with that, we've really kind of transformed how Atlas can serve these vector search workloads by both giving you kind of a unified interface, uh, and a consistent use of the document model, yet at

[8:09](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=489s) the same time, kind of decoupling how you go about scaling for your workloads. And that's really kind of the, the true power of what we've done with vector search.

[8:16](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=496s) But along with this, we've also built several different AI integrations, and so we're integrated into some of the most popular AI frameworks, right? We have integrations inside of LlamaIndex, uh, LangChain, Microsoft Semantic Kernel, AWS Bedrock, and Haystack.

[8:30](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=510s) And in each of them, we support quite a different, uh, quite a few different, uh, primitives. And so we have, you know, just to name a few, inside of LangChain, we have vector store, but you can also have a chat message history, uh, you know, uh, abstraction inside of LangChain.

[8:45](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=525s) We have quite a few in LlamaIndex, then, you know, same for, for Haystack and, and AWS Bedrock. And so all of these allow you to do that next level of RAG that I was talking about at the very beginning, where you not only get to combine kind of just your typical vector search with RAG, but you also

[9:00](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=540s) get to now use kind of transactional data inside of your database to augment your prompts. And so to give you just, like, a couple examples of what that ends up looking like,

[9:11](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=551s) right, when you think about kind of more broad usage of memory for large language models, you might think about semantic caching. So this is a capability inside of LangChain, and you can use MongoDB as the back end of that semantic cache.

[9:22](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=562s) And now, right, when a user comes in and asks a question, we'll first kind of send it over to the retriever and, and figure out kind of what the, the question should look like, right?

[9:31](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=571s) Find the prompt plus the, uh, additional kind of augmented data, and then we'll send it to a semantic cache. And if that semantic cache says it's a hit based on a semantic similarity, then we'll just fetch the cached answer instead of having to hit the LLM again.

[9:45](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=585s) Uh, or if it's not a hit, we'll send it to the LLM and do the prompt and get the answer back to the user. And so in this way, you can use caching to kind of reduce the amount of calls that are being sent to your large language model.

[9:54](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=594s) And this is, you know, hugely powerful, just kind of reducing the amount of resources that you're using. And again, it can all be done using one database, uh, with LangChain, uh, in this case.

[10:06](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=606s) Separately though, right, we also now have, uh, chat history, right? And so with LangChain, if you wanted to build, uh, on top of MongoDB a experience that was maybe similar to, you know, ChatGPT, right, where you have the chat history and it's continuously fetching that data and putting it back into the prompt, uh, so that you can

[10:24](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=624s) kind of have continuity in the conversation that's happening with the large language model, well, you could use the chat message history abstraction inside of LangChain, and you could basically store the history of chats that are going through the platform.

[10:34](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=634s) And each time a prompt is sent back into the large language model, you could use the chat history, send it back through, include the vector search, and then, you know, send the, the prompt to the LLM and, and send the answer back.

[10:46](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=646s) And so just another way where you can really kind of evolve this. A, a cool startup that's using us right now to do a lot of these different things where they're taking advantage of kind of all of the flexibility of having a transactional database kind of built in with your vector search capability is a company called 4149.

[11:02](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=662s) I would, you know, recommend checking them out. Basically, they're building a, an AI teammate, and not like a coding teammate, but instead one that kind of, you know, listens to your meeting, tracks what you're doing, fetches additional information, and kind of prompts you, the user, with that information that you may need to kind of complete a task,

[11:19](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=679s) uh, you know, write an email, or kind of schedule a project. And they're using MongoDB not just to store their vector data and do, you know, semantic similarity search, but also to store data about their users, data about, um, you know, specific meetings, chat history meetings, all of this information that's not necessarily kind of your typical semantic

[11:38](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=698s) search type data use case, but instead it really benefits from having a single operational transactional database that also has vector search attached. And so that's where we're seeing, like, a lot of the excitement as we move into this, you know, world of agents and doing kind of complex, differentiated RAG.

[11:54](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=714s) Having a full transactional database really kind of opens up a, a new world of kind of storing and giving, you know, these agents more affordances to, to interact with the data.

[12:04](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=724s) And, you know, just one more thing to mention is that, you know, at the end of the day, all of this is built inside of MongoDB Atlas, which gives you comprehensive security controls, uh, and privacy.

[12:14](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=734s) It, you know, gives you kind of total uptime and automation to en- ensure that you have kind of optimal performance to serve your application. And finally, it's deployable in over 100-plus regions across all of the major cloud providers, including our Search Node offering that I mentioned earlier, that really allows you to optimize how you deploy these resources.

[12:33](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=753s) And so we're really thrilled to have this. Uh, just kind of a, a quick call-out. Uh, thanks all for, for coming to, to check out this, this talk. Um, i- if you wanna try MongoDB Atlas for free, we have a forever free tier where vector search is available, um, and you can also learn more of our AI

[12:48](https://www.youtube.com/watch?v=2Ey275TX4ZU&t=768s) capabilities using this other QR code as well. Uh, and with that, I'm done. [outro music]
