AI Engineer World's Fair 2024
RAG and the MongoDB Document Model
Read the talk
RAG and the MongoDB Document Model
A useful AI application needs more than similar documents: MongoDB combines vector retrieval with the operational records, conversation history, and cache entries that supply application context.
From a talk by Ben Flast
Before you start: Familiarity with JSON objects, database queries, and LLM prompts is helpful; the article introduces the retrieval and vector-search concepts it uses.
Why an LLM cannot know your bank balance
Ask an unconnected LLM how much money is in your bank account, and it cannot answer. The missing ingredient is access to your private data. Ben Flast opens with this example because it exposes a basic application boundary: a model can generate language without possessing the information needed to make that language useful to a particular user.
Retrieval-augmented generation supplies context at prompting time. A model has a training cutoff and lacks your private, personalized information. An application can retrieve company records, product information, or order history and include that material in its prompt. The aim is more relevant, consistent, and accurate answers; supplying context is not itself a guarantee of correctness. MongoDB’s role in this architecture is to make data already powering an application available to its AI features.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The standard RAG request path
The baseline architecture turns a user’s question into a retrieval request before asking the LLM to answer:
- Receive the user’s question.
- Send it to an embedding model, which produces a vector representation.
- Use that vector to search MongoDB Atlas Vector Search for similar documents.
- Combine the retrieved documents with the original question in the LLM’s input.
- Return the generated answer to the user.
This is the familiar chatbot or copilot pattern: semantic retrieval selects context, and generation turns that context into a response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Documents hold richer application context
More capable applications need context beyond a set of semantically similar passages. The MongoDB document model provides a way to keep richer application objects together. Developers work with JSON-like documents, stored internally as BSON, or binary JSON, containing fields such as a name and a nested profile.
Flast compares this with reconstructing a customer or contact object from relational tables. With a document, the application can store and retrieve the object in a form closer to the one it already uses.
| Representation | How the application obtains an object |
|---|---|
| Multiple relational tables | Assemble fields from related records |
| A document containing the object | Retrieve the stored object structure |
The benefit in this comparison is less reconstruction between the application’s data structures and its database representation.
Documents can represent tabular, key-value, geospatial, and graph-shaped data as well as conventional JSON objects. Flast associates this flexibility with greater efficiency and developer productivity, and points to sharding as MongoDB’s mechanism for horizontal scaling. These are architectural benefits he argues for, rather than measured comparisons presented in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add embeddings to existing documents
Vector retrieval adds another access path to those documents. Atlas uses HNSW indexes to support approximate nearest-neighbor search over stored vectors. In Flast’s example, a document already contains symbol, quarter, and content; adding content_embedding puts the vector alongside the data it describes.
The embedding can represent the whole document, a selected part of it, or external data mapped back to that record. At the time of the talk, Flast states a maximum of 4,096 dimensions per vector. That historical limit has since increased: the linked current Vector Search overview lists 8,192 dimensions. The mechanism remains the same—store the embedding as a field and index that field.
The vector index definition identifies the field’s type and path, its dimension count, and the similarity function used to compare vectors. For example, an embedding model producing 1,536 values could use this definition for the content_embedding field:
json
{
"fields": [
{
"type": "vector",
"path": "content_embedding",
"numDimensions": 1536,
"similarity": "cosine"
}
]
}
The dimension count must match the embedding’s length. The similarity function determines how the index judges closeness between the query vector and stored vectors.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Query, tune, and filter retrieval
Atlas builds the vector index and maintains it as database records change. That removes the need to manage a separate index-update path in the application, but should not be read as a promise that every write is immediately visible to search. The $vectorSearch aggregation stage names the index, supplies the query vector, and identifies the document field to search.
Two parameters control different parts of retrieval:
| Parameter | Role |
|---|---|
limit | Maximum number of returned results |
numCandidates | Number of nearest-neighbor candidates considered during search |
Increasing candidate exploration gives approximate search more opportunity to find relevant neighbors. Flast describes this using HNSW graph entry points; the documented query terminology is candidates considered, rather than a literal count of graph entry points.
A pre-filter restricts which documents are eligible during retrieval. This differs from retrieving a final set of neighbors and then discarding unwanted results. The displayed query uses numCandidates: 100, limit: 10, and a filter requiring year to be greater than 1995.
The same structure can be expressed as a JavaScript function, with the query vector supplied by the application’s embedding step:
javascript
function vectorSearchPipeline(indexName, queryVector) {
return [
{
$vectorSearch: {
index: indexName,
queryVector,
path: "content_embedding",
numCandidates: 100,
limit: 10,
filter: { year: { $gt: 1995 } }
}
}
];
}
Here content_embedding retains the field used in the preceding document example; the filter adds a structured eligibility condition to semantic retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale search separately from transactions
A unified database interface does not require every workload to share the same resource allocation. Flast’s transactional topology has a primary and two secondaries, providing the replication structure needed for durability and availability. Search can put a different kind of pressure on infrastructure than transactional reads and writes.
Search Nodes separate search capacity from transactional capacity. They hold the vector indexes and can scale independently of the infrastructure storing transactional data. An application keeps a consistent document model and query interface while allocating resources according to each workload’s needs. The separation is in deployment and scaling, without requiring the application to abandon its shared data model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Framework integrations beyond a vector store
The integration ecosystem connects these database capabilities to application frameworks. Flast names LlamaIndex, LangChain, Microsoft Semantic Kernel, AWS Bedrock, and Haystack among MongoDB’s AI integrations. Their supported primitives differ; the list does not imply feature parity.
For example, the LangChain integration includes both a vector-store abstraction and chat message history. These give an application different ways to use the same database: semantic retrieval supplies relevant material, while stored operational data supplies additional context for the prompt. This is the extension beyond the baseline RAG loop that the following examples illustrate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse answers with a semantic cache
A semantic cache can avoid a new model call when an existing answer matches the meaning of a request. MongoDB can serve as the cache backend for LangChain. In the flow shown here, the retriever first gathers additional context and combines it with the user’s prompt. That augmented prompt then goes to the cache for a semantic-similarity check.
The lookup creates two paths:
- Cache hit: Fetch the cached answer and return it without calling the LLM again.
- Cache miss: Send the prompt to the LLM and return its generated answer.
The saving comes from bypassing generation on a hit. Retrieval and caching can use the same MongoDB database, with LangChain connecting the pieces. Flast supplies no measured hit rate or quantified resource savings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Persist conversation history for continuity
Conversation history solves a different problem: keeping a later request connected to what the user and assistant already discussed. LangChain’s chat message history abstraction can persist those messages in MongoDB, supporting the continuing conversation experience familiar from ChatGPT.
On a subsequent request, the application fetches prior messages, includes them alongside context from vector search, and sends the assembled prompt to the LLM. The returned answer continues that conversation. Semantic retrieval finds relevant data; message history preserves the interaction that gives the current question its meaning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
4149 combines retrieval with operational memory
Flast presents 4149 as a customer example of an AI teammate beyond coding. In his account, the system listens to meetings, tracks activity, retrieves additional information, and surfaces context that could help a user complete a task, write an email, or schedule a project. This is a reported application use case, not a live demonstration of autonomous task completion.
The database holds more than embeddings. User records, meeting information, and conversation history sit alongside the data used for semantic search. That combination lets an application draw on both similarity-based retrieval and ordinary operational records. Flast’s broader point about agents follows from this combination: a transactional database gives agents more ways to store and interact with application state than vector retrieval alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploying the application on Atlas
The closing layer is operational. Atlas supplies security and privacy controls around the database. Flast also emphasizes availability and performance automation, without presenting an SLA or performance benchmark.
Flast states that Atlas is deployable in more than 100 regions across major cloud providers. He also highlights Search Nodes as a way to optimize resource placement, but Atlas’s overall regional footprint should not be treated as identical to dedicated Search Node availability; the deployment documentation specifies supported regions for that offering.
For an entry point, Flast advertises an Atlas free tier that includes vector search. The path through the talk builds from that initial retrieval capability: keep embeddings with application documents, add the context needed for useful responses, and separate search resources from transactional resources as the workload grows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Reference for MongoDB documents and their field-and-value structure.
Further reading
Explains the package's vector store, conversation-history and semantic-cache components and links to examples.
MongoDB's account of AI assistants, including how 4149.AI combines summaries, conversation history and embeddings.
Updates since the talk
Current overview of embedding storage and vector retrieval, including today's dimensionality limit.
Deployment guidance for separating search resources from database nodes, with tier and regional requirements.
Implementation examples for vector retrieval, semantic caching and persistent chat history.
Read the complete timestamped transcript
- 0:00
[upbeat music] Great to see everyone, and that was a great talk.
- 0:16
Uh, I'm very interested in this RAG group, um, though I have some, some concerns. Uh, so I- I'm here from MongoDB. I'm gonna be talking about RAG and specifically what's unique about doing RAG with the MongoDB document model and, and MongoDB Atlas, the, the platform.
- 0:31
I'm gonna start by talking a bit about just retrieval-augmented generation in general. I'm sure a lot of us are very familiar with it already, but I think it'll be good to cover some of the basic concepts.
- 0:41
Then I'm gonna talk about the document model, and so for those of you who are not so familiar with MongoDB, this, this will be kind of a, a nice little brief intro to, to what it means to use MongoDB and, and why we're a unique database.
- 0:52
Uh, then I'm gonna talk about vector search and a capability that exists inside of MongoDB now. Uh, and then I'll talk about some of our AI integrations, uh, and then some use cases to, to kind of help, you know, stimulate some ideas for all of you.
- 1:04
I'm gonna do all this in a quick 15. Um, so obviously, LLM's super exciting. It's been, been crazy, right, over the past year and a half. Um, but, but there has been a question, right, around kind of, you know, what all can they do, and, and when do you need to use RAG and when do you not?
- 1:21
And if you took kind of a vanilla LLM connected to nothing and asked it how much money is in your bank account, it wouldn't know. And I think we can all understand why that's the case and, you know, hope for the foreseeable future it continues to be the case. [laughs]
- 1:35
Um, but, uh, all that said, if we wanna make, you know, useful applications with these LLMs, then the reality is that without context, there's only so much you can do with the LLM, and so that's where RAG comes in.
- 1:49
So RAG stands for retrieval-augmented generation. Um, I'm sure this is old hat to most of you, but we're just gonna go through quickly. What this means is that you take a generic AI or ML model, um, that, you know, today we're generally talking about LLMs, but it has a training cutoff.
- 2:06
It, you know, it's, um, missing your private data. Maybe it hallucinates, maybe it doesn't. But overall, it's not personalized. And you take your data, right, and you augment it at the, the time of, of prompting to give it the context that it needs to answer the questions that you want it to for the use cases that you're
- 2:24
bringing it to bear for. And so that could be company-specific data, could be product info, could be order history, anything that you're storing inside of your application database that's already powering kind of your in-app experiences.
- 2:36
And with that, you get a transformative AI-powered application, right, that's gonna be refined and consistent and accurate in the responses that it gives, uh, when you're prompting the models.
- 2:47
So the typical RAG that you've all probably seen and, and in most cases have probably implemented is... will look something like this, right? So you have a user that enters a prompt.
- 2:58
The, the question that they, they enter will get sent, uh, to an embedding model. It'll be embedded. It'll then do a search, a semantic search on a vector database, in this case, MongoDB Atlas Vector Search, obviously, which will pull back similar documents.
- 3:11
So then those documents, along with the original prompt for the most cases, will go into the large language model, and that'll give an answer which goes back to the user.
- 3:19
And this is kind of what, you know, most people are doing for, for all, you know, chatbot and Copilot and other types of use cases, right?
- 3:26
But what's really interesting is that when you use MongoDB, you can go quite a bit farther than this and do things that are, you know, in many cases a bit different.
- 3:34
So, um, with, you know, RAG, the standard RAG is really not gonna be enough. The applications of tomorrow are gonna need more context, right? And that's where the MongoDB document model comes in.
- 3:50
So the document model is really just JSON, and it gets stored inside of MongoDB in something called BSON, which stands for binary JSON. But you have things like a name, a profile.
- 4:00
Um, you know, y- you can include whatever you want as long as it's JSON, uh, and that is actually what you store inside of your database and what you fetch from the database.
- 4:09
So with the document model, if you're comparing it to something that you would do in kind of a relational system where you have objects that your applications are interfacing with, right, like a customer object or a contact object, and you're, you know, stitching together different tables inside of a relational database.
- 4:24
Instead of having to kind of go through all of this pain and hassle, you get to go to something like this, right, where you just store the objects that your application is using directly inside of the database, and there's not all of this kind of reconfiguring and, and reconnoitering.
- 4:37
The way we look at this is that, you know, documents are universal, right? In many cases, they're kind of the superset of all, you know, data types that you might want to model.
- 4:46
And so you can have JSON, you can have tabular data, key-value stores, geospatial graph, it goes on. And what this translates to is, you know, it's more efficient in many places.
- 4:57
It is more productive for developers who are building systems, and in many cases it can be more scalable since MongoDB is just naturally very horizontally scalable through, uh, sharding.
- 5:06
So that's documents, and that's kind of just the, the core benefit of MongoDB. But now when we add on vectors is where things get, you know, really interesting, right?
- 5:16
So what we've done is we've added in HNSW indexes into MongoDB Atlas, which allows you to do approximate nearest neighbor vector search over data that's stored in your database.
- 5:27
And so what you do is you take your embeddings and you add them directly into the documents that you're already storing in your database. And so if you had this JSON that had symbol, quarter, and content fields, you could add a content_embedding field, which would just be the vectorization of, you know, either your entire document, some piece
- 5:46
of data in your document, or some piece of data that's living elsewhere that you're gonna map back to. And you can store all of that inside of your documents, and you can store vectors that are up to four thousand and ninety-six dimensions.
- 5:57
Once that's done, you add in an index definition. Uh, in this case, you know, the type of index is a vector search index. You would say the type of field that you're indexing is a vector.
- 6:08
You would say the... where the path is, where it's located, the number of dimensions, and the similarity function. So how do you want to determine the distance between the vectors that you're searching for and the ones that you're gonna find?
- 6:18
So once that's done, behind the scenes, the vector index is immediately built and kept in sync with data as it's updated inside of the database. And then you can use our $vectorSearch aggregation stage to, to go ahead and compute an approximate nearest neighbor search.
- 6:31
And so you have your index, you have the query vector, which is the vectorization of the data that you're searching for. Uh, you have the, the path where the data lives inside of your documents, and then you have numCandidates and limits.
- 6:43
Uh, and so the limit is how many results you wanna get back from this stage, and the numCandidates is how many entry points into your HNSW graph do you wanna make, uh, which is allows you to kind of tune the accuracy of your results.
- 6:55
Um, and then finally, uh, you can use a filter. And this filter is basically a pre-filter. So as we traverse this graph, we'll allow you to kind of fetch the, the documents, um, and, and filter out the ones that are less relevant for your specific query.
- 7:10
So that is vector search capability, but there's one other kind of core thing that's really important to just call out that we've also introduced alongside vector search, which is something called search nodes.
- 7:20
And this allows you to decouple your approach to scaling. So with a transactional database, right, you have a primary and two secondaries, and this allows you to have, you know, durability, high availability, and all of these guarantees that you would want for a transactional database.
- 7:33
But when you're adding search to it, the profile of resource usage may be a bit different. And so what we've done is we've added in a new type of node into the platform that allows you to store your vector indexes on those nodes and scale them independently from the infrastructure that's storing your transactional data.
- 7:51
And this allows you to really tune the amount of resources that you bring to bear to perfectly serve, uh, your workload. And so w- with that, we've really kind of transformed how Atlas can serve these vector search workloads by both giving you kind of a unified interface, uh, and a consistent use of the document model, yet at
- 8:09
the same time, kind of decoupling how you go about scaling for your workloads. And that's really kind of the, the true power of what we've done with vector search.
- 8:16
But along with this, we've also built several different AI integrations, and so we're integrated into some of the most popular AI frameworks, right? We have integrations inside of LlamaIndex, uh, LangChain, Microsoft Semantic Kernel, AWS Bedrock, and Haystack.
- 8:30
And in each of them, we support quite a different, uh, quite a few different, uh, primitives. And so we have, you know, just to name a few, inside of LangChain, we have vector store, but you can also have a chat message history, uh, you know, uh, abstraction inside of LangChain.
- 8:45
We have quite a few in LlamaIndex, then, you know, same for, for Haystack and, and AWS Bedrock. And so all of these allow you to do that next level of RAG that I was talking about at the very beginning, where you not only get to combine kind of just your typical vector search with RAG, but you also
- 9:00
get to now use kind of transactional data inside of your database to augment your prompts. And so to give you just, like, a couple examples of what that ends up looking like,
- 9:11
right, when you think about kind of more broad usage of memory for large language models, you might think about semantic caching. So this is a capability inside of LangChain, and you can use MongoDB as the back end of that semantic cache.
- 9:22
And now, right, when a user comes in and asks a question, we'll first kind of send it over to the retriever and, and figure out kind of what the, the question should look like, right?
- 9:31
Find the prompt plus the, uh, additional kind of augmented data, and then we'll send it to a semantic cache. And if that semantic cache says it's a hit based on a semantic similarity, then we'll just fetch the cached answer instead of having to hit the LLM again.
- 9:45
Uh, or if it's not a hit, we'll send it to the LLM and do the prompt and get the answer back to the user. And so in this way, you can use caching to kind of reduce the amount of calls that are being sent to your large language model.
- 9:54
And this is, you know, hugely powerful, just kind of reducing the amount of resources that you're using. And again, it can all be done using one database, uh, with LangChain, uh, in this case.
- 10:06
Separately though, right, we also now have, uh, chat history, right? And so with LangChain, if you wanted to build, uh, on top of MongoDB a experience that was maybe similar to, you know, ChatGPT, right, where you have the chat history and it's continuously fetching that data and putting it back into the prompt, uh, so that you can
- 10:24
kind of have continuity in the conversation that's happening with the large language model, well, you could use the chat message history abstraction inside of LangChain, and you could basically store the history of chats that are going through the platform.
- 10:34
And each time a prompt is sent back into the large language model, you could use the chat history, send it back through, include the vector search, and then, you know, send the, the prompt to the LLM and, and send the answer back.
- 10:46
And so just another way where you can really kind of evolve this. A, a cool startup that's using us right now to do a lot of these different things where they're taking advantage of kind of all of the flexibility of having a transactional database kind of built in with your vector search capability is a company called 4149.
- 11:02
I would, you know, recommend checking them out. Basically, they're building a, an AI teammate, and not like a coding teammate, but instead one that kind of, you know, listens to your meeting, tracks what you're doing, fetches additional information, and kind of prompts you, the user, with that information that you may need to kind of complete a task,
- 11:19
uh, you know, write an email, or kind of schedule a project. And they're using MongoDB not just to store their vector data and do, you know, semantic similarity search, but also to store data about their users, data about, um, you know, specific meetings, chat history meetings, all of this information that's not necessarily kind of your typical semantic
- 11:38
search type data use case, but instead it really benefits from having a single operational transactional database that also has vector search attached. And so that's where we're seeing, like, a lot of the excitement as we move into this, you know, world of agents and doing kind of complex, differentiated RAG.
- 11:54
Having a full transactional database really kind of opens up a, a new world of kind of storing and giving, you know, these agents more affordances to, to interact with the data.
- 12:04
And, you know, just one more thing to mention is that, you know, at the end of the day, all of this is built inside of MongoDB Atlas, which gives you comprehensive security controls, uh, and privacy.
- 12:14
It, you know, gives you kind of total uptime and automation to en- ensure that you have kind of optimal performance to serve your application. And finally, it's deployable in over 100-plus regions across all of the major cloud providers, including our Search Node offering that I mentioned earlier, that really allows you to optimize how you deploy these resources.
- 12:33
And so we're really thrilled to have this. Uh, just kind of a, a quick call-out. Uh, thanks all for, for coming to, to check out this, this talk. Um, i- if you wanna try MongoDB Atlas for free, we have a forever free tier where vector search is available, um, and you can also learn more of our AI
- 12:48
capabilities using this other QR code as well. Uh, and with that, I'm done. [outro music]