RAG and the MongoDB Document Model

Talk transcript

1. 0:00

   [upbeat music] Great to see everyone, and that was a great talk.

2. 0:16

   Uh, I'm very interested in this RAG group, um, though I have some, some concerns. Uh, so I- I'm here from MongoDB. I'm gonna be talking about RAG and specifically what's unique about doing RAG with the MongoDB document model and, and MongoDB Atlas, the, the platform.

3. 0:31

   I'm gonna start by talking a bit about just retrieval-augmented generation in general. I'm sure a lot of us are very familiar with it already, but I think it'll be good to cover some of the basic concepts.

4. 0:41

   Then I'm gonna talk about the document model, and so for those of you who are not so familiar with MongoDB, this, this will be kind of a, a nice little brief intro to, to what it means to use MongoDB and, and why we're a unique database.

5. 0:52

   Uh, then I'm gonna talk about vector search and a capability that exists inside of MongoDB now. Uh, and then I'll talk about some of our AI integrations, uh, and then some use cases to, to kind of help, you know, stimulate some ideas for all of you.

6. 1:04

   I'm gonna do all this in a quick 15. Um, so obviously, LLM's super exciting. It's been, been crazy, right, over the past year and a half. Um, but, but there has been a question, right, around kind of, you know, what all can they do, and, and when do you need to use RAG and when do you not?

7. 1:21

   And if you took kind of a vanilla LLM connected to nothing and asked it how much money is in your bank account, it wouldn't know. And I think we can all understand why that's the case and, you know, hope for the foreseeable future it continues to be the case. [laughs]

8. 1:35

   Um, but, uh, all that said, if we wanna make, you know, useful applications with these LLMs, then the reality is that without context, there's only so much you can do with the LLM, and so that's where RAG comes in.

9. 1:49

   So RAG stands for retrieval-augmented generation. Um, I'm sure this is old hat to most of you, but we're just gonna go through quickly. What this means is that you take a generic AI or ML model, um, that, you know, today we're generally talking about LLMs, but it has a training cutoff.

10. 2:06

    It, you know, it's, um, missing your private data. Maybe it hallucinates, maybe it doesn't. But overall, it's not personalized. And you take your data, right, and you augment it at the, the time of, of prompting to give it the context that it needs to answer the questions that you want it to for the use cases that you're

11. 2:24

    bringing it to bear for. And so that could be company-specific data, could be product info, could be order history, anything that you're storing inside of your application database that's already powering kind of your in-app experiences.

12. 2:36

    And with that, you get a transformative AI-powered application, right, that's gonna be refined and consistent and accurate in the responses that it gives, uh, when you're prompting the models.

13. 2:47

    So the typical RAG that you've all probably seen and, and in most cases have probably implemented is... will look something like this, right? So you have a user that enters a prompt.

14. 2:58

    The, the question that they, they enter will get sent, uh, to an embedding model. It'll be embedded. It'll then do a search, a semantic search on a vector database, in this case, MongoDB Atlas Vector Search, obviously, which will pull back similar documents.

15. 3:11

    So then those documents, along with the original prompt for the most cases, will go into the large language model, and that'll give an answer which goes back to the user.

16. 3:19

    And this is kind of what, you know, most people are doing for, for all, you know, chatbot and Copilot and other types of use cases, right?

17. 3:26

    But what's really interesting is that when you use MongoDB, you can go quite a bit farther than this and do things that are, you know, in many cases a bit different.

18. 3:34

    So, um, with, you know, RAG, the standard RAG is really not gonna be enough. The applications of tomorrow are gonna need more context, right? And that's where the MongoDB document model comes in.

19. 3:50

    So the document model is really just JSON, and it gets stored inside of MongoDB in something called BSON, which stands for binary JSON. But you have things like a name, a profile.

20. 4:00

    Um, you know, y- you can include whatever you want as long as it's JSON, uh, and that is actually what you store inside of your database and what you fetch from the database.

21. 4:09

    So with the document model, if you're comparing it to something that you would do in kind of a relational system where you have objects that your applications are interfacing with, right, like a customer object or a contact object, and you're, you know, stitching together different tables inside of a relational database.

22. 4:24

    Instead of having to kind of go through all of this pain and hassle, you get to go to something like this, right, where you just store the objects that your application is using directly inside of the database, and there's not all of this kind of reconfiguring and, and reconnoitering.

23. 4:37

    The way we look at this is that, you know, documents are universal, right? In many cases, they're kind of the superset of all, you know, data types that you might want to model.

24. 4:46

    And so you can have JSON, you can have tabular data, key-value stores, geospatial graph, it goes on. And what this translates to is, you know, it's more efficient in many places.

25. 4:57

    It is more productive for developers who are building systems, and in many cases it can be more scalable since MongoDB is just naturally very horizontally scalable through, uh, sharding.

26. 5:06

    So that's documents, and that's kind of just the, the core benefit of MongoDB. But now when we add on vectors is where things get, you know, really interesting, right?

27. 5:16

    So what we've done is we've added in HNSW indexes into MongoDB Atlas, which allows you to do approximate nearest neighbor vector search over data that's stored in your database.

28. 5:27

    And so what you do is you take your embeddings and you add them directly into the documents that you're already storing in your database. And so if you had this JSON that had symbol, quarter, and content fields, you could add a content_embedding field, which would just be the vectorization of, you know, either your entire document, some piece

29. 5:46

    of data in your document, or some piece of data that's living elsewhere that you're gonna map back to. And you can store all of that inside of your documents, and you can store vectors that are up to four thousand and ninety-six dimensions.

30. 5:57

    Once that's done, you add in an index definition. Uh, in this case, you know, the type of index is a vector search index. You would say the type of field that you're indexing is a vector.

31. 6:08

    You would say the... where the path is, where it's located, the number of dimensions, and the similarity function. So how do you want to determine the distance between the vectors that you're searching for and the ones that you're gonna find?

32. 6:18

    So once that's done, behind the scenes, the vector index is immediately built and kept in sync with data as it's updated inside of the database. And then you can use our $vectorSearch aggregation stage to, to go ahead and compute an approximate nearest neighbor search.

33. 6:31

    And so you have your index, you have the query vector, which is the vectorization of the data that you're searching for. Uh, you have the, the path where the data lives inside of your documents, and then you have numCandidates and limits.

34. 6:43

    Uh, and so the limit is how many results you wanna get back from this stage, and the numCandidates is how many entry points into your HNSW graph do you wanna make, uh, which is allows you to kind of tune the accuracy of your results.

35. 6:55

    Um, and then finally, uh, you can use a filter. And this filter is basically a pre-filter. So as we traverse this graph, we'll allow you to kind of fetch the, the documents, um, and, and filter out the ones that are less relevant for your specific query.

36. 7:10

    So that is vector search capability, but there's one other kind of core thing that's really important to just call out that we've also introduced alongside vector search, which is something called search nodes.

37. 7:20

    And this allows you to decouple your approach to scaling. So with a transactional database, right, you have a primary and two secondaries, and this allows you to have, you know, durability, high availability, and all of these guarantees that you would want for a transactional database.

38. 7:33

    But when you're adding search to it, the profile of resource usage may be a bit different. And so what we've done is we've added in a new type of node into the platform that allows you to store your vector indexes on those nodes and scale them independently from the infrastructure that's storing your transactional data.

39. 7:51

    And this allows you to really tune the amount of resources that you bring to bear to perfectly serve, uh, your workload. And so w- with that, we've really kind of transformed how Atlas can serve these vector search workloads by both giving you kind of a unified interface, uh, and a consistent use of the document model, yet at

40. 8:09

    the same time, kind of decoupling how you go about scaling for your workloads. And that's really kind of the, the true power of what we've done with vector search.

41. 8:16

    But along with this, we've also built several different AI integrations, and so we're integrated into some of the most popular AI frameworks, right? We have integrations inside of LlamaIndex, uh, LangChain, Microsoft Semantic Kernel, AWS Bedrock, and Haystack.

42. 8:30

    And in each of them, we support quite a different, uh, quite a few different, uh, primitives. And so we have, you know, just to name a few, inside of LangChain, we have vector store, but you can also have a chat message history, uh, you know, uh, abstraction inside of LangChain.

43. 8:45

    We have quite a few in LlamaIndex, then, you know, same for, for Haystack and, and AWS Bedrock. And so all of these allow you to do that next level of RAG that I was talking about at the very beginning, where you not only get to combine kind of just your typical vector search with RAG, but you also

44. 9:00

    get to now use kind of transactional data inside of your database to augment your prompts. And so to give you just, like, a couple examples of what that ends up looking like,

45. 9:11

    right, when you think about kind of more broad usage of memory for large language models, you might think about semantic caching. So this is a capability inside of LangChain, and you can use MongoDB as the back end of that semantic cache.

46. 9:22

    And now, right, when a user comes in and asks a question, we'll first kind of send it over to the retriever and, and figure out kind of what the, the question should look like, right?

47. 9:31

    Find the prompt plus the, uh, additional kind of augmented data, and then we'll send it to a semantic cache. And if that semantic cache says it's a hit based on a semantic similarity, then we'll just fetch the cached answer instead of having to hit the LLM again.

48. 9:45

    Uh, or if it's not a hit, we'll send it to the LLM and do the prompt and get the answer back to the user. And so in this way, you can use caching to kind of reduce the amount of calls that are being sent to your large language model.

49. 9:54

    And this is, you know, hugely powerful, just kind of reducing the amount of resources that you're using. And again, it can all be done using one database, uh, with LangChain, uh, in this case.

50. 10:06

    Separately though, right, we also now have, uh, chat history, right? And so with LangChain, if you wanted to build, uh, on top of MongoDB a experience that was maybe similar to, you know, ChatGPT, right, where you have the chat history and it's continuously fetching that data and putting it back into the prompt, uh, so that you can

51. 10:24

    kind of have continuity in the conversation that's happening with the large language model, well, you could use the chat message history abstraction inside of LangChain, and you could basically store the history of chats that are going through the platform.

52. 10:34

    And each time a prompt is sent back into the large language model, you could use the chat history, send it back through, include the vector search, and then, you know, send the, the prompt to the LLM and, and send the answer back.

53. 10:46

    And so just another way where you can really kind of evolve this. A, a cool startup that's using us right now to do a lot of these different things where they're taking advantage of kind of all of the flexibility of having a transactional database kind of built in with your vector search capability is a company called 4149.

54. 11:02

    I would, you know, recommend checking them out. Basically, they're building a, an AI teammate, and not like a coding teammate, but instead one that kind of, you know, listens to your meeting, tracks what you're doing, fetches additional information, and kind of prompts you, the user, with that information that you may need to kind of complete a task,

55. 11:19

    uh, you know, write an email, or kind of schedule a project. And they're using MongoDB not just to store their vector data and do, you know, semantic similarity search, but also to store data about their users, data about, um, you know, specific meetings, chat history meetings, all of this information that's not necessarily kind of your typical semantic

56. 11:38

    search type data use case, but instead it really benefits from having a single operational transactional database that also has vector search attached. And so that's where we're seeing, like, a lot of the excitement as we move into this, you know, world of agents and doing kind of complex, differentiated RAG.

57. 11:54

    Having a full transactional database really kind of opens up a, a new world of kind of storing and giving, you know, these agents more affordances to, to interact with the data.

58. 12:04

    And, you know, just one more thing to mention is that, you know, at the end of the day, all of this is built inside of MongoDB Atlas, which gives you comprehensive security controls, uh, and privacy.

59. 12:14

    It, you know, gives you kind of total uptime and automation to en- ensure that you have kind of optimal performance to serve your application. And finally, it's deployable in over 100-plus regions across all of the major cloud providers, including our Search Node offering that I mentioned earlier, that really allows you to optimize how you deploy these resources.

60. 12:33

    And so we're really thrilled to have this. Uh, just kind of a, a quick call-out. Uh, thanks all for, for coming to, to check out this, this talk. Um, i- if you wanna try MongoDB Atlas for free, we have a forever free tier where vector search is available, um, and you can also learn more of our AI

61. 12:48

    capabilities using this other QR code as well. Uh, and with that, I'm done. [outro music]