RAG in 2025: State of the Art and the Road Forward

Talk transcript

1. 0:00

   [on-hold music] Thanks for coming.

2. 0:15

   Thanks for having me here. Um, I'm Tengyu Ma. I'm, uh, um, I was, uh, uh, the CEO and co-founder of Voyage AI. We just recently got acquired by MongoDB.

3. 0:25

   I'm also teaching at Stanford as well. So, um, this is about RAG, which is, uh, the main focus of Voyage AI, uh, the startup who is focusing on how to make retrieval better.

4. 0:35

   So, um, um, but I would just generally talk about, you know, RAG and, and we'll touch on some of the products we make as well very quickly. So I guess, uh, why we are, um, doing RAG or anything like that, right?

5. 0:45

   So I guess the main reason is that large language model or these days agent, um, which are, you know, using large language models as well. Uh, if they out-of-the-box, they cannot just, uh, uh, have, uh, uh, propriety information from any of the companies, right?

6. 0:59

   Because if they know anything about what MongoDB, for example, internally has, then the, the data was leaked. So that means that if you want to apply any of this to enterprise, uh, uh, then, um, uh, you need to ingest a lot of data from the, uh, propriety information.

7. 1:15

   So, um, and, uh, I'm gonna discuss, you know, why, uh, which kind of technologies to, to enable us to ingest the data. I guess there are, uh, a few options: RAG, fine-tuning and long context, which are all ways to ingest data, and I'll focus on RAG for the rest of the talk.

8. 1:30

   So I guess, you know, for this audience, probably most people knows these technologies, and they are all very simple on, uh, on high level. So for long context, it's just the, the most simple, you just, uh, dump all your documents, uh, uh, to, uh, a large language model's context, and maybe it's like one million tokens, maybe it's

9. 1:46

   one billion tokens. Um, so and then you have a query, and you just, uh, get a response. Uh, fine-tuning is like you first fine-tune a large language model, you update the parameters, and then you say, "I'm not gonna look at the documents anymore.

10. 1:59

    Uh, when the query comes, I just use the updated parameters to generate response." Uh, and RAG is, um, um, also pretty simple. So basically what happens is that on the fly, you use the query to retrieve some subset of the documents.

11. 2:12

    You use a retrieval or search method, uh, um, and then you get some relevant documents. You give this small set of relevant documents to the large language model, and then you generate response based on those contexts.

12. 2:23

    So this is one-- my one slide kind of like, uh, uh, summary of, you know, how I think about the differences between these technologies. You know, some of the, these are, uh, uh, inspired by some of the research at Stanford when we kind of started to, uh, uh, um, uh, uh, build Voyage.

13. 2:38

    You know, we kind of like believe in RAG, and one of the reason is that we don't believe that fine-tuning can work. And, and long context, I think, um, um, I also don't really, uh, believe that it can be cost efficient in the long run.

14. 2:49

    So, so basically, I think the way that I think about this is that, um, I, uh, try to make analogy to how humans, uh, are, uh, learning from or using the additional propriety information.

15. 3:01

    Um, um, so in some sense, long context is kind of like you scan the entire library to answer any single question, right? Every time you answer a question, you, you need to sc-- go through the entire library, which has like probably one billion tokens.

16. 3:14

    And fine-tuning is kind of like you read this library in your, in advance. You muscle memorize them. You try to internalize them in your brain, in your neurons, in your synapses, and you update your brain, basically rewire your brain, so that you really know all of those deeply.

17. 3:27

    Um, the challenge there is that, you know, it's very difficult and, um, somewhat unnecessary, um, um, because, you know, you cannot really memorize all the books in your-- in, in the, in the world and, uh, and, and memorizing a subset of them sometimes is kind of like, you know, which subset you wanna memorize is kind of, uh,

18. 3:44

    tricky as well. So and another thing is that, um, um, it makes, you know, forgetting the knowledge also tricky because you don't know which part of the knowledge you should forget and how to clearly forget all of them.

19. 3:55

    And also this makes the, the access, the, the, the data governance also kind of tricky because, you know, uh, maybe there are so many libraries, so many books in the library, and not everyone can access everything and, uh, how to organize those.

20. 4:07

    And on the other hand, RAG is very, very simple and modularized, uh, as I've shown. So, um, and very reliable and, and, you know, and, and also kind of fast and, and cheap.

21. 4:17

    So, um, and it's kind of like similar to how humans actually are using the libraries, right? You retrieve the most relevant, you know, book chapters or books or book chapters and then answer the question.

22. 4:27

    It's kind of a hierarchical way to store information, right? You don't really put all of the information in your brain. You put them in a library and then use them, uh, uh, when you need it.

23. 4:36

    So, um, that's why I believe in RAG, and, uh, and this is kind of how you implement the retrieval part. So basically, there is a breakdown of two components.

24. 4:44

    Actually, there are three, you know, if you are advanced. So, uh, so this, uh, embedding models which vectorize the, the documents and query, uh, into vectors, and the vectors are representations of the, uh, this kind of like representations of the content or the meanings of, uh, the documents and queries, and then use the vector database to store

25. 5:02

    the data and also search, uh, uh, with the k-nearest neighbor search in the vector space. And then you get the relevant documents, and then you can use large language model to generate answers.

26. 5:11

    So, um, we have seen significant improvements over the retrieval accuracy in the last, uh, two years. Um, when we started Voyage, you know, I think, uh, uh, OpenAI V three was not, uh, yet launched.

27. 5:23

    I think OpenAI V three was launched one-point-five years ago. Um, and in the last one-point-five years, you know, Voyage, you know, uh, uh, has made significant progress. You know, Cohere also made some progress.

28. 5:33

    So, uh, we can see that the new model has much better, uh, accuracy and with lower cost. Uh, um, and, uh, uh, generally, we have much better scaling law, right?

29. 5:41

    So the same number of parameters, the, the quality becomes better or the same quality, the, the, the parameters become smaller, and it becomes cheaper. Um, and all of these are through kind of like, uh, you know, optimizing the research stack, uh, the tuning stack, you know, as much as possible, you know, all the way from like data

30. 5:58

    curation, data selection, uh, uh, uh, architecture, loss functions, you know, evaluation, so on and so forth.

31. 6:04

    Uh, and we still, you know, believe that there's a big headroom here because, you know, n- right now you can see that in this plot, you know, we are averaging over about 100 data sets, and the accuracy is about 80%.

32. 6:14

    So that means that you still have, like, probably 20% of the improvement, um, uh, headroom. But that said, you know, just to be clear, uh, it's not like for every data set you only have 80% accuracy.

33. 6:24

    For probably half of the data sets, the accuracy is probably 90% or even 95%, and for some of the other ones, it's kind of 60, sometimes 20, sometimes 30, so that's why your average is 80%.

34. 6:35

    Um, so for-- so basically I'm saying, like, for some of the tasks that are common, uh, I think you can get a-already very high accuracy in the retrieval step.

35. 6:43

    Um, and another thing that, uh, uh, Voyage and other, uh, uh, uh, companies has, uh, uh, offered is this, uh, so-called matrix learning and also quantization while training. So basically these are two approaches to reduce the, the storage cost, uh, for the vectors.

36. 6:59

    So basically, matrix learning means that you, uh, make sure that, uh, even you have like a, a high-dimensional, uh, embedding, right? You can use a subset of the, uh, the coordinates.

37. 7:10

    Uh, it's, it's usually the first, uh... Let's say suppose you have, like, 2048 dimensional vectors, and the first, uh, 256 dimensional, uh, sub, uh, vector is still a reasonable embedding.

38. 7:22

    The accuracy wouldn't be as high as two, uh, 2048, but it will be almost the same, uh, maybe with, like, 1 or 2% loss. And quantization is kind of this, in a similar vein, so where you are-- even you lower your precision of the vectors, you still get pretty high performance.

39. 7:37

    Uh, and you can see the, uh, the trade-off on the right of the figure, uh, here. So basically, you can save, you know, 100X, you know, at least 10X without losing much.

40. 7:47

    If you save 100X in the storage cost, then you start to lose probably five to 10%. Um, but Voyage, you know, is doing a great job here because, you know, you can save, uh, 100X but still doing better than OpenAI.

41. 7:59

    That's just because the parental frontier is, uh, different. So, um,

42. 8:05

    um, and, and you can actually see better trade-off, you know, for domain-specific models, which I, I'm gonna discuss in a moment. Um, I have nine minutes here, so I will probably just quickly go through some of the, uh, uh, the techniques that you can use.

43. 8:17

    Um, so basically the next question is: how do you do better RAG, you know, without-- besides using better embedding models? Using better embedding models is probably one of the simplest way.

44. 8:25

    Um, so I'm just gonna go through it quickly. So one of them is to use hybrid search and rerankers. You can use, you know, lexical search and other kind of search, and then combine them with a reranker.

45. 8:36

    Uh, and Voyage provides a, a reranker as well. Um, and another one is you can enhance the queries and documents by the so-called query decomposition and, and document enrichment.

46. 8:46

    Um, so this is probably the most common one, maybe I spend one minute on it. So it's actually very simple. You just say if you have a query RAG, then you try to improve the query by, uh, making it longer using a large language model.

47. 8:56

    Uh, you can also decompose the longer query to small sub-queries so that you can have, like, a few different queries and search for different subset of documents. Um, and you can also enrich the document by adding additional meta co-- information in there.

48. 9:09

    You can add titles, you know, headers, you know, categories, authors, dates. Sometimes you trunk the document so that in the trunk you don't even have this information anymore, so that's why you have to add the global information into each of the trunks.

49. 9:21

    And some of this global information can be added by large language models. Anthropic wrote a blog post which does chieve-- achieve pretty good results. So basically they use large language models to generate additional context, um, that you can add to the trunks so that you can make the trunks, you know, more, uh, informative, and then the, uh,

50. 9:38

    it's easier to search, uh, through them. So, um, another one is you can use domain-specific embeddings where, um, you, um, c- you customize embeddings for certain kind of domains.

51. 9:48

    You know, in, uh, MongoDB or Voyage, we customize it for code, for example, and you can see that, you know, you get much better performance and also, uh, it's a better trade-off in terms of the, the storage cost and the, um, uh, and the, and the accuracy.

52. 10:01

    So basically, you don't lose as much if you compress the vectors even further. Um, so, so here we lose probably 5% by compressing, uh, for, like, uh, about 100X, um, but before we lose probably 10% or 15%.

53. 10:15

    Um, fine-tuning is another one. You can fine-tune the embedding models with your own data, um, and you can also use other, um, sometimes I call them tricks, on, on, on top of the embeddings, right?

54. 10:24

    So these are different type, type of ways to retrieve, um, uh, using additional information like graph, you know, iterative retrieval, so on and so forth. They're all based on embeddings, um, but, uh, you can, uh, use the embeddings in many different ways, uh, uh, as addit- an additional layer.

55. 10:38

    Um, so, um, I guess I'll use the next probably five minutes to discuss some of the, uh, uh, the, uh, my vision for how RAG will go in the future.

56. 10:48

    I do believe that RAG will be there forever because this is, as I sh- argued in the first slides, uh, the first set of slides, this is the kind of like very similar to how humans are, uh, using, uh, additional large amount of data.

57. 11:00

    You retrieve, you hierarchically select some subset, and then you use those, uh, to, uh, uh, to, to answer the questions, uh, or, or, or take some actions. And this is very efficient because you only use a small subset of the data.

58. 11:13

    Um, and, um, and as oppo-- uh, uh, um, regarding, uh, how, um, how RAG will evolve in from a technical point of view, uh, I'd like to draw some analogy, uh, um, uh, um, from, uh, how the, the AI generally is evolving.

59. 11:29

    So I think I was reflecting on when I was teaching at Stanford, you know, starting to teach at Stanford about seven years ago. Uh, I started to teach with Chris Ré on this machine learning course, and, uh, one of the slides literally have these seven steps on how do we build ML systems in enterprises.

60. 11:44

    Um, so this actually, this slide is still in the, the lecture notes. Uh, uh, we still teach them, but just with more kind of like, uh, uh, a- asterisks, uh, around it.

61. 11:53

    So you can see, like you, you need to go through, you know, many steps, you know, collecting data, you know, train test split, you know, define your loss functions, you know, build models and, and iterate and you repeat.

62. 12:03

    And then, uh, for the large language model world, it's kind of like this, right? You don't n- need to do any of this. You just take a large language model out of the box and just, uh, you know, you can deploy it in enterprise in most of the cases.

63. 12:14

    Of course, it's not gonna be perfect, but this is already better than in the old days you do all of these steps in the enterprise using all the enterprise data.

64. 12:21

    Just out of the box, you are doing already very, very well. Of course, you still have this issue that you cannot-- the out-of-the-box large language model cannot, um, uh, access propriety information, then you can use RAG, uh, for it.

65. 12:34

    So, but I think the point here is that before, uh, all of these steps have to be done by the kind of like the users or the enterprise or the customers in some sense.

66. 12:44

    Um, uh, and now, uh, you largely speaking just can take off-the-shelf components and connect them and build your, uh, AI applications very fast without going through these training steps.

67. 12:54

    The trainings still have to be done, but all of these steps still are done, uh, um, but they are done by OpenAI, Anthropic, or Voyage, MongoDB, uh, the providers of the models, but not the, uh, uh, the, the users, um, uh, the end users.

68. 13:08

    So, um, and I think for RAG, I, I would say probably the same kind of evolution should happen. Um, um, um, um, so right now what happens is that, um, we have the several different layers, where we have the computing infrastructure layer above the GPUs, you know, uh, or some of the KNs on the CPUs, and there's

69. 13:25

    also a model layer where, uh, you have the embedding models, the rerankers, the large language models. And then on top of all of this, um, people use a lot of, like I call it, tricks to make RAG, uh, uh, accuracy much better, right?

70. 13:37

    You can, uh, uh, use all kind of parsing strategies. You can use all kind of trunking strategies, you know, uh, uh, that the, the, the, um, you can do some recursive search, you can do some contextual trunks, graph RAGs, so on and so forth.

71. 13:51

    Right? That's what happens right now, and it's kind of necessary. These tricks are somewhat necessary because the embeddings and rerankers and large language models, none of them are perfect yet, right?

72. 14:00

    So, um, and but I do believe that in the future, I think this model layer will grow, uh, uh, and the tricks will be s- smaller. So it's gonna be fewer and fewer tricks, and the models can capture many of the, uh, the, the performance gain, um, by the tricks.

73. 14:15

    I think we have seen this in the large language model space as well, right? So like, um, um, two years ago, I think you need to do a lot of things on top of the GPT-3 to make your application work, and now even off the, uh, uh, out of the box, you can get the same performance as

74. 14:29

    before with all of the tricks. Um, of, of course, you still probably need some kind of tricks because some information are not, um, um, um, uh, uh, some information, um, the embedding models and rerankers don't have just, right?

75. 14:42

    So the, the, the general purpose models or off-the-shelf models don't have certain information, and then you can incorporate those, uh, into your tricks. For example, one thing that, uh, uh, is that, you know, the definition of the similarity metrics, uh, uh, could be something that you should customize in your prompt.

76. 14:59

    And, um, uh, in this one, you know, um, I think, uh, there are several things that we are developing towards this vision, right? So one of them is, uh, multi-model embedding.

77. 15:07

    This is to dramatically simplify the workflow so that you don't have to do many things, right? So these days, the multi-model embedding pr- provided by Voyage can just take in screenshots, right?

78. 15:16

    Before you take a PDF, you have to do the data extraction to turn them into image and text, and then probably do some embeddings for the image and the embedding for the text separately.

79. 15:25

    Um, and, and, and parsing this PDF is actually complex, you know. And for videos, you have to do, turn them into transcript, um, um, and then use the text embedding, so on and so forth.

80. 15:35

    Right? Right now, uh, right now we have the multi-model embedding which just takes in screenshot. You can deal with PT, P- PDF, you know, PPT, uh, uh, PowerP- you know, PowerPoint, you know, any of the other kind of slide stack in the same way.

81. 15:46

    Just take a screenshot and then use the multi-model embedding. We don't have the, uh, we can even do the, uh, same thing for video. Uh, not necessarily the perfect way, but, like, you just take screenshots of the frames, you know, consecutive frames, and you give it to the, uh, multi-model embedding, and you can, uh, turn them into

82. 16:01

    vectors, and you can search over, uh, on those documents or, or videos or slide stack. So, um,

83. 16:08

    s- and these are some performance metrics that we have evaluated. You know, we have tried kind of like... Oh, by the way, another one application is tables, right? Now you can just take a screenshot of the tables.

84. 16:17

    You don't have to think too much about what is the header, what is the row, so on and so forth. And we have done evaluations on many of these document screenshot, you know, table figures and, and also text only, and you can see that it's, uh, improving across the board.

85. 16:31

    So, um, the, the final one I would like to mention, which is something that we're gonna launch, uh, soon, is that this context aware and auto-chunking embedding. So, uh, right now what happens is that when you have a long document, you do have to chunk the data.

86. 16:46

    Um, uh, one of the reason is that the co- the context, uh, length of the embeddings is limited, and if you have like 100K tokens, you do have to chunk it into three or four chunks.

87. 16:55

    You know, even though Voyage AI has the probably, I think it, we have the longest context window, it's still like 32K. So that's one reason to chunk. And another reason to chunk is that sometimes it's a long, the long document, even you don't chunk.

88. 17:08

    Suppose you can have a way to, uh, uh, put all of them in a context window. Still, when you retrieve, you're gonna retrieve on a document level, then you retrieve a very, very long document, and then you should give this long document to large language model.

89. 17:20

    It's gonna be very, very expensive, right? If you give 100K, uh, tokens to la- large language model every time you answer any question, if you do some cost analysis, you'll find that that query is very, very expensive.

90. 17:30

    So that's why you do have to work on a smaller unit, so that you can cut the cost and, and be also more focused, right? So sometimes you give a long document to large language model, it misses some of, some of the context in the middle, and you have to use the retrieval to focus on a paragraph,

91. 17:44

    a page, so on and so forth. So, so that's what happens right now with the chunking and, uh, but all of these are done by the users. Uh, and w- our vision is that we're gonna do this for you, and also we're gonna, uh, get all the meta information about from other chunks.

92. 17:58

    So basically, um, uh, in a nutshell, the, the, the, the interface will be that you give us a long document, and we're gonna chunk it for you, and then also we return the chunks and also the vectors for each of the chunk.

93. 18:10

    And each of these vector is not only representing that chunk but also representing some of the global, uh, uh, meta information from other chunks. So it has all the details, uh, uh, of the corresponding chunk and also has some kind of like a cross-grain information from other chunks so you can get the best of the both worlds.

94. 18:27

    Um, and, uh, that's what I'm gonna launch, you know, soon. And another one is that we're gonna have some fine-tune API at some point, uh, to make you, uh, uh, uh, so that you can fine-tune, uh, with your own data.

95. 18:38

    Um, I guess, uh, it's exactly time. Thanks very much. [upbeat music]