The Hidden Costs of Building Your Own RAG Stack — Ofer Mendelevitch, Vectara

Talk transcript

1. 0:01

   Hi, everyone, and welcome to the hidden cost of building your own RAG stack. My name is Ofer. I run developer relations at Vectara, and I am a machine learning engineer by training and a software engineer, and I was fortunate enough to work on LLMs since GPT just turned two in 2019.

2. 0:18

   Now, I'm gonna talk about hidden cost of RAG stack. I think we all know what it is, but just for those of you who may, may not be familiar, it's a way of using LLMs not directly like we used to send them a question and get an answer, but instead you wanna point it into your own data.

3. 0:34

   Could be a bunch of PDF files, it could be data from Notion or wherever, wherever else. And instead of calling the LLM directly, you first retrieve the most relevant facts for the question out of, out of your data using a retrieval engine, and then the LLM is grounded in this data when it creates the response.

4. 0:53

   Now, it turns out that building a RAG platform that is enterprise scale is much harder than most people seem to realize, and this is what I'm gonna talk about mostly in this talk.

5. 1:03

   Before I go there, just to kind of align everybody with a couple of components and things that are included in the stack. So this is kinda my, uh, version of the diagram of how RAG kinda works.

6. 1:13

   It has two main flows. The first one is kind of in the blue arrow, is the ingest. So you take data from wherever it is. It could be a set of documents, it could be a database, it could be a Jira or Salesforce or whatever it may be.

7. 1:26

   And you... This is the data you want to be grounded on or, or retrieved, uh, from, and you ingest it into, uh, a bunch of, of steps that you have to take care of.

8. 1:39

   You extract the data, if it's a PDF file or a document of some kind. You do something called chunking, which is breaking the text into smaller pieces, shown here.

9. 1:47

   And then each chunk is then encoded using an embedding model stored in a, a vector database, uh. There's also other pieces here, like you have to store the text somewhere and things like this, but, um, so for simplicity, I'm gonna skip those.

10. 1:59

    And then the green arrow is where a query comes in, sometimes with some filtering parameters, but the query text itself is again embedded into a, a vector. The vector is run against the vector store, and that those relevant facts are retrieved.

11. 2:14

    In many cases, you want a stronger retrieval engine than just a, a vector search, so you kinda wanna do sometimes hybrid search, which combines vector and keyword search. You also might wanna do re-ranking, which, uh, can be relevance re-ranking or diversity re-ranking or some other business logic-related ranking you wanna do.

12. 2:34

    At the end of this retrieval step, you get a, a set of relevant chunks or relevant facts, and you send them with a prompt to the LLM and say, "Please, you know, respond to this query given the facts."

13. 2:47

    And then, and I'll talk more about hallucination detection, but sometimes what you wanna do also is detect the hallucination to see if the LLM actually did what it was supposed to do and u-use the facts in a proper way.

14. 2:58

    And then once you have the response, you send it back to the user. So this is kind of, uh, a build of a RAG stack, and as you can see, there's a lot of different components here.

15. 3:08

    And just a, a quick, a quick thing about Vectara, what we do is we essentially put all of this in a box. So think of it as a RAG-as-a-service, where all these components end up being part of what we do, what we provide.

16. 3:21

    It could be in a SaaS format, uh, when you send all the data to us, or it could be on your own VPC. It could also be on-premise in your own hardware.

17. 3:29

    But this stack is pre-built, and you don't have to build it or, or, or work on it. All you have to do is use these external APIs for indexing and for querying to first index the data you want the RAG stack to work on, and then to...

18. 3:43

    When you have some user interface, uh, you can use the query API to run the query flow here. But all of the internals are hidden from you and saved from the trouble of managing those components.

19. 3:54

    Okay. Now I'm gonna get to the, the meat of what I wanted to talk about, which is there is a very big difference between a sort of DIY RAG stack.

20. 4:05

    When you build all these components yourself, you put in the vector database and the retrieval engine, and you figure out which embedding model to use and which LLM to use and all those pieces.

21. 4:16

    A big gap between that and using a platform like Vectara, which is all taken care of, uh, for you. And as we talk to people in the community and, and, and customers, we, we figured there's about seven things I wanna mention today that are kinda pitfalls when you do build your DIY RAG, and I wanna talk about

22. 4:34

    these seven. So irrelevant responses and hallucinations, which I'll talk about. Maintaining low latency, not getting high latency. Scaling up without making the cost be too high. You wanna, uh, make sure you don't have any issues with compliance and security, of course.

23. 4:48

    There's something called vendor, vendor chaos. Sustainable keep and addition language. So let me talk about each of those separately. So the first one is the quality of responses or hallucinations.

24. 5:00

    We know that hallucinations are still a, a really big deal, a really big problem, both in pure LLMs but also in RAG. And at scale, getting the right results first, getting the right facts is a problem, so you have to invest when you build your own infrastructure in, in parsing, in chunking.

25. 5:19

    There's a lot of different chunking strategies in parsing of tables and table understanding in general. Hybrid search. All these things take a lot more work than kinda the initial PLC usually provides.

26. 5:30

    And so you have to realize that when you really wanna get high-quality results, you have to invest in all of these components and get them to, to work properly.

27. 5:38

    Last but not least, I wanna mention there's the evaluation, uh, of the, of the response quality, which is a complicated topic of its own, and it's not just evaluating on the first deployment.

28. 5:49

    You kinda wanna continuously evaluate the results as you add more data, refresh the data, change components. You wanna make sure that your quality of the results remains the same or, or better.

29. 5:59

    And then hallucinations, again, still remains a big issue, and implementing techniques to fight hallucinations or, uh, or even correct hallucinations can be quite complicated. The last thing I wanna mention here is just explainability.

30. 6:12

    This means that when you have a result, you wanna usually show what the facts are. You have a path to-- for people to look at the facts, click on a link, and see where it came from.

31. 6:23

    Again, that's something that you wanna integrate into the, your UI, but your system has to keep the path or the audit trail of where the results came from so that you can show them.

32. 6:33

    Okay. The second one is pretty straightforward for any, you know, architect or systems en-engineer, right? You want the latency to be low, and a lot of times it gets much higher than you want, again, just because there's multiple components and not super orchestrated and, and harmonized.

33. 6:49

    And so retrieval, right? Simple retrieval of vector store could be fast, but when you wanna do, let's say, hybrid search or some other rankers, you may get into a lot more trouble in, in latency, and it kinda starts to add up.

34. 7:01

    And your system depends on the weakest link, right? So if one of your components has a, a latency issue, the net suffers. That becomes the latency for, for the whole flow of query.

35. 7:11

    So we all know that users just want to get answers fast, and just wanted to highlight that this is not as trivial as, as you think when you come to kind of production grade.

36. 7:23

    Okay. Pitfall number three, uh, scaling and cost. So you have a lot of components. You have embedding models, you have LLMs, a vector database, a bunch of other things.

37. 7:32

    You have, uh, to sometimes, uh, use external, uh, services to enhance your PDF parsing or table understanding, and generally maintaining low latency and high quality just requires good components in your stack.

38. 7:47

    And then on top of that, you have cost of GPUs and CPUs and storage. And so all these things really s-- uh, ramp up when you start going production scale with not just, you know, 10 documents, but thousands or, or hundreds of thousands or millions of documents.

39. 8:03

    And, you know, the s- the token cost also starts becoming something that's, that's hard to, uh, to maintain. Number four, security compliance issues. So in security, I wanted to highlight first the idea of attribute-based access control.

40. 8:18

    A lot of times you want to implement that as part of your RAG stack. This means that if some of the documents in your application, um, are only visible in your organization, let's say, to the HR or to the CEO, you wanna make sure that your RAG stack doesn't leak that information, or you wanna have some-- the

41. 8:37

    same access controls, uh, implemented in your, in your RAG flow. Uh, similarly, if you're have, uh, sensitive information like PHI or PII, you wanna make sure that that's ingested in the right way and treated in the right way throughout the stack.

42. 8:50

    A lot of these failures to do these kinda things and similar things in security and compliance can have a pretty large cost i-in any organization, so you wanna make sure that you do that right.

43. 9:02

    Okay, this is one of my favorite ones to talk about actually, vendor chaos. What I mean by that is, if you remember the diagram I started with that has all the different components and the flows, there's quite a bit of components there.

44. 9:14

    You have your vector database, you have your LLM, you have your embedding models, you have your, you know, maybe, uh, keyword search for hybrid search, et cetera. If you use different components for this, and it's not just one u- kind of harmonized stack, you start having multiple vendors.

45. 9:29

    So you have multiple contracts, you have to on-onboard all of them, and the integration can become much more costly than you might imagine. The thing that frightens me the most with these kind of things when I think about kind of real enterprise scale is:

46. 9:42

    How do you diagnose issues? You have multiple vendors, you can have things like, you know, finger-pointing coming up, where you really just wanna get, have the problem solved. So it could be here, it could be there.

47. 9:52

    How do you diagnose? How do you know? And, and who gives you the support? How do you get the support coordinated? So that could be a, a, a pretty big nightmare.

48. 10:00

    So something to be, uh, watchful for. All right. Um, I'm going to the last two here. So number six is unsustainable keep. And again, this means that RAG stack is a, a pretty unique, uh, set of skills and one that moves really quickly.

49. 10:16

    So you have to have in your team, if you build it yourself, experts in LLMs, experts in embedding models, two different expertise areas. Experts in hybrid search in, if you implement that, et cetera.

50. 10:27

    Everything you do, you have to have some specific expertise there. Not, not easy to get in, in, in many organizations. Uh, you have to have data engineering expertise to do all the data ingest and ETL flows and parsing the PDFs and parsing the documents and things like this.

51. 10:41

    Metadata management. And then you have to have somebody running the system, making sure it's up and running all the time. So you gotta have your DevOps, SREs, et cetera, uh, security experts.

52. 10:53

    So it's not always easy to find those resources and keep them as a separate team that can continuously maintain the system, uh, up and running to support all your GenAI use cases.

53. 11:05

    Takes a lot of work. And then last is, you know, a lot of people start with English. It's the easiest one. Everything supports English, and you get a POC up and running.

54. 11:15

    But when you go to non-English users, which may be a requirement for your organization, ends up that it's not as easy. You have to make sure that each of the components supports the language you need, and sometimes they don't, and then you have to change and then your quality suffers or something like this.

55. 11:29

    So you gotta think about it up front if you wanna go to a, uh, production scale as well.

56. 11:37

    All right, so those are kind of the seven pitfalls and things that we see in many transitions that we see, uh, of customers moving from kind of the initial POC and then starting to think about, uh, their production deployment.

57. 11:49

    And again, this is why our approach is to have a platform. It's a plug-and-play RAG-as-a-service platform. And we've built all of these components, the extraction, the encoding, the indexing, the vector database, the retrieval.

58. 12:02

    All of these pieces are part of the platform. You don't have to create them. You don't have to manage them. You don't have to upgrade them. We will do all that for you.

59. 12:12

    And as a user of Vectara, what you do is you have APIs around it that you can use to upload your data or index your files. And then to, to run queries or chat or any other agentic RAG applications through that.

60. 12:24

    So that's really the idea behind Vectara and why it's, uh, so, so powerful for, for big enterprises that have a lot of use cases. The three things that people usually choose us for are kind of aligned with some of the things I said earlier.

61. 12:37

    Accuracy, uh, we focus a lot on hallucinations and reducing hallucinations, and I'll share in a minute what that means. A lot of emphasis on good retrieval and, and quality results and, and retrieval of the facts.

62. 12:49

    Security mechanisms like access controls and prompt, uh, attack prevention, and also explainability to make sure that, uh, you have observability of what's going on in the system, and you have citations throughout the, the flow.

63. 13:03

    I talked about hallucinations a little bit. I wanna highlight this a little bit more. So we've seen hallucinations a lot in twenty twenty-four and, and still this year it continues to be a significant issue for adoption.

64. 13:15

    And this is an example of, uh, just an article from last, uh, Thanksgiving and a couple of quotes we got from, from that article. So, you know, especially in regulated environment, hallucinations can become a really, uh, big hurdle for, for adoption and for production deployment.

65. 13:30

    One of the things we focus on at Vectara is called HHEM or hallucination, uh, evaluation model, detection model. We have this as part of our system. We've implemented this as part of the flow, and with every call to Vectara you get your hallucination score, which is really helpful because it can help you understand if the LLM did

66. 13:49

    a good job or not and use that. This model was also open sourced on Hugging Face. You can see here it's got over three million downloads since it started, so it's very popular.

67. 13:58

    It's by far the most popular evaluation for, for hallucinations out there. And then we also built this leaderboard, which might be interesting for everybody here listening. This, we take a bunch of the LLMs and evaluate how likely they are to hallucinate on, on some datasets.

68. 14:15

    So we use HHEM, sort of the commercial version of it, to build this leaderboard, and this gives a lot of people sort of a good indicator on top of the core capabilities of the LLM, how much they might hallucinate or not.

69. 14:28

    And I couldn't show all the leaderboard here 'cause there's a lot of models here, but some of them hallucinate at much higher rates. This is just the top. It could be, you know, sometimes like ten percent or even fourteen percent and things like this.

70. 14:40

    So you gotta choose your LLM wisely. I mentioned that we also we're, we're, we're a RAG as a service, but of course you can also deploy this in your VPC and on premises, and that's important to know.

71. 14:53

    Many of our customers require that. And that's it. Thank you so much for listening. I encourage you to try Vectara. I have a QR code here. You can check it out.

72. 15:01

    We have a, a thirty-day free trial with all the full capabilities of the platform. And if you are interested to talk more, please feel free to reach out. Thank you so much.