← All AI Engineer talks

AI Engineer Summit 2025

The Hidden Costs of Building Your Own RAG Stack — Ofer Mendelevitch, Vectara

Ofer Mendelevitch· Head of Developer Relations, Vectara15:14

Read the talk

The Hidden Costs of Building Your Own RAG Stack

Retrieving facts before generation is only the beginning: enterprise RAG also requires continuous evaluation, coordinated services, access controls and a team that can maintain the whole system.

From a talk by Ofer Mendelevitch

Answering questions over your own data

How do you get an LLM to answer a question using your PDFs or Notion workspace? Rather than send the question directly to the model, first retrieve relevant facts from those sources, then ask the model to answer using that evidence. That is the basic operation of retrieval-augmented generation, or RAG. Ofer Mendelevitch, Vectara’s head of developer relations, approaches it as a software and machine learning engineer who has worked with LLMs since GPT-2 in 2019.

The difficulty is turning that basic operation into an enterprise platform. A prototype establishes that retrieval and generation can work together. Production requires the surrounding system to keep working as its data, traffic and requirements grow. The hidden cost is operating the complete stack, not merely assembling its first working version.

Slide lists production, scaling, and keeping pace with innovation as challenges, beside an illustration of a burning computer.
Building a RAG platform is hard, expensive, and slow.
0:010:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Two flows, many components

The architecture starts with two flows. In the diagram, the blue arrow represents ingestion: moving information from documents, databases, Jira, Salesforce or other sources into a form the retrieval system can search. The steps are:

  1. Extract the content from its original format, such as a PDF.
  2. Chunk the text into smaller pieces.
  3. Embed each chunk with an embedding model.
  4. Store the resulting vectors in a vector database.

The original text must also be stored somewhere so it can later be supplied to the LLM. That storage is omitted from the simplified diagram, but it remains part of the implementation.

The green arrow represents the query flow. A query arrives, sometimes accompanied by filtering parameters. The system embeds the query text and searches the vector store for relevant material. Vector similarity may be only the first retrieval step: hybrid search combines vector and keyword search, while reranking can prioritize relevance, diversity or application-specific business rules.

Once retrieval produces the relevant chunks, the system sends them to the LLM together with the question and a grounding prompt. An optional hallucination detector then checks whether the generated answer used the supplied facts properly. Only after those steps does the response return to the user. What looks like one question-and-answer interaction is therefore a chain of extraction, storage, retrieval, ranking, generation and checking responsibilities.

1:031:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:03 · section reference included

Where a managed platform draws the boundary

Vectara’s proposed alternative is to put those components inside a managed boundary: RAG as a service. Mendelevitch describes three deployment options—SaaS, where customers send their data to Vectara; deployment in a customer’s VPC; and deployment on the customer’s own hardware.

The application then interacts with two external interfaces. An indexing API loads the data the system should use, and a query API connects the application’s user interface to the retrieval-and-generation flow. Under this arrangement, Vectara takes responsibility for managing the internal components. The application developer still supplies the data and builds the user-facing experience, but does not assemble each service inside the boundary.

Architecture diagram shows enterprise data entering an indexing API and user queries entering a query API, with embeddings, retrieval, generation, and hallucination detection inside a rectangular boundary.
Vectara's platform boundary encloses the RAG internals behind indexing and query APIs.
3:083:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:08 · section reference included

Quality continues after the first deployment

Building the stack yourself means choosing and integrating the vector database, retrieval engine, embedding model and LLM. From community and customer conversations, Mendelevitch groups the difficulties of moving that stack into production into seven categories: response quality and hallucinations, latency, scaling cost, security and compliance, vendor coordination, upkeep, and support for additional languages. These are the operating responsibilities behind the build-versus-buy decision.

The first responsibility is retrieving the right evidence. Parsing and chunking determine what information becomes available to search. Tables add another layer: extracting their text is not enough if the resulting representation loses the relationships needed to understand it. Different chunking strategies, table parsing and understanding, and hybrid search all require work beyond the initial proof of concept. Better generation cannot substitute for getting the relevant facts into the context.

Evaluation is an ongoing operation. A deployment can change when new data arrives, existing data is refreshed or a component is replaced. Response quality needs to be evaluated after those changes, not only before the first release, so that the team can determine whether results remain as good or improve.

Even with relevant evidence, hallucination prevention and correction remain difficult. Explainability adds a separate requirement: users need a way to inspect the facts behind an answer. A clickable citation in the interface depends on the system preserving a path back to the source. That provenance or audit trail must survive the processing flow; it cannot be reconstructed reliably by adding a link at the very end.

3:544:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:54 · section reference included

Latency and cost accumulate across the stack

A simple vector lookup can be fast, yet the complete query path can still be slow. Hybrid search and additional rankers introduce more work, and poorly coordinated components add delay. A slow component on the response path holds up the answer even if the other services perform well. Production latency therefore depends on the orchestration of the whole flow, not just the speed of the vector database.

The same expansion appears in cost. Embedding models, LLMs and the vector database are only the beginning; some stacks also use external services for PDF parsing or table understanding. Maintaining both quality and low latency requires capable components, each with its own expense.

Underneath the services sit GPU, CPU and storage costs. Mendelevitch contrasts a prototype containing ten documents with collections containing thousands, hundreds of thousands or millions: the example is a change in operating scale, not a measured cost curve. Token consumption adds another growing expense. The relevant budget is the cost of the entire system at its intended scale, rather than the price of one model call.

6:336:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:33 · section reference included

Document permissions must survive retrieval

Security begins with attribute-based access control. If a document is visible only to HR or the CEO, an employee must not gain access to its contents by asking the RAG application a question. The document system’s restrictions must carry into the RAG flow, including the evidence that can reach generation.

For a concrete implementation illustration, consider an employee asking, “What is our leave policy?” The candidate records include a general employee policy and an HR-only case note. This TypeScript filter keeps only records whose allowed roles intersect the user’s roles:

typescript

type Role = "employee" | "hr" | "ceo";
type Chunk = {
  id: string;
  text: string;
  allowedRoles: Role[];
};

function authorizedChunks(chunks: Chunk[], roles: Role[]): Chunk[] {
  return chunks.filter(chunk =>
    chunk.allowedRoles.some(role => roles.includes(role))
  );
}

const question = "What is our leave policy?";
const candidates: Chunk[] = [
  {
    id: "leave-policy",
    text: "Employees request leave through the staff portal.",
    allowedRoles: ["employee", "hr", "ceo"]
  },
  {
    id: "hr-case-note",
    text: "An individual employee's leave case is under review.",
    allowedRoles: ["hr"]
  }
];

const userRoles: Role[] = ["employee"];
const generationInput = {
  question,
  context: authorizedChunks(candidates, userRoles)
    .map(({ id, text }) => ({ id, text }))
};

Here, generationInput includes leave-policy and excludes hr-case-note. The role values must come from authenticated identity, and the document attributes must come from trusted policy metadata. The example illustrates the permission boundary; an enterprise policy may depend on more attributes than roles alone.

Sensitive information requires protection beyond access checks. Protected health information (PHI) and personally identifiable information (PII) need appropriate handling during ingestion and throughout the stack. Failures in those controls can impose substantial organizational costs, making security and compliance part of the architecture rather than a final deployment task.

8:038:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:03 · section reference included

Who owns the incident?

A stack assembled from separate providers can have one vendor for the vector database, another for the LLM, another for embeddings and another for keyword search. Each relationship introduces contracts, onboarding and integration work. Those costs exist even when every component works as intended.

When something fails, the boundaries become harder to manage. Which component caused the problem? Who can diagnose it? Who coordinates support across providers? Mendelevitch’s concern about vendor chaos is especially an incident-resolution concern: the application owner needs a working answer while individual vendors may each locate the fault somewhere else. Integration ownership includes resolving failures that cross service boundaries.

Pitfall #5: Vendor Chaos slide with four bullets covering vendor onboarding, costly integration, issue diagnosis, and contracts and negotiations.
Vendor chaos includes onboarding, integration, uncoordinated support, and multiple contracts.
9:029:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:02 · section reference included

Maintaining expertise and language coverage

The sixth cost is the team needed to keep the stack current and operational. RAG combines several specialties in a rapidly changing field:

AreaContinuing responsibility
LLMsGeneration expertise
Embedding modelsRepresentation expertise
Hybrid searchRetrieval expertise
Data engineeringIngestion, ETL, document parsing and metadata
DevOps and SREAvailability and continuous operation
SecurityProtection of the running system

LLM expertise and embedding-model expertise are distinct; knowing one does not automatically cover the other. The staffing challenge is recruiting and retaining the combination needed to support multiple generative AI use cases over time.

The seventh cost often appears after an English-only proof of concept. English enjoys broad component support, but an organization’s users may require other languages. Every component in the flow must support those languages. If one does not, replacing it can change the behavior and quality of the overall system. Language requirements therefore belong in the initial component selection, before the prototype’s choices become production dependencies.

10:0010:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:00 · section reference included

What the platform proposes to take over

These seven costs emerge as teams move from proofs of concept toward production. Vectara’s proposition is to take responsibility for extraction, encoding, indexing, the vector database and retrieval—including maintaining and upgrading those components. Application teams use APIs to upload data, index files and build query, chat or agentic RAG experiences.

Mendelevitch identifies three reasons enterprises choose that arrangement:

  • Accuracy: retrieval of the right facts, response quality and reduced hallucination.
  • Security: access controls and prevention of prompt attacks.
  • Explainability: observability into the system and citations throughout the flow.

These are Vectara’s stated platform priorities. They connect the managed-service boundary to the operational responsibilities described earlier.

11:3711:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:37 · section reference included

Checking groundedness and choosing a model

Hallucinations remain a barrier to adoption, particularly in regulated environments. Mendelevitch returns to that problem with an article from the preceding Thanksgiving, using its discussion of hallucinations to explain why retrieving evidence is not the end of the quality-control process. The generated answer still needs to be checked against that evidence.

Vectara integrates HHEM, the Hughes Hallucination Evaluation Model, into that flow. Its factual consistency score assesses whether generated text is supported by the supplied evidence; a higher score means greater consistency, not independent proof that the underlying evidence is true. Mendelevitch describes scoring as automatic with query responses. That historical behavior should be distinguished from later API documentation, which instructs callers to set enable_factual_consistency_score to true. The scoring claim concerns generated query answers, not indexing operations.

The open HHEM model is also available on Hugging Face. Mendelevitch reports over three million downloads since release. He also describes it as the most popular hallucination evaluator, although no comparative usage figures accompany that claim.

The Vectara Hallucination Leaderboard uses a commercial version of HHEM to compare LLM outputs. Its document-summarization task provides a view of factual consistency alongside a model’s other capabilities. Mendelevitch shows only the top of the leaderboard and reports that some models elsewhere in it have hallucination rates around 10% or 14%. Those are benchmark-specific examples: the talk does not identify the models or test conditions behind those percentages, and the current leaderboard is a different snapshot. They should not be read as general error rates for every use of those models.

13:0313:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:03 · section reference included

Model choice and deployment constraints

The model-selection implication is practical: consider a model’s tendency to produce unsupported claims alongside its other capabilities. Deployment constraints matter as well. Mendelevitch reiterates that Vectara can run in a customer’s VPC or on premises, describing those options as requirements for many customers rather than incidental hosting preferences.

Deployment slide with a world map, colored location pins, and labels for Cloud, Virtual Private Cloud, and On-premises.
Deployment options include cloud, virtual private cloud, and on-premises environments.

He closes with a QR-code invitation to try the platform and contact him. Mendelevitch offers a 30-day free trial, described in the talk as including all platform capabilities. For a team evaluating the managed approach, that invitation follows a specific question: which parts of ingestion, retrieval, generation and ongoing operation should the application team own?

14:4014:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:40 · section reference included

Resources

From the talk

Updates since the talk

Read the complete timestamped transcript
  1. 0:01

    Hi, everyone, and welcome to the hidden cost of building your own RAG stack. My name is Ofer. I run developer relations at Vectara, and I am a machine learning engineer by training and a software engineer, and I was fortunate enough to work on LLMs since GPT just turned two in 2019.

  2. 0:18

    Now, I'm gonna talk about hidden cost of RAG stack. I think we all know what it is, but just for those of you who may, may not be familiar, it's a way of using LLMs not directly like we used to send them a question and get an answer, but instead you wanna point it into your own data.

  3. 0:34

    Could be a bunch of PDF files, it could be data from Notion or wherever, wherever else. And instead of calling the LLM directly, you first retrieve the most relevant facts for the question out of, out of your data using a retrieval engine, and then the LLM is grounded in this data when it creates the response.

  4. 0:53

    Now, it turns out that building a RAG platform that is enterprise scale is much harder than most people seem to realize, and this is what I'm gonna talk about mostly in this talk.

  5. 1:03

    Before I go there, just to kind of align everybody with a couple of components and things that are included in the stack. So this is kinda my, uh, version of the diagram of how RAG kinda works.

  6. 1:13

    It has two main flows. The first one is kind of in the blue arrow, is the ingest. So you take data from wherever it is. It could be a set of documents, it could be a database, it could be a Jira or Salesforce or whatever it may be.

  7. 1:26

    And you... This is the data you want to be grounded on or, or retrieved, uh, from, and you ingest it into, uh, a bunch of, of steps that you have to take care of.

  8. 1:39

    You extract the data, if it's a PDF file or a document of some kind. You do something called chunking, which is breaking the text into smaller pieces, shown here.

  9. 1:47

    And then each chunk is then encoded using an embedding model stored in a, a vector database, uh. There's also other pieces here, like you have to store the text somewhere and things like this, but, um, so for simplicity, I'm gonna skip those.

  10. 1:59

    And then the green arrow is where a query comes in, sometimes with some filtering parameters, but the query text itself is again embedded into a, a vector. The vector is run against the vector store, and that those relevant facts are retrieved.

  11. 2:14

    In many cases, you want a stronger retrieval engine than just a, a vector search, so you kinda wanna do sometimes hybrid search, which combines vector and keyword search. You also might wanna do re-ranking, which, uh, can be relevance re-ranking or diversity re-ranking or some other business logic-related ranking you wanna do.

  12. 2:34

    At the end of this retrieval step, you get a, a set of relevant chunks or relevant facts, and you send them with a prompt to the LLM and say, "Please, you know, respond to this query given the facts."

  13. 2:47

    And then, and I'll talk more about hallucination detection, but sometimes what you wanna do also is detect the hallucination to see if the LLM actually did what it was supposed to do and u-use the facts in a proper way.

  14. 2:58

    And then once you have the response, you send it back to the user. So this is kind of, uh, a build of a RAG stack, and as you can see, there's a lot of different components here.

  15. 3:08

    And just a, a quick, a quick thing about Vectara, what we do is we essentially put all of this in a box. So think of it as a RAG-as-a-service, where all these components end up being part of what we do, what we provide.

  16. 3:21

    It could be in a SaaS format, uh, when you send all the data to us, or it could be on your own VPC. It could also be on-premise in your own hardware.

  17. 3:29

    But this stack is pre-built, and you don't have to build it or, or, or work on it. All you have to do is use these external APIs for indexing and for querying to first index the data you want the RAG stack to work on, and then to...

  18. 3:43

    When you have some user interface, uh, you can use the query API to run the query flow here. But all of the internals are hidden from you and saved from the trouble of managing those components.

  19. 3:54

    Okay. Now I'm gonna get to the, the meat of what I wanted to talk about, which is there is a very big difference between a sort of DIY RAG stack.

  20. 4:05

    When you build all these components yourself, you put in the vector database and the retrieval engine, and you figure out which embedding model to use and which LLM to use and all those pieces.

  21. 4:16

    A big gap between that and using a platform like Vectara, which is all taken care of, uh, for you. And as we talk to people in the community and, and, and customers, we, we figured there's about seven things I wanna mention today that are kinda pitfalls when you do build your DIY RAG, and I wanna talk about

  22. 4:34

    these seven. So irrelevant responses and hallucinations, which I'll talk about. Maintaining low latency, not getting high latency. Scaling up without making the cost be too high. You wanna, uh, make sure you don't have any issues with compliance and security, of course.

  23. 4:48

    There's something called vendor, vendor chaos. Sustainable keep and addition language. So let me talk about each of those separately. So the first one is the quality of responses or hallucinations.

  24. 5:00

    We know that hallucinations are still a, a really big deal, a really big problem, both in pure LLMs but also in RAG. And at scale, getting the right results first, getting the right facts is a problem, so you have to invest when you build your own infrastructure in, in parsing, in chunking.

  25. 5:19

    There's a lot of different chunking strategies in parsing of tables and table understanding in general. Hybrid search. All these things take a lot more work than kinda the initial PLC usually provides.

  26. 5:30

    And so you have to realize that when you really wanna get high-quality results, you have to invest in all of these components and get them to, to work properly.

  27. 5:38

    Last but not least, I wanna mention there's the evaluation, uh, of the, of the response quality, which is a complicated topic of its own, and it's not just evaluating on the first deployment.

  28. 5:49

    You kinda wanna continuously evaluate the results as you add more data, refresh the data, change components. You wanna make sure that your quality of the results remains the same or, or better.

  29. 5:59

    And then hallucinations, again, still remains a big issue, and implementing techniques to fight hallucinations or, uh, or even correct hallucinations can be quite complicated. The last thing I wanna mention here is just explainability.

  30. 6:12

    This means that when you have a result, you wanna usually show what the facts are. You have a path to-- for people to look at the facts, click on a link, and see where it came from.

  31. 6:23

    Again, that's something that you wanna integrate into the, your UI, but your system has to keep the path or the audit trail of where the results came from so that you can show them.

  32. 6:33

    Okay. The second one is pretty straightforward for any, you know, architect or systems en-engineer, right? You want the latency to be low, and a lot of times it gets much higher than you want, again, just because there's multiple components and not super orchestrated and, and harmonized.

  33. 6:49

    And so retrieval, right? Simple retrieval of vector store could be fast, but when you wanna do, let's say, hybrid search or some other rankers, you may get into a lot more trouble in, in latency, and it kinda starts to add up.

  34. 7:01

    And your system depends on the weakest link, right? So if one of your components has a, a latency issue, the net suffers. That becomes the latency for, for the whole flow of query.

  35. 7:11

    So we all know that users just want to get answers fast, and just wanted to highlight that this is not as trivial as, as you think when you come to kind of production grade.

  36. 7:23

    Okay. Pitfall number three, uh, scaling and cost. So you have a lot of components. You have embedding models, you have LLMs, a vector database, a bunch of other things.

  37. 7:32

    You have, uh, to sometimes, uh, use external, uh, services to enhance your PDF parsing or table understanding, and generally maintaining low latency and high quality just requires good components in your stack.

  38. 7:47

    And then on top of that, you have cost of GPUs and CPUs and storage. And so all these things really s-- uh, ramp up when you start going production scale with not just, you know, 10 documents, but thousands or, or hundreds of thousands or millions of documents.

  39. 8:03

    And, you know, the s- the token cost also starts becoming something that's, that's hard to, uh, to maintain. Number four, security compliance issues. So in security, I wanted to highlight first the idea of attribute-based access control.

  40. 8:18

    A lot of times you want to implement that as part of your RAG stack. This means that if some of the documents in your application, um, are only visible in your organization, let's say, to the HR or to the CEO, you wanna make sure that your RAG stack doesn't leak that information, or you wanna have some-- the

  41. 8:37

    same access controls, uh, implemented in your, in your RAG flow. Uh, similarly, if you're have, uh, sensitive information like PHI or PII, you wanna make sure that that's ingested in the right way and treated in the right way throughout the stack.

  42. 8:50

    A lot of these failures to do these kinda things and similar things in security and compliance can have a pretty large cost i-in any organization, so you wanna make sure that you do that right.

  43. 9:02

    Okay, this is one of my favorite ones to talk about actually, vendor chaos. What I mean by that is, if you remember the diagram I started with that has all the different components and the flows, there's quite a bit of components there.

  44. 9:14

    You have your vector database, you have your LLM, you have your embedding models, you have your, you know, maybe, uh, keyword search for hybrid search, et cetera. If you use different components for this, and it's not just one u- kind of harmonized stack, you start having multiple vendors.

  45. 9:29

    So you have multiple contracts, you have to on-onboard all of them, and the integration can become much more costly than you might imagine. The thing that frightens me the most with these kind of things when I think about kind of real enterprise scale is:

  46. 9:42

    How do you diagnose issues? You have multiple vendors, you can have things like, you know, finger-pointing coming up, where you really just wanna get, have the problem solved. So it could be here, it could be there.

  47. 9:52

    How do you diagnose? How do you know? And, and who gives you the support? How do you get the support coordinated? So that could be a, a, a pretty big nightmare.

  48. 10:00

    So something to be, uh, watchful for. All right. Um, I'm going to the last two here. So number six is unsustainable keep. And again, this means that RAG stack is a, a pretty unique, uh, set of skills and one that moves really quickly.

  49. 10:16

    So you have to have in your team, if you build it yourself, experts in LLMs, experts in embedding models, two different expertise areas. Experts in hybrid search in, if you implement that, et cetera.

  50. 10:27

    Everything you do, you have to have some specific expertise there. Not, not easy to get in, in, in many organizations. Uh, you have to have data engineering expertise to do all the data ingest and ETL flows and parsing the PDFs and parsing the documents and things like this.

  51. 10:41

    Metadata management. And then you have to have somebody running the system, making sure it's up and running all the time. So you gotta have your DevOps, SREs, et cetera, uh, security experts.

  52. 10:53

    So it's not always easy to find those resources and keep them as a separate team that can continuously maintain the system, uh, up and running to support all your GenAI use cases.

  53. 11:05

    Takes a lot of work. And then last is, you know, a lot of people start with English. It's the easiest one. Everything supports English, and you get a POC up and running.

  54. 11:15

    But when you go to non-English users, which may be a requirement for your organization, ends up that it's not as easy. You have to make sure that each of the components supports the language you need, and sometimes they don't, and then you have to change and then your quality suffers or something like this.

  55. 11:29

    So you gotta think about it up front if you wanna go to a, uh, production scale as well.

  56. 11:37

    All right, so those are kind of the seven pitfalls and things that we see in many transitions that we see, uh, of customers moving from kind of the initial POC and then starting to think about, uh, their production deployment.

  57. 11:49

    And again, this is why our approach is to have a platform. It's a plug-and-play RAG-as-a-service platform. And we've built all of these components, the extraction, the encoding, the indexing, the vector database, the retrieval.

  58. 12:02

    All of these pieces are part of the platform. You don't have to create them. You don't have to manage them. You don't have to upgrade them. We will do all that for you.

  59. 12:12

    And as a user of Vectara, what you do is you have APIs around it that you can use to upload your data or index your files. And then to, to run queries or chat or any other agentic RAG applications through that.

  60. 12:24

    So that's really the idea behind Vectara and why it's, uh, so, so powerful for, for big enterprises that have a lot of use cases. The three things that people usually choose us for are kind of aligned with some of the things I said earlier.

  61. 12:37

    Accuracy, uh, we focus a lot on hallucinations and reducing hallucinations, and I'll share in a minute what that means. A lot of emphasis on good retrieval and, and quality results and, and retrieval of the facts.

  62. 12:49

    Security mechanisms like access controls and prompt, uh, attack prevention, and also explainability to make sure that, uh, you have observability of what's going on in the system, and you have citations throughout the, the flow.

  63. 13:03

    I talked about hallucinations a little bit. I wanna highlight this a little bit more. So we've seen hallucinations a lot in twenty twenty-four and, and still this year it continues to be a significant issue for adoption.

  64. 13:15

    And this is an example of, uh, just an article from last, uh, Thanksgiving and a couple of quotes we got from, from that article. So, you know, especially in regulated environment, hallucinations can become a really, uh, big hurdle for, for adoption and for production deployment.

  65. 13:30

    One of the things we focus on at Vectara is called HHEM or hallucination, uh, evaluation model, detection model. We have this as part of our system. We've implemented this as part of the flow, and with every call to Vectara you get your hallucination score, which is really helpful because it can help you understand if the LLM did

  66. 13:49

    a good job or not and use that. This model was also open sourced on Hugging Face. You can see here it's got over three million downloads since it started, so it's very popular.

  67. 13:58

    It's by far the most popular evaluation for, for hallucinations out there. And then we also built this leaderboard, which might be interesting for everybody here listening. This, we take a bunch of the LLMs and evaluate how likely they are to hallucinate on, on some datasets.

  68. 14:15

    So we use HHEM, sort of the commercial version of it, to build this leaderboard, and this gives a lot of people sort of a good indicator on top of the core capabilities of the LLM, how much they might hallucinate or not.

  69. 14:28

    And I couldn't show all the leaderboard here 'cause there's a lot of models here, but some of them hallucinate at much higher rates. This is just the top. It could be, you know, sometimes like ten percent or even fourteen percent and things like this.

  70. 14:40

    So you gotta choose your LLM wisely. I mentioned that we also we're, we're, we're a RAG as a service, but of course you can also deploy this in your VPC and on premises, and that's important to know.

  71. 14:53

    Many of our customers require that. And that's it. Thank you so much for listening. I encourage you to try Vectara. I have a QR code here. You can check it out.

  72. 15:01

    We have a, a thirty-day free trial with all the full capabilities of the platform. And if you are interested to talk more, please feel free to reach out. Thank you so much.