AI Engineer World's Fair 2024
RAG for VPs of AI
Read the talk
RAG for VPs of AI: The Data Work Between Prototype and Production
Reliable enterprise RAG depends on how documents become usable context, how pipelines are evaluated, and how retrieval fits into deployment, conversation and agent workflows.
From a talk by Jerry Liu
Before you start: Familiarity with LLM prompts and basic database concepts is helpful; the article explains the RAG pipeline as it goes.
From a directory of files to an answer
Point an LLM application at a directory of files and ask it questions. That is the familiar promise of retrieval-augmented generation, or RAG: make private information available to a model so it can answer from that information. Enterprise demand extends beyond question answering to document extraction, ongoing conversations and agents. Jerry Liu reports seeing more constrained agents in production than fully autonomous ones.
The query side needs an LLM for final synthesis, an embedding model, and storage suited to the information being retrieved. That storage need not be exclusively a vector database: document stores, graph stores and SQL databases also have roles. Before any of those components can produce useful answers, an ingestion pipeline must turn source material into accessible context. A PDF has to be parsed, divided into chunks, indexed and represented in appropriate storage forms. This is the data-processing layer that LlamaIndex focuses on, and its requirements differ from traditional ETL built for analytics.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why a working prototype is not enough
Liu describes building a roughly working RAG prototype with LlamaIndex tooling in ten minutes. The difficult transition comes afterward: document counts grow, layouts become more complex, additional sources enter the system, and the quality bar rises. Accuracy problems and a growing set of tuning parameters consume developer time. A proof of concept prepared for executives can stall before it delivers the value that justified the project.
Enterprise data silos compound the problem. Unstructured documents, structured records, semi-structured content and APIs all need to become usable by the application. A unified knowledge interface would let the model synthesize answers—and eventually take actions—across those sources. But a stronger model cannot compensate reliably for badly represented information. If ingestion loses the relationships in the original data, the resulting context can lead the application to hallucinate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build application expertise; manage the data plumbing
For an enterprise AI leader, Liu recommends investing somewhat more in building than in buying only out-of-the-box solutions. The reason is adaptability: procurement can address an immediate pain point, while developers can translate new models, techniques and workflows into applications specific to the business. That does not mean assigning developers endless ingestion troubleshooting. The tooling should reduce setup and maintenance work while improving response quality and supporting more sources.
The product split follows that division of responsibility. The open-source framework handles application orchestration; LlamaCloud manages the data infrastructure behind it.
| Layer | Responsibility |
|---|---|
| LlamaIndex framework | Retrieval, prompting, agent reasoning and tool use |
| LlamaCloud | Source integration, data processing and managed ingestion pipelines |
After a period focused on productionizing RAG, Liu expects more agentic use cases over the following six months. LlamaCloud initially targets unstructured sources, including complex PDFs, PowerPoints and spreadsheets, so developers can spend more time on custom retrieval and application logic. The overview includes upcoming capabilities as well as available ones.
LlamaParse, a component of LlamaCloud, addresses the document-quality problem directly. Financial reports and presentations can contain messy text layouts, tables, images and diagrams; parsing must preserve enough of their structure for an LLM to interpret them. Liu reports half a million monthly client SDK downloads and tens of millions of pages processed within a few months of LlamaParse’s release. These are reported adoption figures, not accuracy measurements. At the time of the session, LlamaCloud had a general waitlist for managed processing and indexing. The broader application direction was from question answering toward actions and automated decision-making for internal and external teams.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where sensitive documents go
Uploading sensitive enterprise documents raises an immediate deployment question. Liu describes two options in the session: a cloud service and a VPC deployment, with AWS and Azure supported and GCP coming later. He directs customers needing deployment details to the enterprise contact form. He also describes LlamaCloud as an orchestration layer that intentionally avoids storing customer data and integrates with existing storage systems. That statement describes the intended architecture; it does not establish a detailed zero-retention policy. These availability statements describe the offering at the time of the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What an ETL practitioner needs to learn
An audience member asks what additional skills an experienced ETL practitioner needs to build these pipelines. Beyond SQL or dbt transformations, the work introduces a sequence of decisions whose quality depends on the eventual answer:
- Parse the document. Use LlamaParse or another parser to produce text, or potentially a multimodal representation. Errors here propagate into later stages.
- Choose meaningful chunks. Liu’s naive example splits every 1,024 tokens. Such boundaries can cut a table in half or separate a section that spans pages.
- Preserve relationships and metadata. Join related content where necessary and attach metadata that helps the application interpret and retrieve it.
- Index the representation. Use a vector, graph or document store appropriate to the retrieval strategy.
The transformations are different from analytics ETL, but so is the feedback loop. A chunk size cannot be judged solely by inspecting the transformation locally. An evaluation dataset and end-to-end tests are needed to determine whether the resulting application answers well.
Audio introduces another representation choice. For an audience member building semantic chunking and paragraphing over transcripts, Liu suggests the simplest route: convert audio to text, then ingest that text. He tentatively recalls audio loaders in the framework. Representing audio itself as a chunk and feeding it directly to a multimodal model is the future alternative he sketches, rather than a capability he considers ready in this session.
The same end-to-end logic applies when choosing a parser. Asked for public datasets demonstrating LlamaParse’s superiority, Liu says a general benchmark is still being developed. The practical comparison is a customer bake-off: build RAG pipelines using LlamaParse and alternative parsers, then compare them on the enterprise’s own data. He describes notebooks supporting that comparison, but supplies no general superiority score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Support is part of the production contract
A LlamaParse user reports failures on a client project two weeks earlier. Support requested job IDs, then stopped responding, leaving the user asking how to obtain help over a weekend. Liu apologizes and acknowledges a cluster of failures. He says the company had fifteen people, with founding engineers often handling support, and promises a more streamlined process. The operational distinction is between the less formal support described for self-serve APIs and dedicated SLAs offered through enterprise plans; no specific response-time commitment is given in the session.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the relationships inside a table
An audience team building hallucination detection and evaluation over thousands of PDFs asks how to combine OCR with other PDF processing. Should LlamaParse handle the documents, or should they build a custom system on LlamaIndex? Liu starts with the minimum requirement: the parser must faithfully represent the text and its spatial relationships. Extra outputs such as bounding boxes do not compensate for an extraction that scrambles the content.
Tables expose the mechanism. A formatted table keeps labels, headers and values associated, giving the LLM a readable structure. A naive extraction can collapse the same text and numbers into a sequence that no longer makes those associations clear. Liu presents spatially aligned table output as a LlamaParse strength and identifies collapsed structure as a source of downstream hallucinations.
For a small illustrative financial table, the representation should retain which year each revenue value belongs to. Once extraction has recovered the correct cells, Python can serialize them into a readable table:
python
headers = ("Year", "Revenue ($m)")
rows = [("2022", "80"), ("2023", "95")]
lines = [
"| " + " | ".join(headers) + " |",
"| " + " | ".join("---" for _ in headers) + " |",
]
lines.extend("| " + " | ".join(row) + " |" for row in rows)
context = "\n".join(lines)
print(context)
The useful property is the preserved relationship between 2023 and 95, under the Revenue ($m) header. Formatting cannot recover that relationship if the parser has already lost it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Long context changes the unit of retrieval
Larger context windows and stronger needle-in-a-haystack results raise a natural question: how much RAG machinery will remain necessary? Liu separates parsing from chunking. Good parsing remains essential because a larger window still receives whatever representation the pipeline produces. Chunking can become simpler: larger chunks reduce the need to split within pages, and an entire document can become an indexed unit. Documents are often self-contained, making them a natural retrieval boundary.
The enterprise corpus still exceeds what it is practical to supply on every request. Liu invokes collections on the scale of billions of documents to explain why even long-context systems need selection. He considers context caching expensive for that use and raises concerns about accountability into the data supplied to a model. No cache size, reuse frequency or retention period is specified, so the cost judgment remains workload-dependent.
One technical distinction matters here: Liu describes caching as storing transformer weights, but context caching reuses input context; it is not training new weights on the documents. Google’s contemporaneous announcement for Gemini 1.5 Pro and Gemini 1.5 Flash describes repeated-token reuse as a cost-reduction feature. That correction leaves the architectural question intact: external vector or graph storage provides a way to select the information for each request. Liu expects fine-grained chunking decisions to diminish, while retrieval from external storage remains useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Store more than one representation
Multimodality creates another option: retain different representations of the same source. A PowerPoint page can be stored as an image alongside its parsed text. Text extraction is inherently lossy; an image retains visual information that the textual representation may omit.
| Representation | What it retains |
|---|---|
| Parsed text | Extracted wording and encoded structure |
| Page image | The visible page, including layout and graphics |
Keeping both lets an application choose among representations as model capabilities improve, balancing cost, performance and latency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From retrieval tools to agent services
Retrieval accuracy does not automatically produce a natural conversation. An audience member describes a system that overemphasizes contextual data when the user merely wants to provide an FYI and continue talking. Liu’s proposed architecture makes the RAG pipeline a tool available to an outer agent. That agent reasons over conversation history and decides what response fits the current turn. The main tuning points are its reasoning prompt and memory. He describes memory modules at the time as primitive, with few strong approaches beyond placing conversation history in the prompt.
The final question asks about the most complex internal use of Llama Agents. The framework, introduced at the preceding keynote, moves agents from notebooks into independently running API services. Liu’s answer is modest: internal use is mostly constrained, simple RAG pipelines, and the framework is still alpha.
Known limitations include the need for more general communication protocols and interfaces, plus a richer message-queue system. These are extensions to the service architecture, not a claim that the original framework lacked communication infrastructure altogether. The enterprise rationale is concrete: encapsulate an agentic capability behind a service boundary so it can be deployed and reused. The closing example therefore returns to constrained RAG, now packaged as a component other applications can call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and getting-started material for the LlamaIndex framework.
Further reading
Jerry Liu's launch explanation of managed data pipelines, structured PDF parsing and financial-document RAG examples.
The 2024 service-oriented agent architecture, including its control plane, message queue and historical setup examples.
Google's contemporaneous announcement explains repeated-context reuse for Gemini 1.5 Pro and Flash.
Updates since the talk
Documentation for reusing cached input context with the Generate Content API.
Read the complete timestamped transcript
- 0:00
[on-hold music] Hi, everyone. I'm Jerry, co-founder and CEO of, uh, of LlamaIndex.
- 0:18
And, uh, I'll probably spend the first ten minutes just giving, like, a brief overview, I, I mean, of RAG and also just, like, LlamaIndex, uh, how we see the enterprise developer space and, and how it's progressing, as well as, uh, give an overview of the product offerings.
- 0:31
Um, and then I think in the li-- next fifteen minutes, happy to, like, you know, generally field questions and kind of answer, uh, actually maybe, like, have a discussion on what's top of mind, uh, throughout enterprises today.
- 0:41
So let's get started. Um, you know, throughout the enterprise, um, what we're seeing, and this might resonate with some of you, um, there's a lot of different use cases that we're seeing pop up, um, and a lot of it's around RAG, right?
- 0:52
I'm pretty sure we all probably know what RAG is. Uh, you know, you point it at some, like, directory of files, and then you get the LLM to somehow understand these files and then generate answers from them.
- 1:03
Um, some other use cases that we see include, like, document processing and extraction, um, being able to maintain conversations over time. And then this year, there's a lot of people, like, you know, building agents.
- 1:13
Um, we haven't seen as many, like, fully autonomous agents in production. They typically are a bit more constrained. Um, but actually curious to get your takes as well. Um, so happy to discuss that.
- 1:24
So obviously, RAG has been a very popular, um, like, set of techniques basically for helping you build a question-answering interface over your data. That's really the end goal, is to help you build a question-answer interface.
- 1:36
And what are the main components of RAG? I won't go into, like, the super technical details, but, you know, you need an LLM, um, to do the final synthesis.
- 1:43
You need an embedding model. Um, you need a vector database, or you need some database. Um, could be, you know, a document store, it could be a graph store, it could be, uh, like a SQL database, um, or a vector database.
- 1:55
And then here's the thing that's interesting, is that you basically need a new data processing stack to handle the data parsing and ingestion side. Um, this is different than traditional ETL, which is primarily for kind of like, you know, analytics workloads as well as there's a lot of, like, technologies that popped up around that.
- 2:13
Here, you know, you're really, you know, at a very basic level, say, taking in a PDF, slicing it up into a bunch of chunks, um, figuring out how to do that well and index it and represent it in a bunch of different, uh, storage forms, um, so that LLMs can have easy access to it, right?
- 2:28
And a lot of what LlamaIndex is trying to solve is on that data processing piece.
- 2:33
So at a very, you know, um, a, a big pain point that we see, um, for a lot of companies, uh, building LLM applications is going from prototype to production.
- 2:46
Um, unlike traditional ML, um, it's actually really easy to build a prototype, uh, with, like, some of the tools that LlamaIndex offers. It takes, like, ten minutes to build a RAG pipeline, um, that kinda works over your data.
- 2:57
But going from kinda works to something that's production quality is a lot harder. Um, and so as we see you scale up the number of documents, as w- as the documents get more complex, as you try to add more data sources, you have a higher quality bar that you need to meet.
- 3:12
And then some of the general, you know, pain points that we see include accuracy issues, um, knowing how to tune, like, a bunch of different knobs, and then also scaling to a lot of data sources.
- 3:21
Oftentimes, this either takes a lot of developer time or, um, they just don't know how to do it. And so what ends up happening is that POC you're building for the higher-ups, uh, just ends up not, like, really working.
- 3:32
Um, and so therefore, like, the value of that overall project is, uh, diminished.
- 3:38
Um, the other problem that we see is that generally speaking, um, most of these-- a lot of these larger companies that we talk to have a lot of data, um, and there's this, like, general issue of just, like, data silos, right?
- 3:49
Um, you have unstructured data, structured data, semi-structured data, APIs. And somehow, you know, uh, like this is, um, a, a similar problem occurs during the LLM application development, where you wanna somehow bring in all this data into some central place so that your LLMs can understand it, right?
- 4:07
And when they're able to understand it, and ideally, you know, somehow if you had this magic tool that made that happen and made it work well, then you're able to have this kind of like holy grail of RAG, uh, just being able to synthesize answers, um, and do stuff over, like, any of your knowledge, um, uh, anywhere
- 4:24
in the enterprise. Um, a thing that we talk about a lot, uh, both during the keynote yesterday as well as more generally, is the importance of data processing and data quality.
- 4:35
Um, like, you know, we've probably all heard the term, like, garbage in, garbage out, and this is true in machine learning, but this is, uh, also true in, uh, LLM application development.
- 4:46
If you don't have good data quality, and I can go into an example of what that means, um, you're not going to get back, uh, well-represented information, um, so that even if your LLM is very good, uh, oftentimes, if your data quality is bad, this leads to hallucinations within your application.
- 5:05
And so we believe in developers. Like, if you're kind of, you know, um, leading AI at, at one of these, like, enterprises, uh, you do wanna make a bet on developers and, and, like, you know, I, I think generally speaking, and I tell-- I say this, like, pretty often, um, you should generally bet on probably building a
- 5:24
little bit more than, more than just, like, buying pure out-of-the-box solutions. And there's a few reasons why this is the case. First, the AI space is moving really quickly.
- 5:32
The underlying technology is shifting. Developers are the best positioned to translate that technology into enterprise value that are custom to your use case. Um, if you, you know, go through the procurement process and purchase, generally speaking, like, out-of-the-box tools, that will solve maybe, like, the current pain point that you have around that, um, or, like, right?
- 5:52
And, and provide a solution for that. But it will probably be a lot slower to basically adapt it as new techniques pop up- Uh, new workflows are possible. And so we care a lot about developers, and we wanna basically provide the tooling and infrastructure to enable developers to build LLM applications over their data.
- 6:11
This helps you get, um, applications with high data response quality that's actually ready for production. Uh, and importantly, it's, like, easier for developers to set up and maintain, so you don't have to keep throwing developers at it and kind of like banging their heads against the wall to figure out how to actually make this thing generate good
- 6:26
responses. Um, and you can scale this to more data sources.
- 6:32
Great. I'm not gonna go through, you know, kind of, um, all the different features of LlamaIndex, but I'm just gonna quickly run through some of the main components. Um, our main goal as a company is to help, uh, any developer build context-augmented LLM apps, uh, from prototype to production.
- 6:47
We have an open source toolkit, right? And this is an open source framework that's a very popular framework to help you build, uh, help developers build production LLM apps over your data.
- 6:57
Uh, a lot of the use case that we've seen in the past year have been around, like, you know, productionizing RAG. Uh, in, you know, the next six months, we anticipate a lot more agentic use cases to arise as well, and it's primarily focused on orchestration around, like, retrieval, prompting, agentic reasoning, tool use.
- 7:14
The other piece that we have is LlamaCloud, which is a centralized knowledge interface for your production LLM application. Unifies your data sources, starting with unstructured data, is able to process and enhance that data for good data quality so that you actually have, you know, good quality data from your very complex, like, PDFs and PowerPoints, for instance, and
- 7:33
spreadsheets, and helps you build managed pipelines so that you as a developer don't have to worry as much about that and can basically worry, uh, about building the actually interesting stuff around the orchestration of that data, uh, with LLMs.
- 7:48
Um, yeah, I think I mentioned this already, open source toolkit, lot of people using it. I'm gonna skip this. And then LlamaCloud is, again, this, like, centralized knowledge interface for your production LLM app.
- 8:00
Um, you spend-- Like, the idea is to help manage a lot of the data infrastructure so that developers, generally speaking, have to spend less time wrangling with data and spend more time building some of the core, you know, uh, prompting, agentic retrieval logic that, uh, makes up, like, the custom use case that they wanna build for.
- 8:18
Um, I'm not gonna run through all the features that we have, 'cause this is basically just, like, one of the, um, you know, some of these things are upcoming.
- 8:25
But one specific thing that I think has actually gotten a decent amount of interest from users is LlamaParse, which is a specific component of LlamaCloud. It's basically our advanced document parser that helps solve this data quality problem.
- 8:40
Um, basically, if you wanna build LLM applications over, like, a complex financial report or a PowerPoint with a lot of different messy text layouts, um, like, uh, tables, images, diagrams, and so on and so forth, we provide a really nice toolkit to basically help you parse that data specifically so that LLMs can understand it and don't hallucinate
- 9:02
over it. Um, so far, you know, we've released this, like, a few months ago. Um, there's been some impressive usage metrics so far. Um, basically half a million monthly downloads on the client SDK, uh, like, tens of millions of pages processed, and a lot of, like, important customers basically using this throughout the enterprise.
- 9:27
And yeah, uh, generally speaking, maybe just in terms of, like, discussion topics, I'm happy to talk about any of these components. Um, I'm very interested in, generally speaking, like, kind of the enterprise, like, data stack and how that translates into LLM applications.
- 9:41
I'm also interested on the use case side, um, basically the kind of like advancements from simple QA interfaces into more agentic workflows that can actually take actions and automate, uh, more decision-making, uh, from, from different teams, right?
- 9:55
Either internally or externally. Um, and just a quick shout-out is, you know, we have like a general wait list for LlamaCloud, um, that's already gotten pretty popular. Uh, there's been a decent number of sign-ups, but, uh, there's-- Uh, the goal is to basically help enable more users to kind of like process and index their unstructured data.
- 10:14
Uh, so again, they can help, like, manage that and still, uh, build a lot of the kind of like important use cases, um, as enterprise developers. Cool.
- 10:25
Go for it.
- 10:28
Um, I, I understand LlamaCloud requires people to upload documents to your cloud. Um, how do you deal with, uh, customers who are very, uh, sensitive to privacy, uh, of their data?
- 10:47
Yeah, that's a great question. Uh-
- 10:48
Can you just repeat part of the question just 'cause I didn't catch it with the microphone, sir.
- 10:51
Um, so the question was about the enterprise product, LlamaCloud, where, um, the understanding is that you upload cl- uh, documents to our cloud. So how do we deal with, like, data privacy?
- 10:59
Uh, there's two, uh, kind of answers to that. The first is that we have both a cloud service as well as a VPC deployment option. I'm happy to chat about that, uh, if you sign up on the kind of like contact form.
- 11:08
So we deploy in, in, um, AWS and Azure, with GCP coming soon. And then the second is, uh, we're all like kind of a data orchestration layer, so we actually intentionally don't store your data.
- 11:18
Um, we try to integrate with the existing storage systems.
- 11:22
Yeah.
- 11:23
Yeah.
- 11:25
Um, you made a comment on, like, the differences between traditional ETL, um, and, um, you know, kind of the, the new skills and tools, et cetera, required. Can you expand on that a bit so that, you know, in, in my company where I get a- like asked maybe, "Hey, let's have this, uh, ETL person who's done a
- 11:42
lot of other ETL do it," what, what kind of instruction would I give them on like, "Hey, these other skill sets or tools might be necessary"? And if there's any other sort of gotchas around that, if, if you could highlight those, that'd be great.
- 11:52
Totally. I think just on a very technical level, the steps you actually take, um, are just different. Um, basically, instead of writing like SQL or using DBT, um, you Uh, just, you know, uh, this is how you, like, set up a RAG pipeline.
- 12:03
You have a PDF. Um, first, you need to parse that PDF, uh, so either using LlamaParse or another document parser. Um, that parsing step, if you don't get it right, then that leads to a lot of kind of like downstream failure modes in, in your LLM application.
- 12:18
Um, after you parse the document into some representation, whether it's text or increasingly we're seeing like multimodal representations as well with, um, well, like image representations of a document, you then need to chunk that, uh, document, right?
- 12:32
And so the very naive approach is, uh, you basically set a chunk size of like a thousand and twenty-four tokens, and you split every thousand and twenty-four tokens, right?
- 12:41
And that specifically also, you know, introduces a bunch of complexities because if you split, like tables down the middle, you split pages, uh, that split, like, um, there's like, th- there's like a section that spans multiple pages or something, you somehow need to better like semantically join them together, um, so that like most information is preserved within
- 13:00
a chunk, and that you add like the right metadata to that chunk. Um, and then you need to figure out a good way to index it, and this is where like a vector database or a graph store or document store comes in.
- 13:11
There's a lot of different ways to index it. So just very fundamentally, it's just like a different set of like steps you need to do. And the issue here, and the difference actually with traditional ETL, is all these steps are kinda like, um, fuzzy to understand without the end-to-end, uh, performance.
- 13:26
Like with traditional ETL, you know, it's kinda like you do some step and then it's-- you, you know exactly what you want. Here, like it's really hard to tell what the chunk size you need to set is without having an eval dataset and having a rigorous end-to-end testing and eval flow.
- 13:40
Yeah. Oh, sorry. Oh, just wanna make sure. I, I think I saw a hand over there. Yeah. Do it. Yeah.
- 13:49
Um, how, uh, how do I integrate audio sources into, into a LlamaIndex pipelines? Uh, would it be a video with parser? I'm building something specifically that does semantic chunking and paragraphing, uh, from transcript.
- 14:05
Uh, so what would be the desired route to integrate that into LlamaIndex?
- 14:09
Yeah. So I think we have a few, uh, audio loaders. So I think the default is just, uh, take-- So the question was basically how do you integrate audio sources into your RAG pipeline using, you know, uh, LlamaIndex or other frameworks.
- 14:20
Um, the simplest is probably just like you just directly, uh, like parse that into text and then ingest it. I think in the future, as models become more natively multimodal, um, you might just be able to represent audio as like a specific entity, right?
- 14:32
And then as a chunk almost, and directly feed that into a model. But I don't think we're there yet. Um, and then, okay, I'm gonna go this way. Yeah.
- 14:39
I got a question to-- for about, uh, how do you determine whether your parsing is better than other factors? Are there datasets, are there eval sets that are public and you can say, yeah, our Llama parser is so much better than-
- 14:57
For sure. I think the benchmarking is important. It's also challenging 'cause we're actively working on that right now to basically find a general benchmark. What typically happens is we do like a, just within the enterprise, they just do a bake-off on their own data, um, and then compare it.
- 15:09
And we basically show them a notebook on, you know, here's how you build a RAG pipeline with LlamaParse. Uh, here's how you could do it with other parsers. Yeah.
- 15:16
Um, just wanna make sure I cover-
- 15:18
Yeah, mic. [laughs]
- 15:20
Yeah.
- 15:20
Yeah. M-my question is, uh, um, what options do you have like for versioning or different promotion across environments to, you know, do staging and production? Uh, that's one part, and the other one is, um, what regions are you available?
- 15:34
So that's maybe a, a little more easier.
- 15:37
Yeah. Um, I think the versioning piece is, is definitely important. I think, um, at a high level, we are building out features to help you, like, better version your pipelines.
- 15:46
We don't have that yet, but it's kind of like upcoming and also requested by some enterprise customers. Um, and then, uh, the, the second question around, um, kind of regions where, uh, the SaaS service is in North America.
- 15:58
Uh, it's just hosted on, uh, uh... But we do, we do, um, kind of like on-prem deployments as well, right? And, and so that's, that's part of, you know, generally the, the enterprise plan that we offer.
- 16:08
Yeah. Yeah.
- 16:10
Hi. Um, I'm building a RAG system for a big, uh, fintech, basically a bank. Uh, the struggle I'm having is I'm obviously working with the servicing team, which has other channels, right?
- 16:20
I'm working on an-- in a chatbot and a WhatsApp chatbot. The servicing team also has, say, like a help center, an IVR, a bunch of other, like, channels, right?
- 16:29
Um, it's been very tough for me to convince them that maybe the CMS that they're using, you know, to feed these other sources is not the best way to feed a RAG.
- 16:38
I'm curious to know if you've seen other customers that have like a similar issue where, you know, internally, they wanna have like this single source of truth-
- 16:45
Mm-hmm
- 16:45
... that kind of feeds into all of these channels, where the RAG system's nature is obviously extremely different than like a help center or an FAQ or that kind of stuff.
- 16:54
I see. Wait, so why, why is that CMS not the right, um, tool to basically help-
- 16:58
I'm curious to know if you think that could be the right tool or like getting a little bit more into the details, that's like we have like Q&A pairs.
- 17:06
That's how the CMS works right now.
- 17:07
Mm-hmm.
- 17:08
Which could work for RAG, but we're missing all the metadata, the different clusterings of like different documents for different maybe use cases, different credit cards. It's a bit tough to explain in a quick question, but like have you seen a single system work as a single source of truth?
- 17:22
And kinda how have you seen that work-
- 17:24
Yeah, I-
- 17:24
... like in real big use cases?
- 17:26
So I, I think, um, the, uh, yeah, I think, I think the full details, there's probably like a lot to dive into there. I think generally speaking, what we see is, um, for like homogenous data sources where it's like of the same kinda like data type, let's say it's all like, uh, financial reports, you can generally use
- 17:40
like the same set of parameters to basically parse it, 'cause there's like an expectation they're roughly the same format. For very diverse and different data sources, like if all of a sudden you're bringing in not just like, uh, PDF documents, but also, um- Like semi-structured data from like, you know, uh, Jira or something or, or, um, uh,
- 17:58
what was it? Like Salesforce for instance, like JSONs. Um, you typically need to set up like a separate pipeline there. And then what, you know, we both offer on the open source, but also the enterprise side, um, is this ability to like combine, um, all these different data sources, and then you just have to like combine them
- 18:14
together and re-rank them, right? And, and have some re-ranking layer at the, at the, at the top.
- 18:19
All right.
- 18:19
Yeah.
- 18:19
Thank you.
- 18:21
Um, just yeah.
- 18:23
So I've been using Llama-
- 18:23
Hang on. I'll give-- I'm gonna give you the mic just because it's for the-
- 18:25
Oh, sorry.
- 18:25
It's for the recording.
- 18:27
So I've been using LlamaParse for a little bit, and first of all, I love it, so it works really well. So thank you for producing it.
- 18:33
Thanks.
- 18:33
Uh, however, two weeks ago, I was working on a project for a client, and, uh, all of a sudden I was getting all these failures.
- 18:40
Mm-hmm.
- 18:40
And I contacted support via the chat, and there was a gentleman helping me out, and he's like, "Oh, pass me the job IDs. Give me the job IDs." And all of a sudden just went MIA, never replied back.
- 18:50
So the question is, what are the support options so in case I get stuck over the weekend, I could actually get somebody to help me out?
- 18:56
Totally. First of all, I'm sorry you ran into those issues. I know we had like a cluster of just, uh, failures. I think that specific weekend it was just-- It was, um, it was a good lesson for us, right?
- 19:05
Uh, keep in mind, we're like fifteen people at the company. Um, and so we're-- Uh, when, when you talk to support, it's probably like one of the, the founding engineers just like jumping in.
- 19:13
Uh, so I promise we're making that process more streamlined. Um, typically on the enterprise side, like especially, uh, for kind of like the, like, enterprise plans that we offer, I'm happy to pa-chat about this offline, like we offer dedicated SLAs, right?
- 19:24
And so this is kinda like there's, uh, some support option we're doing on the casual, like kind of, uh, like self-serve APIs. But, um, we're offering like dedicated SLAs on, on the enterprise.
- 19:42
Hey. So [clears throat] we are building, uh, building hallucination detection and other evaluation systems for our customers that have a very large, uh, collection of documents. And typically that's like, uh, you could have, of course, like thousands of PDFs, and also those PDFs typically, of course, contain a lot of tables and all that.
- 19:59
Uh, and then there's question of how to combine like OCRs-
- 20:02
Mm-hmm
- 20:03
... and other PDF processing on that. So the question is like what is your general recommendation? Like last-- Does LlamaParse, uh, take care of all this, or do you recommend like building some kind of custom system directly on top of LlamaIndex, or how do you, how would you recommend handling that?
- 20:18
Yeah, I think, I mean, I, I guess I didn't actually show the capabilities of LlamaParse in, in these slides. Um, but, uh, maybe if I dig around a little bit, I can-- I can try to find the, the specific, um, uh, slides where, where it showcases.
- 20:31
Um, yeah, like the-- Basically, what you want when you parse these documents is you want some generally good parser, um, that will lay out the text like, uh, in a spatially aligned way.
- 20:43
Um, and so it doesn't matter if you have all the bells and whistles of like bounding boxes and all these things. You generally like bare minimum, like just want the text to be like faithfully represented, and that's exactly what LlamaParse does, especially for like tables.
- 20:55
So we have a few examples, for instance, where like you have tables within a document, um, and then you lay it out in a spatially aligned way. And then when you feed this to an LLM, LLMs generally are trained to respond pretty well to like well spa-- like just formatted pieces of text, uh, so they can understand
- 21:11
what's going on, um, in that text chunk. Uh, whereas if you use like a very naive parser, like a baseline PDF parser, um, it's gonna like collapse the text and numbers and therefore kind of, uh, it's gonna generate a lot of hallucinations.
- 21:23
But yeah. Yeah.
- 21:29
With the, with the increase in size of context windows that are available to us-
- 21:33
Mm-hmm
- 21:33
... and also the improvements that we're finding for like doing the haste attack kind of problems-
- 21:37
Yeah
- 21:37
... what is your perspective on where we're headed towards RAG?
- 21:41
Yeah. I think there's two general trends. Uh, one is longer context windows, the other is like multimodality. Um, I do think, uh, there's a few things that will probably go away and a few things that will stay.
- 21:51
One is, uh, good parsing is still important. Um, the reason is like, you know, in the end, if your parser is bad, you're just gonna feed bad data into the LLM, and it's gonna hallucinate information.
- 22:00
Um, what I think will probably go away is as context windows get bigger, chunk sizes can also get bigger. Um, so you know, you are probably not gonna need to worry about like intra-page splitting, like splitting a single page into a bunch of smaller chunks.
- 22:14
Um, in the future, we could see you just, uh, like putting entire documents as chunks and basically indexing stuff at a document level. I think that actually makes a lot of sense because documents are typically like self-contained entities.
- 22:25
Um, and I think that'll make it a lot easier for developers. Um, however, in general, for a multi-doc system, which, you know, if you're in a company, you probably have like billions of documents, um, many gigabytes of documents.
- 22:37
It's-- You're probably not gonna feed all billion documents into the, uh, context window on every inference call, even with context caching, um, which I think Gemini has. Uh, because context caching is right now super expensive, um, probably doesn't make sense from a cost perspective, and also is a black box, so you don't get accountability into the data.
- 22:54
Um, you basically store the transformer weights for those of you who like kind of are familiar with that. Um, and you don't really get like full, uh, transparency into what the data is actually being fed, um, into the language model at each step.
- 23:04
So actually, I think for a variety of reasons, the overall idea of retrieval from an external storage system, whether it's a vector database or graph database, still matters for a variety of reasons.
- 23:13
Um, but you know, the minute chunking decisions will pro- will probably go away. The second thing which you didn't ask about but which I'll talk about anyways is, is multimodal.
- 23:21
Um, I think as multimodal, uh, models get better, um, I think it actually makes sense to basically start having like diverse representations of the same thing. Um, so for instance, we have a PowerPoint presentation.
- 23:32
Um, you're able to, uh, like represent each page, for instance, as an image in addition to just like parse text. And by storing native image chunks, you basically preserve all the information within that data.
- 23:43
Um, anytime you do parsing, it's in-inherently lossy, right? 'Cause you're inherently like trying to extract out stuff in like a textual format as opposed to preserving the full picture.
- 23:52
Um, and by having like, um, like different ways of representing the same amount of data, you can basically trade off between like cost, performance, and latency.
- 24:03
Just checking the...
- 24:06
Hi, um, so I see you- you've done a lot of work improving the accuracy, reduce the hallucination. I wonder if you are working on anything to make the conversation flow better.
- 24:16
Uh, in my experience, uh, th- it is so hard to, to get, um, the, the conversation to, to feel natural. Sometimes they overemphasize the, the, the context, uh, data, while I just want, want to give it an FYI and just continue talking like a normal human.
- 24:30
Hmm. So you're, you're talking about like basically how to create more natural conversation flows? That's, uh, that's, uh... Yeah, I think I'll-- So that, that's very interesting. I think, um,
- 24:42
the, the, um, the overall answer to that is I think the default way most people are building these conversation flows is you have some like, say, RAG pipeline as like a tool, right?
- 24:53
Um, and then you basically have an agent as an outer layer, um, that reasons over the conversation history and can, um, basically, you know, synthesize the, the right answer at the given point in time.
- 25:03
So the, the, the knobs basically that you wanna tune are the, the agent reasoning like prompt, um, as well as the memory. And I think the memory is actually pretty important because, um, right now most memory modules are like very primitive.
- 25:15
Um, there's not a lot of good things beyond just like dumping the conversation history into the prompt. Um, so happy to chat more about that as well, but I think there's like a lot of, lot of stuff there that you could probably try.
- 25:26
Um, just wanna double-check the time. Yeah. We are at time. Oh, okay. It's the last one. Okay. No worries. Yeah.
- 25:31
How are you using Llama Agents internally? What's the most complex task?
- 25:36
That's a great question. Um, so for those of you who weren't at the keynote, we launched this thing called Llama Agents, which is an open source, uh, multi-agent, um, framework basically for helping you basically deploy agents as microservices.
- 25:46
Right now, agents primarily live in like notebooks, um, and the idea is to spin them up as like API services. Right now, I think we're mostly just like, uh, using it to build like kind of more constrained simple RAG pipelines, and it's actually still in a alpha state, so I encourage all of you to basically try it
- 26:01
out. Um, there's a lot of things that I already know it can't do. Um, for instance, have like more general, um, kind of like, uh, there's like communication protocols and interfaces that we wanna build in, a more interesting message queue system.
- 26:14
But, you know, if you have an enterprise use case that's like going agentic, and you wanna basically kind of understand it as microservices, uh, so that you can basically reuse, encapsulate it, um, please check it out, come talk to us.
- 26:26
But cool. Fantastic. Thank you. Yeah, sorry for going over. No, that's all fantastic. [clapping] [outro jingle]