Building Production-Ready RAG Applications: Jerry Liu

Jerry Liu, co-founder and CEO of LlamaIndex, presents two broad approaches for helping language models work with data outside their training: retrieval augmentation, which supplies external context in the prompt while keeping the model fixed, and fine-tuning, which incorporates new knowledge through updates to model weights or an adapter. A basic Retrieval Augmented Generation (RAG) system consists of data ingestion followed by querying, which itself includes retrieval and synthesis. 0:14

This architecture underlies familiar applications that answer questions over PDFs and other unstructured data, but a working prototype is not necessarily production-ready. If retrieval misses relevant material, the language model never receives the context required to answer correctly. Low precision introduces irrelevant chunks, unnecessary material, hallucination risk, and lost-in-the-middle problems; low recall means necessary information never appears in the retrieved set. Outdated information, irrelevant outputs, toxicity, and bias introduce additional limitations beyond retrieval itself. 2:04

Improvement opportunities span the entire pipeline: the underlying data and chunk boundaries, embedding representations, retrieval algorithms, and the final synthesis stage. Liu also argues that language models can contribute before answer generation by breaking complicated questions into simpler questions, routing requests across data sources, and supporting more sophisticated reasoning. 3:59

A production improvement strategy needs a task-specific benchmark before architectural changes can be meaningfully compared. Liu distinguishes between evaluating the complete query-to-response experience and evaluating individual components. When retrieval appears to be the bottleneck, measuring final answer quality alone does not adequately explain whether a proposed retrieval change actually returns more relevant evidence. 4:55

For retrieval evaluation, construct a dataset linking input queries to the document identifiers relevant to each query. Those labels can come from human annotation, production user feedback, or synthetic generation. Once the dataset exists, standard ranking measures such as success rate, hit rate, MRR, and NDCG make retrieval quality measurable; Liu frames this component as an information-retrieval problem rather than an exclusively language-model problem. 6:45

For end-to-end evaluation, assemble queries alongside human annotations, user feedback, reference answers, or synthetic examples generated with GPT-4. Run those examples through both retrieval and synthesis, then apply language-model-based evaluations, with or without reference labels. This separates the question of whether the system found useful source material from whether it ultimately produced a useful answer. 6:45

Liu recommends beginning with table-stakes RAG techniques rather than immediately adopting the most complex interventions. Practical starting points include better parsing instead of uniformly splitting documents, adjusting chunk sizes, using hybrid search where supported, and applying metadata filters. More advanced options such as reranking and recursive retrieval can follow, but their value should be established against the benchmark rather than assumed. 8:34

Chunk size is particularly consequential because more retrieved tokens do not necessarily improve answer quality. Information located in the middle of a long context window can be harder for the model to use, and reranking retrieved material does not automatically improve the final generated response. Liu describes observing a dataset-specific optimal chunk size and cases where reranking increased error metrics, reinforcing the need to evaluate synthesis outcomes as well as retrieval changes. 9:30

Metadata filtering adds structured context to otherwise unstructured chunks. Useful metadata can include page numbers, document titles, summaries of adjacent chunks, or generated questions that a chunk could answer. For a question about risk factors in a 2021 SEC 10Q document, semantic search alone might retrieve documents from other years; inferring a structured filter equivalent to year equals 2021 and combining it with semantic search narrows the candidate set and improves precision. 10:26

Small-to-big retrieval addresses a mismatch between what makes text easy to retrieve and what makes it useful for answer synthesis. Embedding a large passage can blur the specific information a query targets because the representation includes surrounding material. Instead, index smaller units, potentially down to individual sentences, and retrieve those more specific units first. 12:18

After identifying the relevant small unit, expand to a larger surrounding window before passing evidence to the language model. This preserves precise matching while still providing enough context to synthesize a coherent answer. Liu describes using a smaller retrieval count, such as K equals 2, instead of retrieving K equals 5 large chunks that may flood the context window and bury the relevant material. 13:05

A related approach embeds a reference to a parent chunk through a smaller excerpt, a summary, or a generated question associated with that larger passage. The retrieval representation is optimized for matching likely queries, while the returned parent context supplies the information needed for synthesis. The key design principle is that the object optimized for embedding-based retrieval does not have to be identical to the evidence ultimately delivered to the model. 13:51

Some questions exceed the capabilities of a single top-K retrieval pass. A task might require multiple reasoning steps, summarizing an entire document, or comparing several documents. Liu describes an exploratory multi-document agent architecture in which each document exposes tools for summarization and fact-specific question answering; because an agent cannot directly access unlimited tools, a retrieval layer first selects the relevant document tools before the agent acts on them. 13:51

Agents are potentially more expressive, but Liu explicitly notes the tradeoffs: they can be harder to implement and may increase latency and cost. Their distinguishing feature is that the system retrieves executable capabilities rather than merely inserting retrieved text into a context window. This combination of retrieval and tool use is presented as a direction for more complex analysis, not as the default replacement for simpler RAG pipelines. 9:30

Embedding fine-tuning can adapt retrieval to domain-specific questions when pretrained representations return inappropriate information. Liu describes generating synthetic queries from raw text chunks and using those examples to fine-tune either the base embedding model or an adapter. Adapter-based tuning does not require access to the base model’s weights, and tuning only the query side can avoid re-indexing the entire document corpus. 15:47

Fine-tuning can also target the language model responsible for synthesis. Liu describes exploring synthetic training examples produced by a larger model such as GPT-4 and distilling those capabilities into 3.5 Turbo, with the aim of improving reasoning, longer responses, and structured outputs. These possibilities are presented as areas of exploration rather than guaranteed production results, and their usefulness still depends on the task-specific evaluations established earlier. 4:55