Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex

The early RAG chatbot followed a fixed recipe: split a private document collection into chunks, embed them, store them in a vector database, retrieve the top k matches, and give those matches to an LLM to generate an answer. In Jerry Liu’s account, this was enough to make a basic application that could chat over a corpus. Every question still passed through the same sequence of steps.

Liu, co-founder and CEO of LlamaIndex, frames RAG in 2026 as an agent harness plus a context layer. Better tool use and agent loops make that separation useful: the harness handles reasoning and repeated actions, while the context layer gives the agent material it can read and search. LlamaIndex’s focus has consequently moved from its origins as a RAG framework toward document infrastructure for agents.

The concrete change is who chooses the search query. A fixed retrieval pipeline takes its query and returns matches. An agent can reason about which keyword or search term is likely to find the information it needs, inspect the results, and search again. Retrieval complexity moves into this loop. Even a basic search tool becomes more useful when the agent can supply a better query rather than accepting the first result set as its only context.

As long contexts and compaction improve, Liu sees more attention moving from managing window overflow toward connecting the right MCP servers, skills, and tasks. These connections determine what an agent can reach and do. The context problem expands from fitting text into a prompt to giving a reasoning system useful access to organizational information.

Program definition is changing alongside access. Python and TypeScript remain part of the earlier development picture, but English increasingly expresses tasks, goals, and runbooks for both engineers and nontechnical teams. The progression runs from asking a simple question, to assigning an end-to-end task, to producing a repeatable program that executes at scale. Liu’s further forecast is that longer-running agents may eventually need a goal and scoring rubric rather than a fully described task; that is a proposed direction, not a capability demonstrated here.

That makes document access a persistent bottleneck even as models improve. Useful context includes web information, tools and skills, warehouses such as Snowflake and Databricks, and files held in SharePoint, Box, Dropbox, or S3. Liu estimates that more than 10 trillion pages of human knowledge live in PDFs, PowerPoints, Word documents, and Excel sheets. The scale explains LlamaIndex’s focus: much of the information an agent needs already exists, but its container does not make it easy to consume. Agents also produce more readily readable formats such as Markdown and HTML.

An agent-native document platform needs three distinct capabilities. Parsing makes a file readable; storage makes it available to operate on; workflows turn recurring document tasks into repeatable execution. Keeping these concerns separate also leaves room to tune a frequent task more carefully than a general-purpose agent would.

• Parsing: Convert PDFs, PowerPoints, and Word documents into accurate, token-efficient context, including Markdown and metadata. The output should preserve the information an agent needs without making it ingest all of the source format’s machinery.
• Semantic storage: Capture and store documents behind an interface that lets agents manage and operate on them. This is document management for humans and agents, extending the role previously served by a file system and applications such as Microsoft Word.
• Repeatable workflows: Specialize recurring jobs such as invoice processing, KYC, and claims. A purpose-built workflow gives the system a place to tune cost and accuracy against a known goal.

This platform is still incomplete. Agent-native formats, document versioning, editing, and collaboration remain open areas in Liu’s framing. He names them as work ahead; the recording develops parsing, extraction, and search rather than explaining implementations for those future capabilities.

Why does document OCR remain hard after more than 20 years? A PDF is designed for printing and display. Text can appear as individual glyphs with coordinates. A table can consist of drawn line segments and text positioned inside apparent cells. The page looks structured to a person, but those drawing instructions do not by themselves give an agent a usable table.

The table example exposes the reconstruction task. First, the parser encounters characters and shapes at positions. It must determine which pieces belong together and recover a representation that humans and agents can interpret. Multicolumn text introduces another problem: the stored sequence of characters does not guarantee the order a person would read the page. Recognizing characters is only part of the job; arranging them into the right relationships is what makes the result useful.

Word documents and PowerPoints offer more structural information, but feeding their native XML directly to an agent creates its own burden. Much of the tagging is unnecessary for understanding the content. A useful parser lifts relevant formatting and semantic metadata, removes irrelevant markup, and can render the page so its overall layout remains visible. More source structure helps, but it still needs to become a representation suited to the reader.

Two approaches attack the reconstruction problem from different directions. Pipeline methods use rules and file information to group text, identify tables and paragraphs, and produce an output representation. A vision-language model, or VLM, can instead read the document visually and turn it into text in one shot. Visual reading can recover layout that a simpler pipeline misses, but Liu identifies three costs: hallucinations even on text-only pages, expensive inference, and insufficient semantics and grounding.

LlamaParse combines file-aware processing with visual understanding. Its described ingredients include optimized PDF, Word, and PowerPoint engines; an agentic harness that routes between cheaper specialized models and frontier models; and parameter-efficient document VLMs focused on particular document classes or elements such as tables and charts. The cost argument follows from that routing: use specialized processing where it suffices, and spend on stronger visual reasoning where the document requires it.

Liu describes this hybrid as lying on the cost–accuracy Pareto frontier: a set of choices where improving one objective requires giving up something on another. He also expects specialized document workflows to outperform general frontier-model use on this tradeoff. These comparative claims are his assessment; the recording does not establish a universal performance advantage across document types.

ParseBench gives this problem a broader evaluation target. Liu describes 2,000 human-verified pages and approximately 50 evaluated frontier models, open-weight models, and specialized OCR solutions. It measures tables, charts, content faithfulness, and semantic formatting, with emphasis on whether agents can understand the result rather than merely whether its syntax is correct. That choice matches the table problem: plausible-looking output is less useful if it loses the relationships needed to interpret the information.

The desired direction is lower cost and higher accuracy, but the distribution of documents complicates any single choice. Simple pages and difficult enterprise documents demand different processing. Liu’s conclusion from the benchmark is that document understanding remains unsolved; covering a range of operating points matters because a real collection contains a changing mix of document types and complexity.

Latency adds a third objective. A parser suitable for an offline indexing job may make an interactive agent wait too long, while a cheap first pass may be inadequate for regulated extraction. Liu distinguishes three workloads by what a mistake or a delay costs.

• High accuracy: Financial services and insurance may require 99% to almost 100% accuracy because an incorrect extraction can damage a financial model or trigger a fraud flag. Those figures describe requirements, not demonstrated achievement. Paying more per page for deeper reasoning can be justified by the consequence of an error.
• Low cost: Indexing a million-plus documents per day from a continually updated SharePoint collection favors a scalable offline pipeline. Some imperfections can be acceptable if an agent can later revisit the source and recover the needed information with citations.
• Low latency: Uploading a thousand documents and wanting them processed within a minute puts pressure on VLM-based processing. Liu explicitly includes his own service among those that find this scenario difficult.

LightParse addresses the fast-pass role. Liu describes a free, open-source Rust parser that produces Markdown without a VLM or deeper model, and calls it the fastest open-source parser. That speed superlative is his claim rather than a quantified comparison in the recording. Its practical role is clear: make an initial reading of many documents cheap and quick, while keeping a slower visual parser available as a tool.

Follow the thousand-PDF example through the change in processing. The agent first gets a fast pass over the whole collection and scans the resulting context. If its task then requires understanding values on a page containing a table or chart, it calls a VLM-based tool for that page. The collection becomes broadly readable before the system spends time on deeper visual interpretation. The table that began as positioned characters and lines now receives the additional inspection needed to read its values correctly. This is a proposed operating pattern, not a timed demonstration of processing a thousand PDFs within a minute.

Where does the slower visual work enter the agent loop? The flow below shows it after the collection-wide pass, on a branch selected by the task. The important relationship is the scope of work: the fast parser covers all documents, while the visual tool handles the pages that need deeper interpretation. Liu describes LightParse as an installable skill that complements other VLM-based OCR tools.

The closing technical section moves from readable context to structured information. Processing a million invoices, expense reports, receipts, or claims requires outputs that a downstream database or system can accept. This automates the familiar human task of scanning paperwork and entering data. Parsing supplies a usable representation; extraction turns the relevant information into structured outputs.

LlamaParse’s described extraction capabilities attach granular source citations to every extracted output and provide confidence scores and uncertainty flags. Those serve different practical purposes: a citation lets someone return to the source, while an uncertainty flag helps decide whether a value should enter another system. The decision occurs before insertion, so extraction does not have to treat every produced value as equally trustworthy. The recording gives no confidence threshold or calibration method.

Document search also becomes a larger toolset: BM25, grep, vector search, reading, and scrolling. That returns to the opening change in RAG. An agent can choose a search term, retrieve material, and continue reading rather than relying on one fixed top-k response. The document context layer earns its place by supporting both discovery across a collection and closer inspection of the source when the task demands it.