# Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex Jerry Liu, LlamaIndex AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/RQi7x-navxU-building-document-context-layer-ai-agents) · [Video](https://www.youtube.com/watch?v=RQi7x-navxU) ## Building the Document Context Layer for AI Agents Jerry Liu explains how RAG separates into agent reasoning and document context, why reading a PDF requires reconstructing its structure, and how fast parsing plus selective visual inspection can balance accuracy, cost, and latency. ### From fixed retrieval to an agent that chooses its searches [0:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=12s) The early RAG chatbot followed a fixed recipe: split a private document collection into chunks, embed them, store them in a vector database, retrieve the top `k` matches, and give those matches to an LLM to generate an answer. In Jerry Liu’s account, this was enough to make a basic application that could chat over a corpus. Every question still passed through the same sequence of steps. ![Source frame: From fixed retrieval to an agent that chooses its searches](https://ai.engineer/api/talk-essay-media/6dc0a664b9e77333b7851989734c9a02d32b42524384069e1f96e7a1a62e2a7e) Source frame: From fixed retrieval to an agent that chooses its searches Liu, co-founder and CEO of LlamaIndex, frames RAG in 2026 as an agent harness plus a context layer. Better tool use and agent loops make that separation useful: the harness handles reasoning and repeated actions, while the context layer gives the agent material it can read and search. LlamaIndex’s focus has consequently moved from its origins as a RAG framework toward document infrastructure for agents. The concrete change is who chooses the search query. A fixed retrieval pipeline takes its query and returns matches. An agent can reason about which keyword or search term is likely to find the information it needs, inspect the results, and search again. Retrieval complexity moves into this loop. Even a basic search tool becomes more useful when the agent can supply a better query rather than accepting the first result set as its only context. ### Context access and programs move up the stack [3:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=189s) As long contexts and compaction improve, Liu sees more attention moving from managing window overflow toward connecting the right MCP servers, skills, and tasks. These connections determine what an agent can reach and do. The context problem expands from fitting text into a prompt to giving a reasoning system useful access to organizational information. ![Source frame: Context access and programs move up the stack](https://ai.engineer/api/talk-essay-media/9cee666d938bedad13ff6564befe5da35e30410168a6e63966b7a8d9bc314f1d) Source frame: Context access and programs move up the stack Program definition is changing alongside access. Python and TypeScript remain part of the earlier development picture, but English increasingly expresses tasks, goals, and runbooks for both engineers and nontechnical teams. The progression runs from asking a simple question, to assigning an end-to-end task, to producing a repeatable program that executes at scale. Liu’s further forecast is that longer-running agents may eventually need a goal and scoring rubric rather than a fully described task; that is a proposed direction, not a capability demonstrated here. That makes document access a persistent bottleneck even as models improve. Useful context includes web information, tools and skills, warehouses such as Snowflake and Databricks, and files held in SharePoint, Box, Dropbox, or S3. Liu estimates that more than 10 trillion pages of human knowledge live in PDFs, PowerPoints, Word documents, and Excel sheets. The scale explains LlamaIndex’s focus: much of the information an agent needs already exists, but its container does not make it easy to consume. Agents also produce more readily readable formats such as Markdown and HTML. ### Three layers turn files into usable work [6:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=412s) An agent-native document platform needs three distinct capabilities. Parsing makes a file readable; storage makes it available to operate on; workflows turn recurring document tasks into repeatable execution. Keeping these concerns separate also leaves room to tune a frequent task more carefully than a general-purpose agent would. ![Source frame: Three layers turn files into usable work](https://ai.engineer/api/talk-essay-media/ee3da59dcb2c0bfb4b106c2b00cdca78170779e6791e19af9ee9cbaaf3780a73) Source frame: Three layers turn files into usable work - **Parsing:** Convert PDFs, PowerPoints, and Word documents into accurate, token-efficient context, including Markdown and metadata. The output should preserve the information an agent needs without making it ingest all of the source format’s machinery. - **Semantic storage:** Capture and store documents behind an interface that lets agents manage and operate on them. This is document management for humans and agents, extending the role previously served by a file system and applications such as Microsoft Word. - **Repeatable workflows:** Specialize recurring jobs such as invoice processing, KYC, and claims. A purpose-built workflow gives the system a place to tune cost and accuracy against a known goal. This platform is still incomplete. Agent-native formats, document versioning, editing, and collaboration remain open areas in Liu’s framing. He names them as work ahead; the recording develops parsing, extraction, and search rather than explaining implementations for those future capabilities. ### A PDF draws a page; the parser reconstructs its meaning [8:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=515s) Why does document OCR remain hard after more than 20 years? A PDF is designed for printing and display. Text can appear as individual glyphs with coordinates. A table can consist of drawn line segments and text positioned inside apparent cells. The page looks structured to a person, but those drawing instructions do not by themselves give an agent a usable table. ![Source frame: A PDF draws a page; the parser reconstructs its meaning](https://ai.engineer/api/talk-essay-media/869994c924cc5cfb702bf13f23c708f697806930fc6d53cb44e70bc1e1151db0) Source frame: A PDF draws a page; the parser reconstructs its meaning The table example exposes the reconstruction task. First, the parser encounters characters and shapes at positions. It must determine which pieces belong together and recover a representation that humans and agents can interpret. Multicolumn text introduces another problem: the stored sequence of characters does not guarantee the order a person would read the page. Recognizing characters is only part of the job; arranging them into the right relationships is what makes the result useful. Word documents and PowerPoints offer more structural information, but feeding their native XML directly to an agent creates its own burden. Much of the tagging is unnecessary for understanding the content. A useful parser lifts relevant formatting and semantic metadata, removes irrelevant markup, and can render the page so its overall layout remains visible. More source structure helps, but it still needs to become a representation suited to the reader. ### Combine file-aware parsing with visual models [10:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=655s) Two approaches attack the reconstruction problem from different directions. Pipeline methods use rules and file information to group text, identify tables and paragraphs, and produce an output representation. A vision-language model, or VLM, can instead read the document visually and turn it into text in one shot. Visual reading can recover layout that a simpler pipeline misses, but Liu identifies three costs: hallucinations even on text-only pages, expensive inference, and insufficient semantics and grounding. ![Source frame: Combine file-aware parsing with visual models](https://ai.engineer/api/talk-essay-media/b278185adbfeddce253aa68b6b17895b6e84b6058fb19515b07d7aad65c77533) Source frame: Combine file-aware parsing with visual models **LlamaParse** combines file-aware processing with visual understanding. Its described ingredients include optimized PDF, Word, and PowerPoint engines; an agentic harness that routes between cheaper specialized models and frontier models; and parameter-efficient document VLMs focused on particular document classes or elements such as tables and charts. The cost argument follows from that routing: use specialized processing where it suffices, and spend on stronger visual reasoning where the document requires it. Liu describes this hybrid as lying on the cost–accuracy Pareto frontier: a set of choices where improving one objective requires giving up something on another. He also expects specialized document workflows to outperform general frontier-model use on this tradeoff. These comparative claims are his assessment; the recording does not establish a universal performance advantage across document types. **ParseBench** gives this problem a broader evaluation target. Liu describes 2,000 human-verified pages and approximately 50 evaluated frontier models, open-weight models, and specialized OCR solutions. It measures tables, charts, content faithfulness, and semantic formatting, with emphasis on whether agents can understand the result rather than merely whether its syntax is correct. That choice matches the table problem: plausible-looking output is less useful if it loses the relationships needed to interpret the information. The desired direction is lower cost and higher accuracy, but the distribution of documents complicates any single choice. Simple pages and difficult enterprise documents demand different processing. Liu’s conclusion from the benchmark is that document understanding remains unsolved; covering a range of operating points matters because a real collection contains a changing mix of document types and complexity. ### Read everything quickly, then inspect selected pages deeply [15:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=908s) Latency adds a third objective. A parser suitable for an offline indexing job may make an interactive agent wait too long, while a cheap first pass may be inadequate for regulated extraction. Liu distinguishes three workloads by what a mistake or a delay costs. ![Source frame: Read everything quickly, then inspect selected pages deeply](https://ai.engineer/api/talk-essay-media/f9368b5c89750d93a4a4bc4d2e909b4240aed0a23d685fefaeaffb441e0dc608) Source frame: Read everything quickly, then inspect selected pages deeply - **High accuracy:** Financial services and insurance may require 99% to almost 100% accuracy because an incorrect extraction can damage a financial model or trigger a fraud flag. Those figures describe requirements, not demonstrated achievement. Paying more per page for deeper reasoning can be justified by the consequence of an error. - **Low cost:** Indexing a million-plus documents per day from a continually updated SharePoint collection favors a scalable offline pipeline. Some imperfections can be acceptable if an agent can later revisit the source and recover the needed information with citations. - **Low latency:** Uploading a thousand documents and wanting them processed within a minute puts pressure on VLM-based processing. Liu explicitly includes his own service among those that find this scenario difficult. **LightParse** addresses the fast-pass role. Liu describes a free, open-source Rust parser that produces Markdown without a VLM or deeper model, and calls it the fastest open-source parser. That speed superlative is his claim rather than a quantified comparison in the recording. Its practical role is clear: make an initial reading of many documents cheap and quick, while keeping a slower visual parser available as a tool. Follow the thousand-PDF example through the change in processing. The agent first gets a fast pass over the whole collection and scans the resulting context. If its task then requires understanding values on a page containing a table or chart, it calls a VLM-based tool for that page. The collection becomes broadly readable before the system spends time on deeper visual interpretation. The table that began as positioned characters and lines now receives the additional inspection needed to read its values correctly. This is a proposed operating pattern, not a timed demonstration of processing a thousand PDFs within a minute. Where does the slower visual work enter the agent loop? The flow below shows it after the collection-wide pass, on a branch selected by the task. The important relationship is the scope of work: the fast parser covers all documents, while the visual tool handles the pages that need deeper interpretation. Liu describes LightParse as an installable skill that complements other VLM-based OCR tools. A fast collection pass with selective visual inspection The agent scans all documents first, then uses slower visual processing when its task requires deeper understanding of a table or chart. - Uploaded PDF collection — The example contains a thousand PDFs. - LightParse fast pass — Produce Markdown without a VLM and scan the collection efficiently. - Agent reads context — Determine whether the task needs deeper understanding of a particular page. - VLM inspects selected page — Use slower processing to read table or chart values more carefully. - Uploaded PDF collection → LightParse fast pass (Process all documents) - LightParse fast pass → Agent reads context (Scan initial context) - Agent reads context → VLM inspects selected page (When table or chart values need deeper reading) - VLM inspects selected page → Agent reads context (Return the more carefully read information) ### Extraction connects document reading to downstream systems [18:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=1108s) The closing technical section moves from readable context to structured information. Processing a million invoices, expense reports, receipts, or claims requires outputs that a downstream database or system can accept. This automates the familiar human task of scanning paperwork and entering data. Parsing supplies a usable representation; extraction turns the relevant information into structured outputs. ![Source frame: Extraction connects document reading to downstream systems](https://ai.engineer/api/talk-essay-media/eeb04b2c4cfe7247375b98ce3f3d566adc8c76ed2f7b8f5d41260ddb58b09b2b) Source frame: Extraction connects document reading to downstream systems LlamaParse’s described extraction capabilities attach granular source citations to every extracted output and provide confidence scores and uncertainty flags. Those serve different practical purposes: a citation lets someone return to the source, while an uncertainty flag helps decide whether a value should enter another system. The decision occurs before insertion, so extraction does not have to treat every produced value as equally trustworthy. The recording gives no confidence threshold or calibration method. Document search also becomes a larger toolset: BM25, `grep`, vector search, reading, and scrolling. That returns to the opening change in RAG. An agent can choose a search term, retrieve material, and continue reading rather than relying on one fixed top-`k` response. The document context layer earns its place by supporting both discovery across a collection and closer inspection of the source when the task demands it. ## Key takeaways - Modern RAG can separate agent reasoning from context access: the agent chooses queries and iterates, while the document layer supplies readable, searchable information. - Document parsing must recover relationships as well as characters. Positioned glyphs, drawn table borders, and multicolumn layouts do not guarantee a useful reading order or semantic structure. - Choose parsing effort by workload. Regulated extraction, large-scale indexing, and interactive uploads place different demands on accuracy, cost, and latency. - A fast collection-wide pass can precede selective VLM inspection. This gives the agent broad context before it spends more time reading particular tables or charts. - Structured extraction needs a path back to the source and a decision about uncertain values before they enter downstream systems. ## Slides ### Agenda Liu frames RAG in 2026 as an agent harness plus a context layer, then previews document context capabilities and what comes next. [0:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=42s) [Agenda covering RAG in 2026, the modern document context layer for agents, and what’s next.](https://assets.ai.engineer/sites/aiecode2025/sha256/743678d5ae9c9fd4022ff192bf07eefa348445c75c8b70e7586b0191ded299ba.webp) Agenda 1. RAG in 2026 2. The Modern Document Context Layer for Agents 3. What’s Next LlamaIndex · Redefine Document Workflows with AI Agents ### The Document Infrastructure for AI Agents LlamaIndex describes its evolution from a RAG framework to document infrastructure for AI agents, reporting 30M+ monthly downloads and 1B+ pages processed. [1:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=63s) [LlamaIndex company overview with customer logos and metrics for downloads, GitHub stars, users, and pages processed.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e42df26e6e3d126912a5c78d13260060c9e453cab47e444f5682bc5bb160fa3.webp) The Document Infrastructure for AI Agents LlamaIndex delivers the world’s best platform for agents to read and operate over documents. Starting with agentic document processing, expanding to the semantic and agentic layer. Backed By: NORWEST greylock tabs Lovable CEMEX vantager Delphi KPMG CARLYLE REDICA Systems 30M+ MONTHLY DOWNLOADS 41K+ GITHUB STARS 300K+ USERS 1B+ PAGES PROCESSED Redefine Document Workflows with AI Agents LlamaIndex ### RAG in 2026 Jerry Liu introduces the question of what RAG means in 2026 and how agents use context. [1:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=91s) [RAG in 2026 section title beside a stepped multicolor graphic.](https://assets.ai.engineer/sites/aiecode2025/sha256/e55f3c8296d15688b53c8f76c2ceab28477f20b45550b2edd3563aa613c5bf93.webp) RAG in 2026 LlamaIndex · Redefine Document Workflows with AI Agents ### Let’s Go Back to March 2023 Naive RAG used a fixed pipeline: split documents into chunks, embed them in a vector database, retrieve the top-k matches, and pass them to an LLM to answer questions. [1:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=102s) [Naive RAG workflow: documents are chunked and stored in a vector database, then retrieved as context for an LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/015fac62a2fed83a6fa4cb9a0549e650ba07063146d3fa08caad70798e4658c7.webp) Let’s go back to March 2023 Everyone was building a simple chatbot! Naive RAG • Chunk text, embed, put in a vector database. • Naive top-k retrieval, generate with LLM Current RAG Stack for building a QA System Data Ingestion Data Querying (Retrieval + Synthesis) Doc Chunk Chunk Chunk Chunk Vector Database Chunk LLM 5 Lines of Code in LlamaIndex! LlamaIndex · Redefine Document Workflows with AI Agents ### Evolving Knowledge Work Patterns Liu maps a shift from questions and workflows built with code or drag-and-drop to tasks and repeatable workflows defined in English. He predicts agents will eventually work from goals and values. [4:17](https://www.youtube.com/watch?v=RQi7x-navxU&t=257s) [Three stages progress from code and drag-and-drop workflows in 2023–2025, through workflows defined in English today, to goals and values in 2026 and beyond.](https://assets.ai.engineer/sites/aiecode2025/sha256/239a36052b2619bd84d42db544ad982425f0ff9d8538d5ec9024de0bd92cf5e9.webp) Evolving Knowledge Work Patterns Ask Simple Questions through English Build Workflow through Code Build Workflow through Drag and Drop 2023–2025 Define and solve ad-hoc tasks through English Define repeatable programs and workflows through English Today Define the goal and it’ll solve it Define culture and values 2026+ LlamaIndex · Redefine Document Workflows with AI Agents ### The Modern Document Context Layer for Agents LlamaIndex focuses on unlocking unstructured context for agents, with document context as a core area. [5:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=310s) [Title slide with a stepped multicolor graphic.](https://assets.ai.engineer/sites/aiecode2025/sha256/c97a4757a35fa6e6e548a93ea31cb94348b1de8ea29bf162a298e1bb3d197405.webp) The Modern Document Context Layer for Agents LlamaIndex · Redefine Document Workflows with AI Agents ### Context is Everything Liu argues that agents need a task and organizational context. The diagram feeds prompts, web context, tools, structured data and documents into an agent to produce outcomes. [5:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=334s) [A diagram shows prompts, web context, tools, structured data and document context feeding an agent that produces outcomes.](https://assets.ai.engineer/sites/aiecode2025/sha256/939618c0a393f9b5005d80fc4d0f1be972776f6a1730662db208b8de44fb9238.webp) Context is Everything Model reasoning is exponentially improving, but models are blank slates. The biggest barrier to AI value is the user’s own ability to provide context to the models. The more complex the business process, the more complex the prompt that users need to define. Context Input Prompts (Question, Task, Goal) Web Context Tools, Skills, MCP Structured Data Document Context Agent Outcomes LlamaIndex · Redefine Document Workflows with AI Agents ### Documents: Universal Containers for Unstructured Context Documents hold human-native knowledge, while agents generate content in formats such as Markdown and HTML. The chart depicts AI-agent page processing rising steeply toward 2035, above human reading and new document pages. [6:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=389s) [Document illustrations and a page-volume chart accompany a comparison of human-native and agent-native formats.](https://assets.ai.engineer/sites/aiecode2025/sha256/56f5e6d50aac5e66bafef20ee660c2d95229d59cc8c4f185ebc897b5bf95aa49.webp) Documents: Universal Containers for Unstructured Context • There are 10T+ pages of human-native knowledge locked up within PDFs, Powerpoints, Word docs, Excel sheets • At the same time, agents are generating exponentially more content for themselves as well as humans Page Volume Pages Processed by AI agents ~50T+ per year Pages read by human eyeballs ~10–12T+ a year Net New Global Document Pages ~5–7T+ a year Legacy IDP ~1.5T+ a year Timeline: 2025, 2035 Human-native documents (.pdf, .pptx, .xlsx) Agent-native documents (.md, .html, ??) MD HTML LlamaIndex · Redefine Document Workflows with AI Agents ### Building an Agent-Native Document Platform Liu proposes three layers between documents and agents: parsing for token-efficient context, storage for document management, and repeatable workflows tuned for accuracy and cost. [6:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=410s) [A document context layer connects documents to an agent through parsing, semantic and storage, and agentic workflow layers.](https://assets.ai.engineer/sites/aiecode2025/sha256/b6bd43c1f5a3ef8a34ae6332fe3bfeecb8a5493bfc1bc7d018298215d1feacfc.webp) Building an Agent-Native Document Platform Agent Document Context Layer Parsing Layer Digitalize the 10T+ pages of human-created documents into accurate, token-efficient context. Semantic and Storage Layer Document management for humans and agents. Edit, version, and collaborate over documents. Agentic Workflow Layer Hillclimb repeatable workflows over documents to optimize accuracy and lower cost. LlamaIndex · Redefine Document Workflows with AI Agents ### Building an Agent-Native Document Platform Liu maps today’s OCR, extraction, search and workflows across three layers, then identifies unexplored work from agent-native formats and document collaboration to hillclimbing and long-horizon agents. [8:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=481s) [A table compares current and unexplored capabilities across parsing, semantic and storage, and agentic layers.](https://assets.ai.engineer/sites/aiecode2025/sha256/b0d4b0a4ff51ddfe4e0eac9ca6c9a11796fcd6e2c84d6bee1dbd0d6ddf5318ce.webp) Building an Agent-Native Document Platform Document Context Layer Parsing Layer Semantic and Storage Layer Agentic Layer Today Parsing Layer: Document OCR Semantic and Storage Layer: Document Extraction; Document Search Agentic Layer: Document Workflows Unexplored Parsing Layer: Agent-native Document Formats Semantic and Storage Layer: Document Versioning; Document Editing; Document Permissions and Collaboration Agentic Layer: Hillclimbing-as-a-service; Long-horizon agents LlamaIndex · Redefine Document Workflows with AI Agents ### Document OCR is Hard A PDF can display a clean sentence while storing text fragments and positioning commands, making the raw file hard for an agent to interpret. [8:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=518s) [Slide contrasts readable document text with the instructions stored in a PDF.](https://assets.ai.engineer/sites/aiecode2025/sha256/9516029f33eaad8f86eb215d0dda3eb9ad67ee1506d95b77cbad4a86bc6e0b70.webp) Document OCR is Hard • 90% of the world’s data lies in unstructured documents like PDFs, Powerpoints, Word, Excel • These documents are not natively machine readable and require machine learning to extract information PDF The quick brown fox jumps over the lazy dog. [clean text] What you see PDF BT /F0 12 Tf 1 0 0 1 72 740 Tm (The quick bro) Tj 1 0 0 1 168 740 Tm (wn)Tj 1 0 0 1 72 726 Tm (fox jumps over) Tj … ET What the PDF actually stores What You See vs. What the PDF Stores LlamaIndex · Redefine Document Workflows with AI Agents ### PDF Parsing is Hard A PDF table can be stored as separate lines and positioned text, leaving a parser to reconstruct their relationship. In a multi-column page, content stream order can also differ from reading order. [9:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=549s) [Slide illustrates PDF tables as positioned elements and shows that content stream order can differ from visual reading order.](https://assets.ai.engineer/sites/aiecode2025/sha256/c44feca45ec87f62448dae0f2544d0ceb91377eaafae24c4963840079832ee4b.webp) PDF Parsing is Hard • PDFs are designed for text display, not for machine readability • Text are glyph shapes • Tables are line segments and floating text • Reading order needs to be inferred Item | Revenue widget A | $1.2 M widget B | $0.8 M Line: (72,700)(540,700) Line: (72,700)(72,500) Text "Item" at (85,680) Text "Revenue" at (241,680) Text "Widget A" at (85,630) no connection between them visual page 1 4 2 5 3 6 pdf content stream order 3rd stored: "Column 1, para 2" 1st stored: "Column 2, para 1" 5th stored: "Column 1, para 3" 2nd stored: "Column 2, para 2" arbitrary order, same visual result LlamaIndex · Redefine Document Workflows with AI Agents ### Even Word Parsing is Hard Word’s OOXML exposes structure through tags, but a useful representation must preserve formatting and metadata while rendering the document to reveal its page layout. [10:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=606s) [Slide describes Word's OOXML structure and the challenge of preserving formatting and metadata.](https://assets.ai.engineer/sites/aiecode2025/sha256/c14d641ea8d97681f1b978b30330a7686c8f05289413a77729fb470c3f749861.webp) Even Word Parsing is Hard • Word documents are OOXML format • More structural information than a PDF, but token inefficient and requires rendering to understanding pagination • Needs a concise representation that preserves formatting, metadata (versioning, comments, out of canvas, speaker notes) word/document.xml ← table ← row ← cell ← bold Revenue 1. ordered | • unordered | • indented 2. list | • list | 1. list paragraph | I. roman | A. uppercase • list | II. numerals | a. lowercase paragraph | i. lowercase | b. alphabet bold text | italic text | underline text strikethrough text | superscript¹ | subscript₁ LlamaIndex · Redefine Document Workflows with AI Agents Interface text: table-formatted.docx [illegible] ### An Evolution of Approaches The hybrid routes body text to extraction and tables and charts to a VLM, then validates the results into structured output with bounding boxes. The alternatives use independent stages or a single VLM. [10:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=652s) [Diagram compares pipeline, vision model, and hybrid document parsing approaches.](https://assets.ai.engineer/sites/aiecode2025/sha256/06b0ced30deaeafd19cb56ae02f9578a33dccc7313017c595d7c83a17ca432d8.webp) An Evolution of Approaches 1. Deterministic Pipelines: Heuristics and ML models 2. Vision Models 3. Hybrid Approach (Ours) pipeline approach page image text direction chat recog layout analysis output each stage independent model approach page image [ VLM ] Structured Output reasons about everything together hybrid approach PDF page layout detection Text extraction (body text) fast / $ VLM processing (tables, charts) agentic validation structured output (markdown + metadata bounding boxes) LlamaIndex · Redefine Document Workflows with AI Agents ### The Document OCR Agent Harness Liu argues that routing, specialized VLMs and optimized PDF extraction can make document OCR cheaper and more accurate than frontier models. The chart illustrates the proposed cost–accuracy gap. [12:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=734s) [Slide lists components of a document OCR agent harness beside a cost and accuracy chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d9e24f41e447b45ad5ca2c8d34bf05939927b907066cc89b7b2738ac13b3bb5.webp) The Document OCR Agent Harness The pareto frontier for document OCR will always be much more accurate + cheap compared to the latest frontier models. Ways to exploit that gap: 1. Agentic harness: auto-route between models with agentic validation. 2. Specialized document VLMs for best-in-class specialized accuracy: tables, charts, hand-writing 3. Optimized PDF engine: Custom fork of common PDF engines. 4. Benchmark & auto-research infra: closing rapid iteration loop from eval to harness/model update. Accuracy Cost SOTA Document OCR (LlamaParse) Latest Frontier Models Increased Accuracy: • Post-train VLMs with RLVR for tables, charts, hand-writing, etc • Agentic judge & refinement Cost Optimization: • Auto-routing (doc complexity model) • Model distillation & quantization • Direct extraction (optimized PDFium fork) LlamaIndex · Redefine Document Workflows with AI Agents ### The Document OCR Agent Harness The chart positions LlamaParse above frontier models on accuracy and shows cost optimization moving left. Auto-routing and model compression target cost; post-training and agentic refinement target accuracy. [13:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=792s) [Slide shows a LlamaParse link beside a document OCR cost and accuracy chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b2cc9c117d24852f63a0cdc63bdec41eb51c645971ca156d1d441f4289b4ea1.webp) The Document OCR Agent Harness LlamaParse: https://cloud.llamaindex.ai/ SOTA Document OCR (LlamaParse) Y-axis: Accuracy X-axis: Cost Increased Accuracy: • Post-train VLMs with RLVR for tables, charts, hand-writing, etc • Agentic judge & refinement Latest Frontier Models Cost Optimization: • Auto-routing (doc complexity model) • Model distillation & quantization • Direct extraction (optimized PDFium fork) LlamaIndex · Redefine Document Workflows with AI Agents ### ParseBench: Enterprise Document Benchmark ParseBench tests whether agents can understand enterprise documents across five dimensions, from tables and charts to formatting and visual grounding. It uses 2,000 human-verified pages and emphasizes semantic correctness. [13:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=796s) [ParseBench overview with benchmark statistics and five document evaluation dimensions.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd0d7923327948a2d320dc20af4e45495c552e90108ec3d5bc0594d736f79e78.webp) ParseBench: An Enterprise Document Benchmark for Agents The most comprehensive enterprise document benchmark, for AI agents: 1. 2k human-verified pages, 167k test rules 2. Measures across tables, charts, content faithfulness, semantic formatting. 3. Optimizes for semantic correctness instead of syntatic correctness PARSEBENCH ~2,000 human-verified pages · 167K+ test rules · 5 capability dimensions · enterprise documents (insurance, finance, government) TABLES Merged cells, hierarchical headers, cross-page continuity Example labels: TABLE RECORD MATCH, COL KEYS, ROW KEY, VALUE [illegible embedded document text] CHARTS Exact data-point extraction with series & axis label matching Example labels: VALUE $56.0, LABEL 2024 [illegible embedded document text] CONTENT FAITHFULNESS Omissions, hallucinations, reading-order fidelity [illegible embedded document text] SEMANTIC FORMATTING Strikethrough, super/subscripts, bold, title hierarchy, links Example labels: bold, strikethrough, hierarchy [illegible embedded document text] VISUAL GROUNDING Element localization, classification, content attribution Example labels: Page-Header, Section, Table, Text, Picture, Page-Footer [illegible embedded document text] ### ParseBench Results: Overall Score vs. Cost Higher scores and lower costs lie toward the chart’s upper left. LP Agentic sits highest at a little over one cent per page; no plotted parser approaches 100%. [14:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=850s) [Scatterplot comparing document parsers and models by overall score and cost per page.](https://assets.ai.engineer/sites/aiecode2025/sha256/3772d7320d4a386f7843c961cc045f8d6a69d7f646658cffbc8e7ff02813b1fb.webp) ParseBench: An Enterprise Document Benchmark for Agents We evaluate frontier models, OSS models, specialized parse providers (including LlamaParse) No parser gets close to 100%, the benchmark is meant to be hard! Overall vs. Cost Y-axis: Overall score (%) Y-axis ticks: 50, 60, 70, 80 X-axis: Cost per page (cents) X-axis ticks: 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0 Legend: LlamaParse; VLMs; Specialized Document Parsers Data labels: LP Agentic; LP Cost-Eff; Gemini (min); Gemini (high); Reducto; Reducto (agentic); Gemini Pro; GPT-5.4; Azure DI; Extend; Haiku 4.5 (think); Opus 4.6; Google DocAI; GPT-mini (med); Textract; GPT-mini (min); Haiku 4.5; LandingAI LlamaIndex · Redefine Document Workflows with AI Agents ### The Accuracy-Cost-Latency Pareto Curve Parsing workloads favor different tradeoffs: financial extraction demands high accuracy, bulk indexing favors low cost, and real-time uploads need low latency. Liu says VLM-based approaches are often too slow for the agent loop. [15:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=906s) [Three document parsing regimes alongside a score-versus-cost chart and commentary on latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/912267f90e11317dfb9278bf6cec509c6b590f3c959497b3f01c984c82bf944d.webp) The Accuracy-Cost-Latency Pareto Curve 1. High Accuracy Regime: require ~99%+ accuracy, price insensitive • e.g. financial data room extraction 2. Low Cost Regime: require 1M docs per day, accuracy above min threshold • e.g. offline indexing a petabyte of Sharepoint data for retrieval 3. Low Latency/Low Cost Regime: require parsing massive volumes in seconds, price sensitive • e.g. real-time file uploads to Cowork VLM-based approaches are primarily good for latency-insensitive batch processing. They are not great in the agent loop. Overall vs. Cost Y-axis: Overall score (%) Y-axis ticks: 50, 60, 70, 80 X-axis: Cost per page (cents) X-axis ticks: 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0 Legend: LlamaParse; VLMs; Specialized Document Parsers Chart labels: LP Agentic; LP Cost-Eff; Gemini (min); Gemini (high); Reducto; Reducto (agentic); Gemini Pro; GPT-5.4; Azure DI; Extend; Haiku 4.5 (think); Opus 4.6; Google DocAI; GPT-mini (med); Textract; GPT-mini (min); Haiku 4.5; LandingAI LlamaIndex · Redefine Document Workflows with AI Agents ### LiteParse: Fast, Free, Surprisingly Good Parsing LiteParse v2 leads on 1- and 457-page PDFs (0.777 s for the latter); PyMuPDF leads on 24 and 60 pages. Timings use five warmup runs with OCR off. LiteParse also has the highest overall accuracy shown: 39.2%. [16:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=1018s) [LiteParse features with speed and accuracy comparison tables.](https://assets.ai.engineer/sites/aiecode2025/sha256/c838574bd51b951837d44a81615fe059d93cb6f2436674a9fe58bdd6d3209ebe.webp) LiteParse: Fast, Free, Surprisingly Good Parsing An open-source, model-free, fast document parser. ✅ Output in text + markdown ✅ Supports CLI, Python, Node, WASM ✅ One-click Skills install ✅ Tops olmOCR-bench, opendataloader-bench, ParseBench vs. other OSS parsers ✅ Outputs bounding boxes https://github.com/run-llama/liteparse speed comparison Document | V2 liteparse | V1 liteparse | pymupdf | pypdf | markitdown | pdftotext | TEXT pymupdf4llm | MD pymupdf4llm | opendataloader 1_page.pdf | • 0.002 s | 0.217 s | 0.003s | 0.012s | 0.040s | 0.003s | 0.115s | 0.113s | 0.359s 24_pages.pdf | 0.041 s | 0.371 s | • 0.029s | 0.165s | 1.084s | 0.042s | 1.436s | 1.464s | 0.701s 60_pages.pdf | 0.123 s | 0.577 s | • 0.086s | 0.919s | 7.129s | 0.092s | 10.693s | 10.633s | 2.560s 457_pages.pdf | • 0.777 s | 2.695 s | 0.809s | 5.804s | 39.335s | 1.909s | 68.882s | 68.614s | 12.386s LICENSE | Apache-2.0 | Apache-2.0 | AGPL-3.0 | BSD 3-Clause | MIT | GPL-2.0 | AGPL-3.0 | AGPL-3.0 | Apache-2.0 • Fastest result in each row TIMES REPORTED AFTER 5 WARMUP RUNS · OCR DISABLED accuracy comparison Category | LiteParse | pymupdf4llm | opendataloader | pdf-inspector | markitd[illegible] Overall | 39.2% | 32.9% | 32.7% | 30.5% | 28.7% baseline | 99.9% | 84.5% | 86.9% | 82.9% | 86.8% headers_footers | 55.9% | 39.5% | 37.5% | 52.0% | 38.8% multi_column | 67.1% | 66.7% | 62.8% | 38.2% | 39.3% table_tests | 48.0% | 46.3% | 25.7% | 40.0% | 19.9% long_tiny_text | 29.2% | 12.7% | 35.1% | 17.4% | 31.2% old_scans | 13.3% | 13.3% | 13.3% | 13.3% | 13.3% arxiv_math | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% old_scans_math | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% LlamaIndex · Redefine Document Workflows with AI Agents ### LiteParse in the Assistive Agent Loop LiteParse provides a fast first pass for agent text search; the agent can use VLM-based OCR for deeper analysis. Beside the recommendation, a browser demo shows a PDF and a parsing interface. [17:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=1052s) [LiteParse workflow benefits beside a browser-based PDF parsing demonstration.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ffe0c787d6b891de65a9eab14b9ab4ae94b7fcec13834be93d2ec3b727f2fcb.webp) LiteParse: Fast, Free, Surprisingly Good Parsing It is perfect as a default in the assistive agent loop: • Extremely fast over any set of data • Good enough that the agent can do text search • The agent can decide to “zoom-in” with a VLM-based OCR tool if it needs to do deeper analysis. Embedded PDF: The NASA Technical Report Server [illegible] Abstract [illegible] Interface text: LiteParse PDF Parsing in the Browser Fast, lightweight document parsing powered by WebAssembly. No server required — everything runs locally in your browser. [illegible] Drop a PDF here or click to browse Ready. Drop a PDF to parse Usage [illegible] LlamaIndex · Redefine Document Workflows with AI Agents ### LiteParse for Assistive Agents An agent can scan documents quickly for text search, then use slower VLM-based OCR to inspect pages with tables or charts in more depth. [18:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=1080s) [Slide describes fast parsing, text search, and deeper VLM-based OCR, alongside a browser demo.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ddd3754753f385c978523bb02be838aee113d698d3ffb8f2dd49c1bc972bb09.webp) LiteParse: Fast, Free, Surprisingly Good Parsing It is perfect as a default in the assistive agent loop: • Extremely fast over any set of data • Good enough that the agent can do text search • The agent can decide to “zoom-in” with a VLM-based OCR tool if it needs to do deeper analysis. LlamaIndex · Redefine Document Workflows with AI Agents Interface text: LiteParse PDF Parsing in the Browser [illegible] Drop a PDF here or click to browse Parsed 16 page(s) in [illegible] Markdown Wrap On Copy The NASA Technical Report Server Abstract Introduction [illegible] ### Beyond Parsing The document context layer spans parsing, extraction and search, and agentic workflows. The slide also marks agent-native formats, document versioning and long-horizon agents as unexplored areas. [18:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=1106s) [Diagram compares parsing, semantic and storage, and agentic layers, with current and unexplored capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1275bbab535ff31000748ca4020d31c51d5532244cbd831223b255c252f6604.webp) Great, what else is there besides parsing? Document Context Layer Parsing Layer Semantic and Storage Layer Agentic Layer Today Parsing Layer: Document OCR Semantic and Storage Layer: Document Extraction; Document Search Agentic Layer: Document Workflows Unexplored Parsing Layer: Agent-native Document Formats Semantic and Storage Layer: Document Versioning; Document Editing; Document Permissions and Collaboration Agentic Layer: Hillclimbing-as-a-service; Long-horizon agents LlamaIndex · Redefine Document Workflows with AI Agents ### Document Extraction Schema-based extraction turns documents into structured outputs for downstream systems. The slide highlights data entry and batch ETL, then asks whether extracted values can be traced to their source. [18:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=1136s) [Slide lists extraction goals, use cases, modalities, and grounding features beside a question about proving where extracted values came from.](https://assets.ai.engineer/sites/aiecode2025/sha256/25fa169bb010e797a32bbbadd1c284db162237fa901a46dadb72316eb7f0d2aa.webp) Document Extraction Goal: Schema-based structured extraction over any document, regardless of document complexity or schema complexity. Use Cases: • Automate human data entry • Batch ETL • Add metadata for search Extraction Modalities: • Fast Extraction • Agentic Extraction Grounding and Semantics: • Granular attribution • Calibrated Confidence Scores ...can it prove where they came from? LlamaIndex · Redefine Document Workflows with AI Agents ### Document Extraction The demo shows extraction output with page and bounding-box fields and selectable content granularity. Liu says outputs include source citations, and confidence scores can flag uncertain values before they enter a system. [19:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=1177s) [A JSON output view shows page and bounding-box fields with a menu of content granularity options.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e31177c1974ed41aa4654a4413a625e559f3952c6acea0b34fc675f17fbdf51.webp) Document Extraction Goal: Schema-based structured extraction over any document, regardless of document complexity or schema complexity. Use Cases: • Automate human data entry • Batch ETL • Add metadata for search Extraction Modalities: • Fast Extraction • Agentic Extraction Grounding and Semantics: • Granular attribution • Calibrated Confidence Scores Interface text: JSON [ { "page": 1, "bbox": { "x": 267.02, "y": 74.5, "w": 38.61, "h": 9.4 }, "span": [ 0, 6 ] }, { "page": 1, "bbox": { "x": 305.38, "y": 74.5, "w": 39.35, View options Content: Word Collapse Expand Job Layout Line Word Cell LlamaIndex · Redefine Document Workflows with AI Agents [19:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=1185s) [Document extraction goals, use cases, modalities, and grounding features beside a LlamaParse video end card.](https://assets.ai.engineer/sites/aiecode2025/sha256/aab2577e09d12cff62272b732a73bf678cb99fbfddae17bb53cfb7f3f3497d8c.webp) Document Extraction Goal: Schema-based structured extraction over any document, regardless of document complexity or schema complexity. Use Cases: • Automate human data entry • Batch ETL • Add metadata for search Extraction Modalities: • Fast Extraction • Agentic Extraction Grounding and Semantics: • Granular attribution • Calibrated Confidence Scores LlamaIndex Sign up for LlamaParse today www.llamaindex.ai Interface text: 0:51 / 0:51 1x LlamaIndex · Redefine Document Workflows with AI Agents ## Full transcript [0:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=1s) [music] [0:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=12s) >> I think we can get started. Hey [0:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=13s) everyone, I'm Jerry, co-founder and CEO [0:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=15s) of LlamaIndex, and today I'm excited to [0:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=18s) uh give a talk called building the [0:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=19s) document context layer for AI agents. Um [0:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=22s) really uh big shoutout to the AI [0:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=24s) Engineer World Fair for hosting. Um and [0:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=26s) if you've seen some of my earlier talks [0:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=28s) from the previous uh AI Engineer [0:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=30s) conferences, uh we've kind of traced [0:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=31s) through a lot of the evolution of how, [0:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=33s) you know, um agent advances have [0:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=35s) correlated with uh you know, how you [0:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=36s) inject context into evolving [0:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=38s) applications. Um and so today, in 2026, [0:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=40s) we'll kind of talk about three main [0:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=42s) topics. One is, you know, what is RAG in [0:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=44s) 2026? Um two, uh basically that [0:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=47s) basically decomposes into an agent [0:49](https://www.youtube.com/watch?v=RQi7x-navxU&t=49s) harness plus a context layer. So kind of [0:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=51s) going into the modern document context [0:53](https://www.youtube.com/watch?v=RQi7x-navxU&t=53s) layer for agents, um and giving you a [0:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=54s) little bit of sense of kind of some of [0:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=56s) the core capabilities today, um [0:57](https://www.youtube.com/watch?v=RQi7x-navxU&t=57s) especially uh unlocking, you know, the [0:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=59s) vast trove of document-based data today, [1:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=61s) and giving you a glimpse of what's next. [1:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=64s) All right. RAG in 2026. But before that, [1:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=66s) just a little bit about the company, and [1:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=67s) I'll kind of get on with the main talk. [1:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=69s) Um you might have seen us as a RAG [1:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=70s) framework. We started in 2023, got [1:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=72s) pretty popular, um kind of created a lot [1:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=74s) of techniques around like advanced RAG, [1:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=76s) that type of stuff. Um today, we're [1:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=78s) basically the main document [1:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=79s) infrastructure for AI agents. Um we [1:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=81s) deliver the best platform for agents to [1:23](https://www.youtube.com/watch?v=RQi7x-navxU&t=83s) actually read and operate over [1:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=84s) documents, and we see ourselves as [1:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=86s) unlocking basically the vast troves of [1:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=87s) document context even as agents and [1:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=89s) models themselves get better. [1:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=93s) All right. So, what does RAG actually [1:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=95s) mean in 2026, and how are agents [1:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=97s) actually inhaling and addressing context [1:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=99s) today? [1:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=100s) This is a snapshot from um actually 3 [1:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=103s) years ago, um when I first gave a talk [1:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=105s) on this, and naive RAG was basically [1:46](https://www.youtube.com/watch?v=RQi7x-navxU&t=106s) just building a simple chatbot over your [1:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=108s) private corpus of data. If you flash [1:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=110s) back all the way to January of like [1:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=111s) 2023, um you know, this technique [1:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=114s) basically consisted of you have some [1:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=116s) corpus of documents, um you chunk it up, [1:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=118s) embed it, and put it into a vector [2:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=120s) database. You do some naive top K [2:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=122s) retrieval, and then you generate some [2:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=123s) stuff with an LLM. All the steps are [2:05](https://www.youtube.com/watch?v=RQi7x-navxU&t=125s) fixed. You use kind of like a fixed set [2:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=127s) of techniques, and then with that you [2:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=129s) actually get some basic application [2:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=130s) results by just being able to chat over [2:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=132s) a corpus of documents. [2:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=134s) Of course, the agent landscape has [2:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=136s) changed quite a bit and quite [2:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=138s) dramatically since then. [2:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=139s) One is agent loops and tool use have [2:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=142s) gotten a lot a lot better, especially as [2:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=144s) both the models and agent harnesses have [2:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=146s) increased. And there's basically a [2:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=148s) cleaner separation now between agent [2:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=150s) reasoning and how they actually interact [2:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=151s) with context. If you look at what I call [2:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=154s) kind of the modern generalized agents, [2:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=156s) which includes, you know, all your [2:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=157s) favorite applications and tools out [2:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=158s) there from Claude Code, Claude Code [2:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=160s) Work, Open Claw, Codex, and a few [2:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=162s) others, [2:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=163s) you know, the retrieval complexity has [2:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=165s) started to get baked into the agent [2:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=167s) layer. So, instead of coming up with a [2:49](https://www.youtube.com/watch?v=RQi7x-navxU&t=169s) variety of hacks to really work around [2:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=171s) the limitations of naive top K [2:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=172s) retrieval, you can start getting the [2:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=174s) agent to really reason about, for [2:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=176s) instance, the best key best keyword to [2:57](https://www.youtube.com/watch?v=RQi7x-navxU&t=177s) search for, best like, you know, search [2:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=179s) term to actually get back good results. [3:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=181s) Even if the retrieval tools are basic, [3:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=183s) the agent can input the right queries to [3:05](https://www.youtube.com/watch?v=RQi7x-navxU&t=185s) basically loop upon itself and help [3:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=187s) solve the task at hand. [3:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=189s) Number two, context is moving up the [3:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=191s) stack. There's been a lot of [3:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=193s) conversations, I think, for the first [3:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=194s) two and a half years of Gen AI of how do [3:17](https://www.youtube.com/watch?v=RQi7x-navxU&t=197s) you actually manage the agent context [3:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=198s) window, make sure it doesn't overflow, [3:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=200s) that type of thing. But, I think as, you [3:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=202s) know, a lot of the evolving techniques, [3:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=204s) compaction, long contexts have evolved, [3:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=207s) I think more and more of the [3:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=208s) conversation is actually how do you just [3:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=210s) hook up the right MCP servers and skills [3:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=212s) and tasks to the agent to enable it to [3:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=215s) do various types of tasks. [3:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=217s) And so, that also applies to agent [3:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=218s) orchestration, right? Coding agents have [3:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=220s) gotten better. [3:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=222s) Abstractions of actually defining what [3:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=224s) types of tasks and programs you want to [3:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=225s) build have moved a little bit upwards [3:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=227s) towards English as opposed to through [3:49](https://www.youtube.com/watch?v=RQi7x-navxU&t=229s) code. And so, maybe in 2023 to 2025, you [3:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=232s) define programs and you still build [3:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=234s) stuff via like importing Python, using [3:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=236s) TypeScript through code. It's pretty [3:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=238s) clear these days more and more people [4:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=240s) are just building stuff using English, [4:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=242s) whether you're a software engineer or [4:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=244s) you're a non-technical function like [4:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=246s) go-to-market marketing. And you're [4:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=248s) defining runbooks through English and [4:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=250s) kind of like defining the right goals [4:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=252s) and making sure the AI is aligned on the [4:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=254s) right task. And we see that carrying [4:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=256s) over going forward as well. [4:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=259s) So, in terms of just like general [4:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=261s) evolving knowledge work patterns, and [4:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=262s) this kind of forms a foundation for what [4:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=264s) we how we think about the context layer. [4:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=266s) You know, [4:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=267s) in the past maybe you could use AI to [4:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=269s) just like ask simple questions, get it [4:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=271s) resolved. [4:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=272s) Today, you're starting to be able to [4:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=274s) define and solve more end-to-end like [4:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=276s) tasks just through English and also use [4:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=279s) English to start [4:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=280s) compiling repeatable programs to [4:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=283s) actually execute something at scale. [4:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=285s) Obviously, there's been a lot of [4:46](https://www.youtube.com/watch?v=RQi7x-navxU&t=286s) discussion on how this will evolve even [4:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=288s) more, and we do see AI agents as [4:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=290s) behaving even more autonomously through [4:53](https://www.youtube.com/watch?v=RQi7x-navxU&t=293s) solving long horizon tasks, looping upon [4:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=295s) themselves to be able to achieve a goal. [4:57](https://www.youtube.com/watch?v=RQi7x-navxU&t=297s) So, the future will move towards a state [4:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=299s) where you actually might not even have [5:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=300s) to define the task in English, but [5:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=302s) actually more of the goal and a scoring [5:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=304s) rubric, and then the agent will use all [5:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=306s) available context available to it to [5:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=308s) actually solve the task. [5:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=310s) So, with that overall framing, let's [5:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=313s) talk about what kind of what is a pretty [5:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=315s) core focus for us as a company, which is [5:17](https://www.youtube.com/watch?v=RQi7x-navxU&t=317s) basically helping to unlock unstructured [5:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=319s) context to basically feed into these [5:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=321s) agents even as they get really, really [5:23](https://www.youtube.com/watch?v=RQi7x-navxU&t=323s) good both in terms of the core model [5:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=324s) capabilities as well as the agent [5:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=326s) harness. And we'll focus a lot on our [5:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=328s) core focus area, which is document [5:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=331s) context, but of course there's plenty of [5:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=332s) other sources of context as well. In [5:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=334s) fact, we do think context really is [5:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=337s) everything. [5:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=338s) In the end, you could have a infinitely [5:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=340s) smart agents, but your ability to [5:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=342s) actually get value out of this [5:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=344s) infinitely AI agent um, able to do more [5:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=347s) and more stuff end to end, is actually, [5:49](https://www.youtube.com/watch?v=RQi7x-navxU&t=349s) you know, giving it the right things to [5:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=351s) do. Um, and the right things to do [5:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=352s) include the actual task, uh, the goal, [5:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=355s) um, but also access to the vast troves [5:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=358s) of organizational context available to [6:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=360s) it. Whether that is web search, web [6:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=362s) context, whether that is connectors [6:05](https://www.youtube.com/watch?v=RQi7x-navxU&t=365s) through, you know, like tools and skills [6:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=366s) and MCP servers, whether it's connectors [6:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=369s) through Snowflake or Databricks [6:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=370s) warehouses, um, or for us, whether it's [6:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=372s) access to the vast trove, like the 90% [6:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=375s) of documents that are stored within [6:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=376s) SharePoint, Box, Dropbox, S3, um, you [6:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=380s) know, all the unstructured, uh, data [6:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=381s) that's locked up within document [6:23](https://www.youtube.com/watch?v=RQi7x-navxU&t=383s) containers. Um, and so for us, you know, [6:25](https://www.youtube.com/watch?v=RQi7x-navxU&t=385s) we care a lot about how do you actually [6:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=387s) build the right tools and unlock the [6:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=389s) right context here. [6:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=391s) We see documents as universal containers [6:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=394s) for unstructured context. Uh, there's [6:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=395s) over 10 trillion plus pages of human [6:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=397s) native knowledge locked up within PDFs, [6:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=399s) PowerPoints, Word documents, and Excel [6:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=401s) sheets. And at the same time, agents are [6:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=403s) also starting to generate exponentially [6:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=405s) more data in terms of, you know, more [6:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=407s) agent native formats in terms of [6:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=408s) markdown and HTML. [6:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=412s) For us, building an agent native [6:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=414s) document platform contains three main [6:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=416s) pieces. Uh, it contains the document [6:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=418s) parsing layer, actually digitalizing, [7:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=421s) um, the vast troves of PDFs, [7:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=422s) PowerPoints, and Word documents into [7:05](https://www.youtube.com/watch?v=RQi7x-navxU&t=425s) kind of accurate token efficient [7:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=427s) context, um, and feeding it, you know, [7:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=429s) either markdown, metadata, and other [7:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=431s) forms that actually enable agents to do [7:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=433s) stuff over your documents. [7:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=435s) The next is actually what we call the [7:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=436s) semantic and storage layer, um, which is [7:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=439s) kind of like this concept of document [7:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=441s) management for humans and agents. [7:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=442s) Instead of humans opening up Microsoft [7:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=444s) Word, using your file system or file [7:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=446s) storage, how do you create some sort of [7:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=448s) agent interface that actually captures, [7:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=450s) stores all your documents, and enables [7:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=452s) agents to actually manage and operate [7:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=454s) over them in various ways. Um, and then [7:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=457s) there's still a room for actually kind [7:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=459s) of repeatable document workflows is [7:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=461s) agent layer they can actually encode [7:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=462s) within the [7:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=464s) uh [7:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=464s) within like this type of platform, too. [7:46](https://www.youtube.com/watch?v=RQi7x-navxU&t=466s) If there is a repeatable workflow, like [7:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=468s) invoice processing, KYC, or claims, [7:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=471s) instead of always offloading it to a [7:53](https://www.youtube.com/watch?v=RQi7x-navxU&t=473s) generalized agent, how do you actually [7:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=474s) develop some sort of specialized [7:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=476s) workflow for it and really carefully [7:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=478s) tune cost and accuracy to make sure that [8:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=480s) you achieve your goal. [8:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=483s) So, when we talk about like the concepts [8:05](https://www.youtube.com/watch?v=RQi7x-navxU&t=485s) today in in terms of these three layers, [8:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=487s) uh we'll talk about this concept of [8:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=489s) document OCR, which is one of the core [8:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=491s) concepts of this track, um to document [8:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=493s) extraction, document search, and [8:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=494s) document workflows. There's still a [8:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=496s) variety of topics yet left unsolved from [8:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=498s) agent native document formats, document [8:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=500s) versioning, document editing, you know, [8:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=502s) hill climbing as a service, and just [8:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=504s) like a wide set of uh kind of remaining [8:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=507s) concepts to actually create a [8:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=508s) comprehensive piece of software where [8:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=510s) agents and humans can collaborate on [8:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=512s) documents and actually do work over [8:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=513s) them. [8:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=515s) So, we'll start with the first section [8:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=517s) um and maybe start kind of with uh the [8:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=519s) document OCR RP. [8:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=521s) Um first off, uh you know, document OCR [8:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=524s) is hard. Um some of you might have seen [8:46](https://www.youtube.com/watch?v=RQi7x-navxU&t=526s) a blog post that we put out a few months [8:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=528s) ago related to this topic. But, the [8:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=530s) reason this problem even exists is an [8:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=532s) agent cannot actually take the like the [8:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=534s) raw PDF like file binary and make sense [8:57](https://www.youtube.com/watch?v=RQi7x-navxU&t=537s) of it. Um that's because the like the [8:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=539s) way PDFs are actually stored as a format [9:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=542s) is it's rendered for kind of like [9:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=543s) display purposes um and not really for [9:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=546s) uh you know, machine consumption in an [9:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=548s) interpretable manner. [9:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=549s) Um [9:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=550s) they're designed for printing. So, [9:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=552s) basically text are represented as almost [9:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=554s) like individual glyphs with coordinates. [9:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=556s) Um tables are not represented as tables, [9:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=559s) they're represented as like typically [9:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=561s) line segments uh drawn with various [9:23](https://www.youtube.com/watch?v=RQi7x-navxU&t=563s) types of borders, and also text drawn at [9:25](https://www.youtube.com/watch?v=RQi7x-navxU&t=565s) certain cell positions. And so, if [9:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=568s) you're kind of an agent that's trying to [9:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=569s) make sense of this like document, you're [9:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=571s) going to have a really hard time [9:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=573s) actually trying to reason about what [9:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=574s) character and what shapes like map to [9:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=576s) what. Um so the whole point of document [9:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=578s) OCR, um, you know, it's been around for [9:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=580s) like 20 plus years, is to really try to [9:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=583s) create some sort of, uh, digitalized [9:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=585s) well-interpretable, uh, representation [9:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=587s) um, that's both interpretable to humans, [9:49](https://www.youtube.com/watch?v=RQi7x-navxU&t=589s) um, as well as AI agents. [9:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=591s) Also, reading order itself, if you have [9:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=592s) a multi-column layout, there is no, uh, [9:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=595s) guarantee that the way it's represented [9:57](https://www.youtube.com/watch?v=RQi7x-navxU&t=597s) in the PDF, um, actually corresponds to [9:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=599s) the typical ways that humans would read [10:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=601s) it. Cuz again, it's basically just an [10:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=602s) arbitrary sequence of characters drawn [10:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=604s) with uh, coordinate positions. [10:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=607s) Related to this, you know, even Word [10:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=609s) doc, uh, parsing is hard. Um, they're a [10:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=611s) little bit more structured than PDFs, [10:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=612s) but they're in kind of like this, uh, [10:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=614s) custom bespoke XML format, and this [10:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=616s) applies to PowerPoints as well. Um, it [10:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=619s) contains more structural information, [10:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=620s) but still there's a ton of like fluff. [10:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=622s) Like you don't actually need to ingest [10:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=624s) all the tags to actually have the agent [10:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=626s) make sense of the Word document. It [10:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=628s) still needs to infer a bunch of [10:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=629s) structure from it. It needs like the [10:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=631s) right, uh, kind of to lift the right [10:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=633s) metadata around like formatting, [10:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=635s) semantics, that type of stuff. But also [10:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=637s) being able to ignore the tags and [10:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=638s) actually be able to render the document [10:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=640s) so that the agent can see the overall [10:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=642s) structure of the page. Um, it's still [10:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=644s) generally hard problem, and there's a [10:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=645s) lot of uplift you can get by actually [10:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=647s) parsing it into a more interpretable [10:49](https://www.youtube.com/watch?v=RQi7x-navxU&t=649s) format instead of just using the native, [10:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=651s) uh, OXML and feeding that to an agent. [10:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=655s) Um, if you're familiar with document [10:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=656s) understanding, um, you know, it's been [10:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=658s) around for quite a bit of time. Um, [11:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=660s) there's a lot of these like heuristic [11:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=662s) and pipeline-based approaches, which, [11:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=664s) uh, focused on kind of more, uh, I guess [11:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=666s) like human-driven, hand-handwritten like [11:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=668s) techniques to analyze like kind of [11:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=670s) various pieces of text, group them into [11:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=672s) clusters, and identify tables, [11:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=674s) paragraphs, and be able to kind of like, [11:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=676s) uh, generate some sort of output [11:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=678s) representation. Um, a lot of basic [11:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=680s) techniques if you use open-source [11:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=681s) libraries like PyPDF, PyMuPDF, um, and [11:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=684s) of course like some of the, [11:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=686s) uh, more recent approaches, um, [11:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=687s) basically use this type of approach. [11:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=689s) There's also, of course, using a VLM to [11:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=691s) one-shot a document um, into text, uh, [11:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=694s) that's what we call like a vision-based [11:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=696s) approach. [11:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=697s) This works decently well. Obviously, you [11:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=699s) know, I think there's kind of a lot of [11:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=701s) uplift you can get by being able to read [11:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=702s) the visual structure, but there's a lot [11:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=705s) of sub-optimal pieces about it. It can [11:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=708s) hallucinate on text-only pages. It costs [11:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=710s) a ton of money, [11:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=712s) and also it still lacks a lot of the [11:53](https://www.youtube.com/watch?v=RQi7x-navxU&t=713s) semantics and grounding that you [11:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=715s) typically expect with some sort of [11:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=716s) document processing tool. [11:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=718s) And so for us, we really think about [12:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=720s) combining both the pipeline-based [12:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=722s) approaches of deeply understanding the [12:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=724s) file containers and binaries with the [12:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=726s) vision-based approaches to help [12:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=728s) generate, you know, kind of [12:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=730s) a hybrid approach that we think is at [12:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=731s) the Pareto frontier of cost and [12:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=732s) accuracy. [12:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=735s) A quick note on this, I'll probably just [12:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=738s) like skim the high level, but the Pareto [12:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=740s) frontier for document OCR [12:23](https://www.youtube.com/watch?v=RQi7x-navxU&t=743s) will always be like much more accurate [12:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=744s) and cheap compared to the Pareto [12:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=746s) frontier for, you know, wherever the [12:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=747s) frontier models are in terms of document [12:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=749s) understanding. It's because it's a very [12:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=751s) specific data type, and there's always [12:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=753s) ways to kind of like distill the latest [12:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=755s) visual understanding capabilities from, [12:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=757s) you know, Gemini, GPT, [12:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=759s) Opus into kind of a carefully tailored [12:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=762s) workflow that's able to process your [12:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=763s) documents at scale and in a highly [12:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=765s) accurate manner. [12:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=767s) To some extent, that's exactly what we [12:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=768s) do. You know, we both optimize [12:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=770s) underlying like PDF engines plus like [12:53](https://www.youtube.com/watch?v=RQi7x-navxU&t=773s) Word, PowerPoint, and others. [12:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=775s) We have an agentic harness that's like [12:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=776s) carefully tuned for auto routing between [12:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=779s) cheaper specialized models to frontier [13:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=781s) models, [13:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=782s) and also kind of specialized fine-tuned [13:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=784s) document VLMs that are parameter [13:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=786s) efficient and focus on specific classes [13:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=788s) of documents, [13:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=789s) elements like tables, charts, and [13:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=791s) others. [13:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=794s) That's our commercial service called [13:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=796s) LlamaParse, which I'll talk about in a [13:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=798s) bit. But I think in general, you know, [13:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=800s) we're extremely committed to advancing [13:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=802s) the frontier of just document [13:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=804s) understanding cuz basically if you're [13:25](https://www.youtube.com/watch?v=RQi7x-navxU&t=805s) within an enterprise organization and [13:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=807s) you have a massive long tail of [13:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=809s) documents across financial services, [13:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=811s) insurance, manufacturing, legal, [13:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=813s) government, and a bunch of others, [13:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=816s) there's just a lot of complexity in a [13:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=817s) lot of these document types. And if [13:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=819s) you're actually trying to unlock context [13:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=821s) at scale, [13:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=822s) most of the models are not up for the [13:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=823s) task. And so, we've created this thing [13:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=825s) called ParseBench, which is a [13:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=827s) comprehensive enterprise document [13:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=828s) benchmark for agents. We we think about [13:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=831s) it as the most comprehensive enterprise [13:53](https://www.youtube.com/watch?v=RQi7x-navxU&t=833s) document benchmark. You know, it [13:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=834s) contains 2,000 human-verified pages. It [13:57](https://www.youtube.com/watch?v=RQi7x-navxU&t=837s) measures tables, charts, content [13:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=839s) faithfulness, semantic formatting. [14:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=841s) And it's optimized for [14:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=843s) just how AI agents are actually able to [14:05](https://www.youtube.com/watch?v=RQi7x-navxU&t=845s) understand these documents instead of [14:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=847s) like syntactic correctness. [14:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=849s) We've If you look at parsebench.ai, [14:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=851s) which is, you know, kind of the it's a [14:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=853s) fully public page on the internet. It's [14:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=855s) also available on Hugging Face and [14:17](https://www.youtube.com/watch?v=RQi7x-navxU&t=857s) Kaggle. [14:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=858s) We benchmark probably like 50 different [14:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=860s) frontier models, open weight models, [14:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=862s) specialized OCR solutions. And you [14:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=864s) really want the Pareto curve to kind of [14:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=866s) be towards the left and up in terms of [14:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=869s) accuracy, like extremely high accuracy, [14:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=871s) but also extremely low cost. And there's [14:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=874s) just so many different types of [14:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=875s) documents, where some are a little bit [14:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=877s) simpler, maybe some are a little bit [14:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=879s) more complex. And ideally, you want to [14:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=881s) cover all the points on the curve to [14:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=883s) deal with the dynamic distribution of [14:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=885s) various types of documents out there. [14:46](https://www.youtube.com/watch?v=RQi7x-navxU&t=886s) It's pretty clear, even if you look at [14:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=888s) this graph, that it's [14:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=890s) like you can increase the complexity of [14:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=892s) the benchmark, and that document [14:53](https://www.youtube.com/watch?v=RQi7x-navxU&t=893s) understanding is definitely not a 100% [14:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=895s) solved. But, you know, you fundamentally [14:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=898s) need to kind of advance a lot of the [15:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=900s) core capabilities to make sure they're [15:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=901s) able to process, unlock the vast trove [15:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=903s) of enterprise context out there. [15:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=908s) There's kind of like a few different [15:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=910s) points on this accuracy cost latency [15:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=912s) Pareto curve. There's what I call like [15:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=914s) the high accuracy regime, where like, [15:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=916s) you know, some institutions basically [15:17](https://www.youtube.com/watch?v=RQi7x-navxU&t=917s) need like 99 to almost 100% accuracy, [15:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=920s) because basically, the downside of an [15:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=922s) incorrect extraction is you completely [15:23](https://www.youtube.com/watch?v=RQi7x-navxU&t=923s) mess up your financial model, you [15:25](https://www.youtube.com/watch?v=RQi7x-navxU&t=925s) completely, you know, you basically get [15:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=927s) flagged for fraud or a bunch of other [15:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=929s) really really bad things. [15:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=930s) In regulated industries like insurance [15:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=932s) and financial services, we see this a [15:34](https://www.youtube.com/watch?v=RQi7x-navxU&t=934s) decent amount. This typically means [15:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=936s) you're willing to pay a little bit more [15:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=937s) money per page for like deeper agent [15:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=939s) tech reasoning to at least make sure [15:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=941s) that you get back the the information in [15:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=943s) the right format. There's also like the [15:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=945s) low cost regime. Let's say you're just [15:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=947s) trying to index, you know, the million [15:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=948s) plus documents per day within your [15:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=951s) that's you know, being continually [15:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=952s) updated within your SharePoint just for [15:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=954s) like rag knowledge base search. You [15:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=956s) know, in these cases, you obviously want [15:57](https://www.youtube.com/watch?v=RQi7x-navxU&t=957s) it to be not like terribly inaccurate, [15:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=959s) but even if it's a little bit messed up, [16:01](https://www.youtube.com/watch?v=RQi7x-navxU&t=961s) it's okay, too. Because in the end if [16:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=963s) you have a sufficiently good agent, it [16:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=964s) can always dive deeper into the document [16:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=966s) and surface the right information with [16:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=968s) the right citations and grounding. [16:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=970s) So for these, you know, being able to [16:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=971s) create some sort of scalable offline [16:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=973s) indexing pipeline that has the best like [16:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=975s) cost constraints [16:17](https://www.youtube.com/watch?v=RQi7x-navxU&t=977s) is something that is optimal. [16:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=979s) One thing about VLMs based approaches [16:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=981s) though is that, you know, they're [16:23](https://www.youtube.com/watch?v=RQi7x-navxU&t=983s) typically not very fast. And I think a [16:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=986s) lot of times if you have like real-time [16:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=988s) file uploads, let's say you're using [16:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=989s) Cloud Co-work, you upload a thousand [16:31](https://www.youtube.com/watch?v=RQi7x-navxU&t=991s) documents and you need to process it [16:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=993s) within a minute a minute, like having a [16:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=995s) bunch of VLMs process that at scale is [16:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=998s) really tough for basically every single [16:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=999s) OCR service out there. And and to be [16:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=1001s) fair, that includes ours, too. I think [16:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=1003s) in general, there's also some sort of [16:46](https://www.youtube.com/watch?v=RQi7x-navxU&t=1006s) need for an extremely low latency [16:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=1008s) solution so that you can actually [16:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=1010s) process stuff in real-time even if you [16:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=1012s) have like deeper VLM enabled processing [16:55](https://www.youtube.com/watch?v=RQi7x-navxU&t=1015s) for kind of like deeper visual [16:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=1016s) inspection and analysis. [16:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=1018s) And so that's what I call kind of being [17:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=1020s) in the agent loop. So besides Lama [17:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=1022s) Parse, which is kind of our commercial [17:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=1023s) service around like document processing [17:05](https://www.youtube.com/watch?v=RQi7x-navxU&t=1025s) and extraction, we also created this [17:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=1027s) tool called Light Parse. [17:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=1029s) It is surprisingly really really good. I [17:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=1031s) don't know if you've been following some [17:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=1032s) of the Twitter threads, but it is [17:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=1035s) Rust-based. It is the fastest [17:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=1036s) open-source parser out there. It is [17:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=1038s) completely free, um, and there is [17:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=1040s) basically no strings attached. I think [17:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=1041s) it's like MIT or Apache license. Um, and [17:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=1044s) it basically is the most accurate like [17:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=1046s) markdown parser out there that doesn't [17:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=1048s) use a VLM or any sort of kind of like [17:29](https://www.youtube.com/watch?v=RQi7x-navxU&t=1049s) deeper model. Um, and so this is kind of [17:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=1053s) nice because you can use it as a default [17:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=1055s) in the assistive agent loop. Um, let's [17:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=1058s) say you're uploading a a bunch of [17:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=1059s) documents to Quad Code, Quad Code Work, [17:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=1061s) Codex, and you want it to process like a [17:43](https://www.youtube.com/watch?v=RQi7x-navxU&t=1063s) thousand PDFs extremely quickly. You can [17:46](https://www.youtube.com/watch?v=RQi7x-navxU&t=1066s) always do that, um, and then, you know, [17:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=1068s) equip a VLM-based parser like Llama [17:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=1070s) Parser or other frontier models as a [17:51](https://www.youtube.com/watch?v=RQi7x-navxU&t=1071s) tool. So, what these agents will do is [17:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=1074s) they'll do like a fast pass over all the [17:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=1076s) documents first, uh, uh, kind of like [17:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=1078s) just scan through all the context [18:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=1080s) extremely efficiently, um, and then if [18:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=1082s) actually needs to dive into a page with [18:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=1084s) like tables, with like charts, and [18:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=1086s) actually needs to more deeply understand [18:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=1087s) the values, um, it will use a VLM-based [18:09](https://www.youtube.com/watch?v=RQi7x-navxU&t=1089s) tool, slower processing, to actually [18:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=1091s) make sure it reads the information [18:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=1092s) correctly. This is available as a [18:14](https://www.youtube.com/watch?v=RQi7x-navxU&t=1094s) one-click installable skill, um, [18:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=1096s) complements kind of any other deeper [18:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=1098s) VLM-based OCR tool you want to use, um, [18:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=1100s) and we kind of designed to make it as [18:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=1102s) fast as possible and also easy to plug [18:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=1104s) in to your favorite AI agent. [18:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=1108s) So, I kind of speed ran through a bunch [18:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=1110s) of this stuff, but basically, you know, [18:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=1112s) uh, we spent a bunch of time on the, uh, [18:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=1115s) parsing layer. There's also other, um, [18:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=1117s) general components around like the [18:38](https://www.youtube.com/watch?v=RQi7x-navxU&t=1118s) semantic and storage layer in terms of [18:40](https://www.youtube.com/watch?v=RQi7x-navxU&t=1120s) document extraction and search, and of [18:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=1122s) course like document workflows. Um, and [18:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=1125s) due to time, I'll probably kind of just, [18:46](https://www.youtube.com/watch?v=RQi7x-navxU&t=1126s) uh, skip some of the, uh, unexplored [18:48](https://www.youtube.com/watch?v=RQi7x-navxU&t=1128s) areas like agent native document [18:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=1130s) formats, hill climbing as a service, and [18:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=1132s) others, but I'll share the full set of [18:53](https://www.youtube.com/watch?v=RQi7x-navxU&t=1133s) slides online. In terms of the semantic [18:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=1136s) and storage layer, you know, besides [18:57](https://www.youtube.com/watch?v=RQi7x-navxU&t=1137s) document parsing, um, a lot of use cases [19:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=1140s) also require actually getting back, uh, [19:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=1142s) structured information at scale from [19:04](https://www.youtube.com/watch?v=RQi7x-navxU&t=1144s) documents. Whether you're processing, [19:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=1146s) you know, a million invoices or expense [19:07](https://www.youtube.com/watch?v=RQi7x-navxU&t=1147s) reports or receipts or claims, you need [19:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=1150s) to make sure that, you know, you want to [19:11](https://www.youtube.com/watch?v=RQi7x-navxU&t=1151s) actually get back structured outputs [19:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=1153s) that you can put into a downstream [19:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=1155s) database or system. [19:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=1156s) And so, a lot of these use cases [19:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=1158s) basically revolve around the form of, [19:20](https://www.youtube.com/watch?v=RQi7x-navxU&t=1160s) you know, how do you automate a lot of [19:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=1162s) workloads that humans typically do in [19:24](https://www.youtube.com/watch?v=RQi7x-navxU&t=1164s) scanning a lot of paperwork and doing [19:26](https://www.youtube.com/watch?v=RQi7x-navxU&t=1166s) data entry. Whether it is kind of, [19:28](https://www.youtube.com/watch?v=RQi7x-navxU&t=1168s) again, invoices, claims, contracts, [19:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=1170s) receipts, or others. [19:32](https://www.youtube.com/watch?v=RQi7x-navxU&t=1172s) We kind of created these capabilities [19:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=1173s) within Llama Parse as well. [19:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=1175s) A lot of our capabilities are actually [19:37](https://www.youtube.com/watch?v=RQi7x-navxU&t=1177s) tuned towards like low cost while [19:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=1179s) extremely high accuracy. And you get [19:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=1182s) back granular citations all the way back [19:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=1184s) to the source document for every [19:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=1185s) extracted output. And of course, you can [19:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=1187s) run this in a pipeline at scale with [19:49](https://www.youtube.com/watch?v=RQi7x-navxU&t=1189s) confidence scores [19:50](https://www.youtube.com/watch?v=RQi7x-navxU&t=1190s) and also, you know, being able to [19:52](https://www.youtube.com/watch?v=RQi7x-navxU&t=1192s) actually flag whether or not we're [19:54](https://www.youtube.com/watch?v=RQi7x-navxU&t=1194s) certain about a certain value before [19:56](https://www.youtube.com/watch?v=RQi7x-navxU&t=1196s) deciding to put it into some sort of [19:58](https://www.youtube.com/watch?v=RQi7x-navxU&t=1198s) system. [20:00](https://www.youtube.com/watch?v=RQi7x-navxU&t=1200s) There's also document search, which is a [20:02](https://www.youtube.com/watch?v=RQi7x-navxU&t=1202s) basically expanded tool set as I [20:03](https://www.youtube.com/watch?v=RQi7x-navxU&t=1203s) mentioned around retrieval, BM25, grep, [20:06](https://www.youtube.com/watch?v=RQi7x-navxU&t=1206s) vector search, reading, and scrolling. [20:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=1208s) And so, all these capabilities are [20:10](https://www.youtube.com/watch?v=RQi7x-navxU&t=1210s) available within some of our commercial [20:12](https://www.youtube.com/watch?v=RQi7x-navxU&t=1212s) platforms as well as open source [20:13](https://www.youtube.com/watch?v=RQi7x-navxU&t=1213s) offerings. But I also just wanted to [20:15](https://www.youtube.com/watch?v=RQi7x-navxU&t=1215s) paint a picture of the general concepts [20:16](https://www.youtube.com/watch?v=RQi7x-navxU&t=1216s) out there today. [20:18](https://www.youtube.com/watch?v=RQi7x-navxU&t=1218s) So, I'll skip this section about kind of [20:19](https://www.youtube.com/watch?v=RQi7x-navxU&t=1219s) what's next and then maybe just go all [20:22](https://www.youtube.com/watch?v=RQi7x-navxU&t=1222s) the way to the end. I know I'm a little [20:23](https://www.youtube.com/watch?v=RQi7x-navxU&t=1223s) bit over time. So, really appreciate you [20:25](https://www.youtube.com/watch?v=RQi7x-navxU&t=1225s) all spending time today and then let me [20:27](https://www.youtube.com/watch?v=RQi7x-navxU&t=1227s) just how do I get to this part really [20:30](https://www.youtube.com/watch?v=RQi7x-navxU&t=1230s) quick? Oh, right. [20:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=1233s) I'm going to skip this piece. I'll put [20:35](https://www.youtube.com/watch?v=RQi7x-navxU&t=1235s) put this online. [20:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=1236s) Our booth is at LG 47. If you guys are [20:39](https://www.youtube.com/watch?v=RQi7x-navxU&t=1239s) interested in stopping by and we're [20:41](https://www.youtube.com/watch?v=RQi7x-navxU&t=1241s) hosting a giant pickleball tournament [20:42](https://www.youtube.com/watch?v=RQi7x-navxU&t=1242s) today. So, thank you for your time. [20:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=1245s) >> [applause]