# From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto Adit Abraham, Reducto AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/0I07YAuF8xA-from-ingestion-agents-ai-teams-build-on) · [Video](https://www.youtube.com/watch?v=0I07YAuF8xA) ## From Ingestion to Agents: How AI Teams Build on Document Intelligence Adit Abraham of Reducto explains how to turn visually encoded documents into useful agent inputs: combine specialized models, correct OCR without rewriting the source, separate retrieval from reasoning, and give difficult extraction tasks tools and a verification loop. ### When documents become inputs to decisions [0:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=13s) A bad document parse can spoil a chatbot answer. In an agent workflow, the same mistake can travel through several decisions and into the finished work. That is the practical problem behind Adit Abraham’s talk. As Reducto’s co-founder and CEO, he draws on processing many billions of customer documents, including historical material that enterprises have struggled to use beyond a demo. ![Source frame: When documents become inputs to decisions](https://ai.engineer/api/talk-essay-media/ed4cd922b0067fe2526b7306d76e810fc785d24edd57d82e4ad66b16a7e2d7b4) Source frame: When documents become inputs to decisions The earlier retrieval-augmented generation pattern mostly synthesized information: retrieve context, then answer a question through enterprise search or a chatbot. Agents expand the job. They make decisions, produce work, and generate or modify documents. Input quality matters throughout that longer sequence because each step can carry forward an earlier misunderstanding while adding information from another source. Enterprise information also arrives without a tidy inventory. Different teams put files in Google Drive, Box, and other repositories; the contents are scattered, unstructured, and multimodal. A useful document system therefore has several jobs: recover the contents accurately, retrieve the relevant context, and support the interactions or modifications the task requires. Parsing is the first problem, but it does not settle the others. ### PDFs preserve appearance; agents need meaning [5:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=328s) PDFs were designed to reproduce a document’s appearance faithfully and make it printable. The present task asks for something different: a representation an agent can reason over. Abraham introduces the difficulty with a document-decision benchmark on which a frontier model scored about 30%. That figure describes the benchmark he cites, rather than a general PDF accuracy rate. ![Source frame: PDFs preserve appearance; agents need meaning](https://ai.engineer/api/talk-essay-media/9c58e2c327b52acdc4f84ea71e0ce2fba9f511756495cab0c01d191bfedac8b7) Source frame: PDFs preserve appearance; agents need meaning Humans encode meaning visually. A financial deck’s author is thinking about readers, rather than whether an agent can consume the slide. Abraham’s memorable example is SoftBank’s imagery of a goose laying eggs: a picture can carry part of the explanation. More routinely, merged cells establish relationships in tables, lines encode numerical trends, and handwriting holds information that even a human may struggle to read. Extracting characters alone can leave those relationships behind. ### Use vision models selectively, then correct the transcription [7:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=443s) A traditional document pipeline ran OCR and then processed the resulting text. That approach could work well when the layout stayed consistent: a W-2 presents a constrained problem that templates can address. **Vision-language models (VLMs)** widened the possibilities by reading visual content across unfamiliar layouts, with handwriting as a particularly useful case. Their generality makes the long tail more approachable. ![Source frame: Use vision models selectively, then correct the transcription](https://ai.engineer/api/talk-essay-media/ff7056d486d12cd2756679e24651ba8e84b013f4d4677e67a44368bc46267031) Source frame: Use vision models selectively, then correct the transcription At hundreds of millions of documents, however, efficiency and determinism also shape the design. The work can be divided between complementary capabilities: - **Layout detection:** Small computer-vision models identify document regions and can run on CPUs at scale. They help locate the difficult parts before more expensive processing. - **Semantic interpretation:** VLMs handle challenging visual content and help identify mistakes that simpler processing leaves behind. This division gives each task a suitable tool instead of making every region depend on a large foundation model. Once the page has been segmented and difficult regions located, a verification layer can play a role similar to a human reviewer. Reducto calls this agentic OCR. The important decision is to apply targeted token edits to an initial transcription, rather than ask a model to regenerate the whole page. Abraham compares the approach to fast edits in an IDE: preserve the existing output and change the small pieces that need correction. Why restrict an intelligent model’s freedom? Consider a table with a total that its human author calculated incorrectly. A model rewriting the OCR can recognize the word “total,” add the entries, and replace the printed value with the correct arithmetic result. The output becomes less faithful precisely because the model understood the table. The desired correction is narrower: distinguish `0` from `O`, or a period from a comma, while retaining the source’s own mistake. Reading a document and checking its arithmetic are separate jobs. ### The table needs different forms for retrieval and reasoning [10:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=645s) A correct parse still needs a useful representation. Simple tables fit naturally in Markdown. Complex tables may need HTML because merged cells encode relationships that must survive conversion. HTML preserves that structure, but its tags consume tokens; applying it to every simple table adds overhead without a corresponding benefit. Reducto chooses the representation according to the table’s complexity. ![Source frame: The table needs different forms for retrieval and reasoning](https://ai.engineer/api/talk-essay-media/838aeb5cf1ba47bbd0efee282557929dae5b0a76e282a89c5f831c1ede7ff21d) Source frame: The table needs different forms for retrieval and reasoning Now follow that complex table into retrieval. A user asks, “how did revenue change over time,” without enumerating the table’s values. The reasoning model may understand the HTML perfectly once it receives it. The embedding model first has to find it among many documents, though, and Reducto finds that matching an ordinary language query to a mass of tags and numbers is difficult. A representation that serves reasoning can therefore obstruct the step that supplies the reasoning context. The change is to create a second, natural-language rendering of the same table. Retrieval uses that block; reasoning uses the structured HTML. What does each consumer receive? The diagram makes the division visible: the retrieval-friendly text helps select the relevant table, while the reasoning-friendly form preserves its relationships. This addresses the matching problem without requiring the reasoning model to work from the natural-language rendering alone. Two representations of one complex table Natural-language content serves retrieval; HTML structure serves reasoning over the selected table. - Complex table — Merged cells carry meaning. - Natural-language rendering — Content formatted for the embedding model. - Retrieve relevant context — Match the human query to the table. - HTML table — Preserve the complex table’s structure. - Reason over the table — Pass the HTML representation to the language model. - Complex table → Natural-language rendering (Render in words) - Natural-language rendering → Retrieve relevant context (Use for retrieval) - Complex table → HTML table (Encode structure) - HTML table → Reason over the table (Use for reasoning) - Retrieve relevant context → Reason over the table (Supply the relevant table) ### Better inputs reduce reconstruction work; routing reduces distraction [13:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=792s) Returning to the document-decision benchmark, Abraham reports that supplying structured parse results alongside the original PDF improved answers across models from Google, Anthropic, and OpenAI. Some tested models surpassed the earlier frontier model’s out-of-the-box result. The comparison does not establish what that earlier model would do with the same improved inputs: Reducto could not test it after losing access. ![Source frame: Better inputs reduce reconstruction work; routing reduces distraction](https://ai.engineer/api/talk-essay-media/50e052a0aa39b192802455ddd0c78450a6c7087afec6c02cc56ae1d545b2d2b6) Source frame: Better inputs reduce reconstruction work; routing reduces distraction The reported benefit also included fewer reasoning tokens and lower latency. Structured inputs let the model spend less effort reconstructing the data and more effort producing the answer. Improving the input pipeline can thus change both the quality and the amount of downstream work, without changing the model itself. The next improvement concerns which documents enter the task. A large context window does not make every page helpful; Abraham describes quality erosion from excessive context as well as token cost. Two operations address different parts of that problem: - **Classification:** Route the right document to the appropriate processing pipeline. - **Splitting:** Select the relevant portions of a large document instead of passing the entire packet. A paper-mail packet can run to hundreds of pages with uncertain contents. Making the final decision model sort through that packet adds a separate job that distracts from extracting information, reasoning over it, or making the intended decision. ### A line chart becomes a table through repeated reconstruction [16:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=970s) Line charts expose a harder extraction problem. A vision encoder may retain the broad trend in revenue while losing the pixel-level detail needed to recover individual data points. The chart contains a potentially large table, but a rough visual reading does not recover that table. Abraham presents a case his team could not solve with a single model response. ![Source frame: A line chart becomes a table through repeated reconstruction](https://ai.engineer/api/talk-essay-media/5251d1c11f121fe56f78a64c3f51b6243d03c8ce7f2659d05c4a72ac7705364e) Source frame: A line chart becomes a table through repeated reconstruction The agent harness changes the task from a one-shot reading into an iterative reconstruction. The agent has a code interpreter and can visualize the chart it generates from extracted data. It examines that reconstruction, finds mistakes, and revises the extraction repeatedly. The reported result is a Markdown table recovered from the original line chart, with a plotted reconstruction shown in the presentation. The talk does not give a numerical error tolerance for the recovered points, so the example demonstrates the method rather than establishing exact numerical recovery. Where does the improvement come from? The loop below makes the generated chart inspectable before the table is accepted. Tools give the agent a way to expose errors in its proposed data, and repeated checks let it change that proposal. The useful capability is the whole loop—extraction, execution, visualization, and correction. Structured extraction can use a similar harness. A parent agent sets validation criteria for subagents, which matters when a form contains tens of thousands of fields and omissions can remain silent. Abraham describes a benchmark tradeoff: frontier models at maximum reasoning were precise about the rows they returned, yet dropped substantial content; dedicated document services recovered more content while making more errors in what they extracted. Precision asks whether returned content is correct. Recall asks how much required content was recovered. Reducto reports that its harness improved both for this task, though the talk supplies no scores to quantify the gain. Chart extraction with a verification loop The agent visualizes its proposed data, finds mistakes, and revises the extraction before producing the final table. - Original line chart — Visual lines encode individual data points. - Proposed table data — Extract and revise the chart’s data. - Generated chart — Use the code interpreter and visualize the reconstruction. - Find mistakes — Inspect and repeat correction. - Final Markdown table — The reported output of iterative chart extraction. - Original line chart → Proposed table data (Extract) - Proposed table data → Generated chart (Plot with tools) - Generated chart → Find mistakes (Inspect) - Find mistakes → Proposed table data (Revise and repeat) - Find mistakes → Final Markdown table (Finish after iteration) ### Evaluate each stage, then let agents choose the document tools [19:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1141s) Evaluations should guide these design choices at two levels. Off-the-shelf datasets provide a test bed, while production monitoring catches the differences between a constructed dataset and the documents users actually submit. Within the pipeline, parsing, retrieval, and final formatting each need attention. Perfect parsing cannot help an agent that receives the wrong context. The final question remains whether the changes improve the agent’s completed work. ![Source frame: Evaluate each stage, then let agents choose the document tools](https://ai.engineer/api/talk-essay-media/d64412e92e505500e67411dc8338b26189ada2bd08e7ff52c9afc73adf6fcdf8) Source frame: Evaluate each stage, then let agents choose the document tools The closing architecture moves some orchestration into the agent itself. Abraham describes customers giving agents a file system to navigate and a CLI through which they can choose document tools as needed. A document no longer has to follow one predetermined end-to-end flow. The agent decides whether it needs to read a particular document and which operation to apply. That interface separates readable content from metadata. The content field supplies what the agent needs to read; metadata retains information such as bounding boxes for citations. This gives the agent useful text while preserving a way to locate the supporting material in the original document. Editing receives only a brief mention, and document generation is presented as a direction for the coming months rather than an explained capability in this talk. Abraham’s final recommendation is to reconsider the workflow as capabilities change. Decompose parsing to balance accuracy, cost, and latency; use verification to catch errors; format data for its consumer; classify and split before downstream work; and evaluate every stage. The file-system example extends those choices: tools and document representations can remain well defined even when the agent chooses the sequence in which to use them. ## Key takeaways - Combine efficient layout detection with VLM interpretation where visual complexity requires it; large models need not process every part of every page. - OCR correction should preserve what the document says, including its mistakes. Targeted token edits reduce the opportunity for a model to rewrite the underlying facts. - Format data for each consumer: natural-language table renderings for retrieval, and structure-preserving representations for reasoning. - Tool use and repeated verification can make difficult extraction tasks tractable, but extraction quality must include both correctness and completeness. - Evaluate individual stages and completed agent work, using production monitoring as well as test datasets. Agents choosing tools still need good inputs, relevant context, and usable outputs. ## Slides ### Better agents through better data Adit Abraham introduces data as a practical challenge in building agents, focusing on unstructured sources such as images, PDFs and spreadsheets. [0:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=16s) [Title slide introducing a talk on ingesting unstructured data at scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/85d962210eacbccd80c266206e84095a12bb7cad5d08ae16d5a26f5405f1cdaf.webp) Better agents through better data. How leading AI teams ingest unstructured data at scale. Adit Abraham Co-founder & CEO, Reducto DAIS Summit · 2026 reducto ### Agentic document processing for AI teams Reducto serves both large enterprises and AI startups. The company says it has processed 4 billion+ documents and raised $108M. [0:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=51s) [Reducto overview showing customer categories, startup logos, funding, and documents processed.](https://assets.ai.engineer/sites/aiecode2025/sha256/324f51ec0399420a8c2d290762f4ced1b3d6d70f1fb08a9dc956af52d7f4451b.webp) reducto Agentic document processing for world class AI teams. CONFIDENTIAL Top 5 Global Tech Co CONFIDENTIAL Top 5 Global Hedge Fund CONFIDENTIAL Top 5 Global CRE CONFIDENTIAL Top 30 Insurance Broker CONFIDENTIAL Top 5 Legal Tech Co CONFIDENTIAL Top 5 Global PE Firm Harvey Legal AI, Series F rogo Finance AI, Series C MERCOR Data AI, Decacorn scale Data AI, Series G Vanta Compliance AI, Series D From global enterprises to leading startups $108M Raised in Funding Backed by a16z, Benchmark, First Round, YCombinator & more 4,000,000,000+ Documents Processed Transforming the world’s unstructured data. Proprietary & Confidential | 2 ### What we'll cover The agenda moves from the enterprise data bottleneck and PDF processing to agent-in-the-loop workflows, evaluations, and the shift from RAG to agents. [1:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=106s) [Agenda listing ten topics on AI agents and document data processing.](https://assets.ai.engineer/sites/aiecode2025/sha256/66735e71ff618423e3ea663eba9251b8220d02da444262a6544f5a90aad1b690.webp) • AGENDA What we'll cover. Ten parts — from the industry shift to the patterns production teams actually ship. 01 The sliding lens of AI From information synthesis to autonomous action. 02 The enterprise data bottleneck Where agents hit the wall. 03 Why PDFs are still hard The most common file — the hardest to read. 04 Traditional CV vs. VLMs Decomposing for task-specific performance. 05 Agent-in-the-loop Agentic OCR and speculative verification. 06 Formatting for the consumer Markdown, HTML, or prose — per region. 07 Beyond parse and extract Classification and splitting as the unlocks. 08 Agent harnesses for data Generate → evaluate → refine loops. 09 Evaluations as the foundation Instrumentation beyond end-to-end metrics. 10 From RAG to agents File-based, bidirectional architectures. Reducto 04 / 29 ### The sliding lens of AI applications Part 1 introduces the shift from information synthesis to autonomous action and how it changes what matters. [2:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=174s) [Part one introduces the shift from information synthesis to autonomous action.](https://assets.ai.engineer/sites/aiecode2025/sha256/5fa9b8db59c3ec53b427ce2399586f302bb94b072135385d30d0fc8191b96f3c.webp) 01 • PART 01 The sliding lens of AI applications. The industry is shifting from information synthesis to autonomous action — and that changes what matters. Part 01 · The sliding lens 05 / 29 ### From information synthesis to real human work Earlier tools delivered answers for humans to act on. Agents aim to complete work end-to-end, including reading, deciding and writing, creating new challenges around unstructured data. [3:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=189s) [Comparison of earlier information synthesis tools with agents performing autonomous work.](https://assets.ai.engineer/sites/aiecode2025/sha256/629d775bdc4aabaff3649dc4234e1b10e609f5795779175ef5d5d30947fee9a6.webp) AI has shifted from information synthesis to real human work. • THE SHIFT Initially most products in the industry were information synthesis tools — search tools, chatbots, summarization engines. Today teams are eager to do real human work with agents, and that opens a new set of challenges around unstructured data. THEN — 2022–2024 Information synthesis Search interfaces, chatbots, summarization engines. Answers delivered to humans who then acted on them. Search · Chat · Summarize · Q&A → NOW — 2025+ Autonomous action Agents that execute real human work end-to-end — reading, deciding, writing, delivering a completed task. Parse · Decide · Write back · Execute Reducto 06 / 29 ### Bad data can create a wrong outcome An insurance answer calculates 2018 fire losses from the wrong row without signaling a problem. The speaker warns that bad inputs can compound as agents make multiple decisions across data sources. [4:05](https://www.youtube.com/watch?v=0I07YAuF8xA&t=245s) [An example answer from an insurance document is paired with notes explaining that it used the wrong row.](https://assets.ai.engineer/sites/aiecode2025/sha256/7fe580d4627e37d4ca3e13cdf237bfece4d5d5526066a68ccc4038a76f9fd293.webp) THE STAKES An agent acting on bad data doesn't just return a wrong answer — it creates a wrong outcome. Based on the calculation of loss development factors for fire, what were the incurred losses at 36 months in 2018? Based on the documents provided, the calculation of loss development factors for fire using the link ratio method shows: For accident year 2018, the incurred losses at 12 months were $15,271,855. The loss development factor from 12 to 36 months is 1.020. Therefore, the incurred losses at 36 months for accident year 2018 can be calculated as: Incurred Losses at 36 months = Incurred Losses at 12 months x Loss Development Factor from 12 to 36 months Incurred Losses at 36 months = $15,271,855 x 1.020 = $15,577,492 So the incurred losses at 36 months in 2018 for fire were $15,577,492 based on the given calculation of loss development factors. Answer pulled from the wrong row Confident, citation-backed — and wrong No visible signal that the input was bad Bad inputs lead to bad outputs. Reducto 07 / 29 Interface text: insurance.pdf 355.86 KB · 1593... Copy Retry ### The enterprise data bottleneck Part 2 turns to the difficulty of accessing the enterprise context agents need. [4:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=276s) [Part 2 introduces the difficulty of accessing enterprise context.](https://assets.ai.engineer/sites/aiecode2025/sha256/c90bad01cb303517911fc3b57f5ce5489c9d012d856bdfb2b545aec57a5fe6ca.webp) The enterprise data bottleneck. 02 • PART 02 Where all the context agents need actually lives — and why it's so hard to get at. Part 02 · The enterprise data bottleneck 08 / 29 ### The reality of enterprise data Enterprise context is scattered across files and systems, often in unstructured formats. Agents must read those files accurately, retrieve the right context, and modify documents without losing structure. [4:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=288s) [Enterprise data is unstructured, scattered and multimodal, creating challenges for extraction, retrieval and editing.](https://assets.ai.engineer/sites/aiecode2025/sha256/d27a7858afed431462c927f010821c7fdbc7241a19bda758f4eaffcfbe372402.webp) THE REALITY Most enterprise data lives unstructured, scattered, and multimodal. The vast majority of enterprise context exists outside databases — in PDFs, scans, and spreadsheets. It's spread across files that aren't always clearly sorted in folders. It lives in both text and multimodal formats like images and figures. 01 · FILE FORMATS Unstructured by default PDFs Scans Handwritten forms Spreadsheets Images Slides Contracts No row-and-column assumption. No schema. Every file is a special case. 02 · WHERE IT LIVES Scattered across context Folder hierarchies Email attachments Shared drives Multi-file threads Version history Context for a single decision often spans multiple files across multiple systems. 03 · THE THREE PROBLEMS AGENTS FACE Parsing & extraction accuracy Read the file faithfully — characters, tables, figures, reading order. Retrieval & categorization Find the right information across a large corpus, reliably. Interaction & editing Modify documents — fill forms, update reports — without losing structure. Reducto 09 / 29 ### Why PDFs remain challenging This section examines why PDFs, a common enterprise file format, remain difficult to read reliably. [5:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=327s) [Part 3 introduces the challenge of reading PDFs reliably.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c9c1b13a2d1bd9bb7f6cc1ad99f95dd62f6e84cbf917147da21cbc57f0b4271.webp) 03 • PART 03 Why PDFs continue to present a challenge. The most common file format in the enterprise — and the hardest to read reliably. Part 03 · Why PDFs are hard 10 / 29 ### PDF encodings and document complexity PDFs were designed to preserve printable appearance. Preparing them for agents means interpreting charts, annotations and dense tables across very different internal representations. [6:02](https://www.youtube.com/watch?v=0I07YAuF8xA&t=362s) [Examples of a sample financial statement, a form and a dense table illustrate varied PDF content and layouts.](https://assets.ai.engineer/sites/aiecode2025/sha256/aae0dabfdebd05a91c548709d6e0520d3b4f05d067ca3a0fb7f42aa2dd3238ee.webp) THE PDF PROBLEM PDFs are fundamentally an image format with a broad range of encodings. Data can include handwritten text, large tables, poor scans, image content, challenging reading order. Same extension, wildly different internal representations. Financial statement example: Ameriprise Financial Your Ameriprise statement Your personal advisor [redacted] Value of your investment accounts Value of your investment accounts over time Your asset allocation [illegible] Form example: PO [illegible] Bill to [illegible] Purchase Order [illegible] Hallucinated Failed to extract Table example: [illegible] Mixed text + charts + tables Forms · handwritten annotations Scans · merged cells · dense tables Reducto 11 / 29 ### The long tail of VLM failures VLMs handle handwriting well, but difficult layouts and scans remain. The slide highlights silent failures: the model gives a confident, wrong answer without warning. [7:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=441s) [Six document-processing challenges include handwriting, merged cells, poor scans, reading order, images and silent failures.](https://assets.ai.engineer/sites/aiecode2025/sha256/dfcfcddeea60f0d0272fc666450e0c15d9eca0e2ce2651cc9a4368f378ad9f07.webp) • THE LONG TAIL VLMs unlock a lot — but the long tail is where production breaks. Most failure modes are difficult to understand in production. The model doesn't tell you when it's wrong — and silent failure is the one that compounds. 01 Handwritten text Margins, annotations, signatures on otherwise-typed forms. 02 Merged table cells Complex financial tables with rowspans and nested headers. 03 Poor or skewed scans Rotated, low-resolution, compressed photocopies of photocopies. 04 Challenging reading order Multi-column layouts, sidebars, footnotes, floating boxes. 05 Embedded images & charts Figures that require interpretation, not just extraction. 06 · THE KILLER Silent failures The model returns a confident answer. It's wrong. You don't know. Reducto 12 / 29 ### Traditional computer vision vs. VLMs VLMs handle handwriting well, but document processing at scale also raises questions of determinism and efficiency. This section introduces task-specific models. [8:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=492s) [Part 4 introduces task-specific computer vision and vision-language models.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1503435efee0e5dafe02172e8e49e04f30e81738be20291896efe0c5972132f.webp) 04 PART 04 Traditional computer vision vs. VLMs. Decomposing the problem allows for better task-specific model performance. Part 04 · CV vs VLMs 13 / 29 ### Traditional CV and post-trained VLMs Reducto combines traditional CV for document structure with post-trained VLMs for semantics, including handwriting and chart interpretation. It says this decomposition is more accurate than either approach alone. [8:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=509s) [Two cards compare traditional computer vision for structure with post-trained VLMs for semantics.](https://assets.ai.engineer/sites/aiecode2025/sha256/22d421763e3b957e95f73f2dc51c5c111092983401b6eb017d864899e36f3de8.webp) • ARCHITECTURE Traditional CV and post-trained VLMs present tradeoffs At Reducto we use a series of traditional CV models along with VLMs post-trained for specific tasks. Decomposition gives higher accuracy than either approach alone. TRADITIONAL CV Structure Object detection, layout understanding, table segmentation, reading order — where classical vision still leads. Fast · Deterministic · Task-specific + POST-TRAINED VLMS Semantics Content extraction, handwriting, chart interpretation — anywhere visual reasoning is required. Flexible · Multimodal · Generalizing = • Reducto 14 / 29 ### From human-in-the-loop to agent-in-the-loop Reducto describes agentic OCR as a shift from human review to agents that find last-mile issues automatically using speculative decoding. [9:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=548s) [Section slide introduces agentic OCR and automatic detection of last-mile issues.](https://assets.ai.engineer/sites/aiecode2025/sha256/accba952a5aafa9ce66678e37ee6da0dfbbef2e9c976e071a02adabee6284e59.webp) From human-in-the-loop to agent-in-the-loop. 05 • PART 05 We pioneered agentic OCR in late 2024. Speculative decoding finds last-mile issues automatically. Part 05 · Agentic OCR 15 / 29 ### Agentic OCR framework The example corrects a misread prescription dosage from lO to 10 mL. Abraham describes targeted token-level edits that preserve what the document says. [9:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=569s) [An example shows a misread dosage corrected in an OCR output.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f8f23ddecbd5ffb9ec5d6456c3867f2c88fd55732f58e6bc57936237db4531d.webp) Agentic OCR Framework output Prescription: Take lO mL dosage daily corrected output Prescription: Take 10 mL dosage daily ### Agentic OCR workflow Both workflows parse the document with CV and a VLM. The agentic workflow then flags inconsistencies for verification and correction, aiming to produce high-confidence output at scale. [10:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=627s) [Two four-stage pipelines compare human review with agent verification and correction.](https://assets.ai.engineer/sites/aiecode2025/sha256/33a54b79a235a9bd0a2aa651c5d4d74c058ec7802c6ba0c91f5a76c8fd11ae24.webp) • AGENTIC OCR Agentic OCR replaces human-in-the-loop with agent-in-the-loop. Combining the two gives the best combination of accuracy and reliability — without the latency and cost of manual review. Old — Human in the loop 01 Document input Raw file enters pipeline 02 CV + VLM parse Initial extraction 03 Human review Reviewers fix errors manually 04 Output Delivered after delay New — Agent in the loop 01 Document input Raw file enters pipeline 02 CV + VLM parse Candidate extractions 03 Verification and correction Agent flags inconsistencies 04 High-confidence output Delivered at scale Reducto 16 / 29 ### Data accuracy and formatting Accurate characters come first; then the output can be shaped for its consumer. Tables illustrate why document-to-Markdown pipelines need attention beyond extraction. [10:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=643s) [Section slide says accurate characters come first, followed by formatting for the consumer.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6d21f299d892b72f3d0770b19504e180ae3ee36be282ed8d969057d9f38bb03.webp) 06 • PART 06 Data accuracy unlocks gains from data formatting. Get the characters right first. Then shape the output for the consumer. Part 06 · Formatting 17 / 29 ### Formatting strategy Use Markdown for simple tables and HTML when merged cells and nested headers carry meaning. For retrieval, turn structured data into prose so embeddings can match meaning rather than syntax. [11:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=691s) [Three columns compare Markdown for simple tables, HTML for complex tables, and natural language for retrieval.](https://assets.ai.engineer/sites/aiecode2025/sha256/a35990b3f769115530524036dcb4a65dc8433cdb330c87a119089de8d402c8a8.webp) FORMATTING STRATEGY With high-fidelity data, formatting becomes the next lever. Tables with merged cells are better represented in HTML than Markdown — but HTML isn't token-efficient for simple tables, so we dynamically change format based on complexity. Embedding models have different limits from LLMs; for retrieval, reformatting as prose often wins. SIMPLE TABLES Markdown Token-efficient. LLMs parse it cleanly. Good default for 2–3 column data. | Q | Rev | |----|------| | Q1 | 4.2M | | Q2 | 5.1M | COMPLEX TABLES HTML Preserves merged cells, rowspans, nested headers. Worth the tokens when structure matters. FOR RETRIEVAL Natural language Reformat structured data as prose so embedding models match on meaning, not syntax. "Revenue grew from $4.2M in Q1 to $5.1M in Q2, driven by..." Same underlying data, different representations — routed dynamically per region and per consumer. Reducto 18 / 29 ### Effective ingestion pipelines Abraham argues that classification and splitting deserve more attention alongside parsing and extraction because they can unlock agent performance. [13:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=790s) [Section slide stating that ingestion pipelines involve more than parsing and extraction, including classification and splitting.](https://assets.ai.engineer/sites/aiecode2025/sha256/f50df6beaf58759128bdef5d4ddbba9f660514bd4f665c0fdd41b9765487233b.webp) Effective ingestion pipelines are more than parse and extract. 07 • PART 07 Classification and splitting are the underinvested stages that unlock real agent performance. Part 07 · Ingestion pipelines 19 / 29 ### Effective ingestion pipelines Classification and splitting need more investment. Abraham describes the human work of understanding a document corpus and routing material to the appropriate pipeline. [14:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=872s) [Section slide stating that effective ingestion pipelines involve more than parsing and extraction, with emphasis on classification and splitting.](https://assets.ai.engineer/sites/aiecode2025/sha256/1f43cfb2a9f75bebf09b630a297bc6adfdc3704378e1ec63c413125de721cb48.webp) 07 • PART 07 Effective ingestion pipelines are more than parse and extract. Classification and splitting are the underinvested stages that unlock real agent performance. Part 07 · Ingestion pipelines 19 / 29 ### Parse, split, extract and edit Reducto illustrates four document operations: parsing content, splitting document piles, extracting fields to a schema, and filling forms. Abraham describes routing and decomposing documents as work beyond the parsing layer. [14:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=881s) [Four columns illustrate parsing, splitting, extracting and editing documents.](https://assets.ai.engineer/sites/aiecode2025/sha256/c217eed7eced23814fff8822d77456fdb03931e8e51baf62bd5b26e87e14c078.webp) Parse Intelligently understand and prepare data how you want it [Document text illegible] Split Break down piles of documents intuitively and efficiently [Document text illegible] Extract Pull only the data you need according to a schema username email address orders [Other document text illegible] Edit Detect fields and fill out forms automatically using AI [Form text illegible] reducto ### Classification and splitting Classification routes documents to the right parsing strategy. After parsing and extraction, splitting gives the LLM focused sections to work on and lets teams verify results section by section. [14:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=892s) [Five pipeline stages from raw input to delivery, with reasons to classify and split documents.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c056466ab35ba6c595d5ed8171e50d857b7c7232b09bd734c59ba218f336294.webp) PIPELINE STAGES Classification and splitting equip LLMs to do their best work. Effectively classifying and splitting information gives the LLM a focused problem and lets you verify results at each step. Stage 01 Raw input Uploaded files Stage 02 Classify Route by type Stage 03 Parse & extract CV + VLM hybrid Stage 04 Split Logical sections Stage 05 Format & deliver Consumer-shaped output WHY CLASSIFY? — Route to the right parsing strategy per document type — Pick the right output format downstream — Adapt agent behavior per document class WHY SPLIT? — Give the LLM a focused, scoped problem — Verify results section-by-section, not just end-to-end — Reduce hallucination from bloated context windows Reducto 20 / 29 ### Agent harnesses for data After classification and splitting, agent harnesses use structured loops to repeatedly improve model output. [16:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=968s) [Section slide introducing agent harnesses as structured loops that repeatedly improve model output.](https://assets.ai.engineer/sites/aiecode2025/sha256/2282d920d21d22a8c0526cb7d23ec0a5fd98bc1f7e64dbcbeda8a5c2037517be.webp) 08 • PART 08 Agent harnesses as a new frontier for data. Structured loops that let models repeatedly improve their own output. Part 08 · Agent harnesses 21 / 29 ## Full transcript [0:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1s) [music] [0:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=13s) Hi. Can everyone hear me? Sweet. We have [0:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=16s) a short 20 minutes here and so I wanted [0:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=19s) to jump in and get right to it. Uh my [0:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=21s) name is De. I'm the co-founder and CEO [0:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=22s) of Redducto. Uh and today we wanted to [0:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=25s) talk about one of the I think really [0:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=28s) practical but maybe less sexy parts of [0:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=31s) building agents that actually work in [0:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=33s) the real world uh which is data. Um I'm [0:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=35s) sure you've seen plenty of talks about [0:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=37s) data today. We primarily have focused on [0:39](https://www.youtube.com/watch?v=0I07YAuF8xA&t=39s) building infrastructure for anybody [0:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=41s) working with some of the hardest sources [0:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=43s) of data which is unstructured images, [0:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=45s) PDFs, spreadsheets, everything that [0:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=47s) humans are used to using day-to-day. Uh, [0:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=50s) have any of you used Reduct already or [0:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=53s) trial it? Cool. Um, I guess helpful [0:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=57s) context maybe to start is we're an [0:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=59s) agentic document processing platform. [1:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=60s) Um, so we help a lot of the world's [1:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=63s) leading AI teams build uh both AI [1:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=66s) applications and also workflows [1:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=67s) depending on what they're trying to do. [1:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=69s) Uh, that includes a lot of the AI [1:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=70s) natives that you've probably seen today. [1:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=72s) It includes the Harveys, the Lores, the [1:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=73s) Rogos of the world. Uh, but also some of [1:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=76s) the largest enterprises in the world. [1:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=77s) And I think that's important context for [1:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=79s) what we're going to talk about. Um, we [1:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=81s) work with the largest tech companies, [1:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=83s) global financial institutions, insurance [1:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=85s) orgs, people that have decades worth of [1:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=88s) historical data that historically has [1:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=90s) been really, really hard to actually use [1:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=92s) outside of a demo context. And across [1:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=95s) all these companies at this point, we've [1:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=96s) processed many billions of documents for [1:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=98s) our customers. I've gotten kind of lazy [1:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=100s) with that plus symbol at the end, uh, [1:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=102s) but the number keeps changing and so [1:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=104s) we'll let it sit there. Uh but the main [1:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=107s) thing that we've learned across those [1:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=108s) and what I wanted to focus on today is [1:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=110s) not actually a reduct product itself. [1:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=112s) It's the intricacies of what we've [1:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=114s) learned that hopefully you can actually [1:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=116s) take home and implement in the work that [1:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=118s) you're doing as well. Um and I think a [2:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=120s) lot of the work that we've done and a [2:02](https://www.youtube.com/watch?v=0I07YAuF8xA&t=122s) lot of the learnings we've had are in [2:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=124s) this sort of broader thematic context of [2:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=126s) the scope of AI applications has changed [2:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=128s) a lot recently. Uh from just like [2:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=130s) information synthesis products to actual [2:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=132s) agents doing work. And with that we [2:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=135s) found that there are a few things that [2:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=136s) are worth talking through. One is [2:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=137s) framing the problem itself. Uh the [2:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=139s) bottleneck that people face. Uh why PDFs [2:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=142s) in particular are hard even though [2:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=143s) you've probably seen two dozen different [2:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=146s) PDF processing launches on Twitter. Uh [2:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=149s) where we find strengths and weaknesses [2:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=150s) with different tools. Uh we think [2:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=152s) there's a right place in time for [2:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=153s) traditional CV versus BLMs. Um and then [2:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=156s) more interestingly I think the latter [2:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=157s) half of this talk will actually be [2:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=158s) focused on the next frontier. uh what [2:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=161s) we've seen be possible as a result of [2:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=163s) having agents in the loop uh things that [2:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=165s) we found as a result of being able to [2:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=166s) use harnesses for different types of [2:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=168s) tasks uh and our learnings around things [2:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=170s) like evaluations uh as you go from rag [2:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=172s) to building agent products but I'll [2:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=175s) start with the first thing which is [2:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=177s) something that I assume has been harped [2:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=179s) on a lot if you went to AI engineer a [3:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=181s) few years ago the word that you would [3:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=183s) have heard in every single talk would [3:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=184s) have been rag um and everybody was [3:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=186s) building some form of rag application [3:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=188s) for a while all applications ations were [3:11](https://www.youtube.com/watch?v=0I07YAuF8xA&t=191s) really some form of information [3:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=192s) synthesis, right? You would pull in [3:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=195s) information from some context, whether [3:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=197s) that's a perfect prompt or a file that a [3:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=199s) user uploaded and you would have [3:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=201s) something like a search product. Um, [3:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=202s) you'd have enterprise search, you'd have [3:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=204s) a chatbot that would do simple question [3:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=206s) answer on top of the content and that [3:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=208s) was it. But today, the buzzword that [3:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=211s) you've probably heard a million times is [3:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=214s) agents. Uh, and for those of you that [3:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=216s) are engineers, you've probably had cloud [3:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=218s) code or similar tool do a lot of your [3:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=220s) endto-end work for many, many tasks. And [3:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=223s) that same sort of shift is starting to [3:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=224s) happen for all sorts of white collar [3:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=226s) work. Um, whether you're in finance or [3:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=228s) insurance or healthcare, people are [3:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=230s) starting to make autonomous decisions. [3:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=232s) They're trying to create end-to-end work [3:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=234s) products, not just answer questions from [3:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=236s) a PDF, but generate and modify PDFs as [3:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=238s) well. And that's a very different [4:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=240s) framing. the tools that you need, the [4:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=241s) problems that you face change a lot [4:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=243s) versus just trying to build a retrieval [4:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=246s) uh platform. [4:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=248s) And the common core for all of this is [4:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=252s) it actually ends up being an even more [4:14](https://www.youtube.com/watch?v=0I07YAuF8xA&t=254s) importance problem to solve for when you [4:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=257s) start having these multi-step pipelines. [4:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=259s) If you're just doing question answer, [4:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=260s) there is obviously a risk that comes [4:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=263s) with just answering the question. But [4:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=265s) when agents are making multiple [4:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=267s) decisions that compound over the course [4:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=269s) of a source of files, when they're [4:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=270s) pulling in multiple sources of data, the [4:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=272s) risk of bad inputs becomes really, [4:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=274s) really pervasive in your pipeline. And [4:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=277s) that's what we focus on because at the [4:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=280s) end of the day, a lot of the value of [4:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=282s) language model tools in the real world [4:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=284s) only applies in the context to which you [4:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=287s) apply that intelligence. And for a lot [4:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=289s) of enterprises, data is unstructured, it [4:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=292s) is scattered, and it is multimodal. uh [4:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=294s) it's not this like cleanly organized [4:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=296s) repository. You have random teams and [4:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=298s) different organizations that go through [5:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=300s) and put things in Google Drive and Box [5:02](https://www.youtube.com/watch?v=0I07YAuF8xA&t=302s) and wherever else you might have things. [5:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=304s) Um you're going to have data for formats [5:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=306s) that are unstructured by default. You [5:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=308s) don't necessarily know what is in your [5:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=310s) corpus of information. And you're going [5:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=312s) to have all sorts of downstream problems [5:14](https://www.youtube.com/watch?v=0I07YAuF8xA&t=314s) that come with that. Um some of the [5:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=316s) problems are going to be parsing [5:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=317s) extraction accuracy and that definitely [5:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=318s) matters. But it's also going to be about [5:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=320s) problems like are you retrieving the [5:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=322s) right context? It's going to be about [5:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=323s) how do you actually interact with that [5:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=325s) context and apply modifications to it. [5:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=328s) And the thing that you've probably heard [5:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=331s) harp on and others in the space is PDFs [5:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=333s) are surprisingly still a very hard [5:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=336s) problem. Um I don't know if any of you [5:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=338s) follow Serge which is a data lab that [5:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=340s) works with a lot of the foundation model [5:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=342s) companies. uh they have this really [5:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=343s) great benchmark called GDP PDF where [5:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=346s) even Fable uh like current frontier of [5:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=349s) model intelligence is at about 30% on [5:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=351s) their benchmark um and it's entirely [5:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=353s) predicated on this idea of can we have [5:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=355s) models go through and actually make [5:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=357s) determinations off of contents that [5:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=359s) would be in documents like PDFs uh and [6:02](https://www.youtube.com/watch?v=0I07YAuF8xA&t=362s) the reason why they're hard I'll come [6:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=363s) back to that benchmark in a second is [6:05](https://www.youtube.com/watch?v=0I07YAuF8xA&t=365s) fundamentally PDFs as a file format are [6:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=368s) both very old but were designed in a [6:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=370s) very different context I've genuinely [6:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=373s) met people that have worked on PDF [6:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=375s) processing longer than I've been alive. [6:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=376s) Uh I've met the people that worked on [6:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=378s) printer drivers for printers to print [6:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=380s) PDFs in the early 1990s. And that's what [6:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=382s) it was like. You wanted to be able to [6:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=384s) represent what was faithfully on the [6:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=386s) documents when you originally created it [6:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=388s) and have it be printable at the end of [6:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=390s) it. A lot of the considerations today [6:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=392s) are not around that. Um ultimately what [6:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=394s) we want is something like a markdown [6:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=397s) representation, something that agents [6:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=398s) will effectively reason on. And in the [6:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=401s) real world, humans encode so much [6:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=403s) context visually. Like the average [6:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=406s) financial analyst that is a new grad in [6:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=409s) their IB role uh is not going through [6:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=412s) and thinking in terms of will agents [6:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=414s) consume this deck. They're creating [6:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=415s) these really creative slides. I'm sure [6:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=418s) you've seen the softbank slides with uh [7:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=420s) goose laying eggs. Those details matter, [7:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=423s) right? Like a lot of the data that [7:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=424s) you're going to reason on is going to be [7:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=426s) tabular structures that maybe don't have [7:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=428s) clean grid lines and separate out the [7:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=430s) merge cells. You're going to have things [7:11](https://www.youtube.com/watch?v=0I07YAuF8xA&t=431s) like line charts and graphs. You're [7:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=433s) going to have messy handwriting that [7:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=435s) even I as a human would often struggle [7:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=437s) to read. And that's the sort of problem [7:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=439s) that you need to solve for if you're [7:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=440s) dealing with that long tail. [7:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=443s) And we started the company in 2023 [7:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=446s) because we felt that there was a sort of [7:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=448s) step change in what was possible here. [7:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=450s) uh for a while as people would think [7:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=452s) about any sort of PDF processing [7:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=454s) problem, it used to be some modified [7:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=457s) version of an NLP pipeline. You do a [7:39](https://www.youtube.com/watch?v=0I07YAuF8xA&t=459s) simple OCR pass and then you try to [7:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=461s) post-process that text. And that worked [7:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=463s) when you would have really consistent [7:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=465s) layouts, right? Like if you knew that a [7:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=467s) W2 is always going to look like a W2, [7:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=469s) that's a constrained problem space that [7:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=471s) you can template your way around. But [7:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=473s) VLMs are interesting because they are [7:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=475s) fundamentally horizontal in nature. um [7:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=477s) for the first time you can have this [7:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=479s) premise of read the document the way [8:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=481s) that a human would have. You can have [8:02](https://www.youtube.com/watch?v=0I07YAuF8xA&t=482s) this premise of we want to address the [8:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=484s) longtail. Uh and so we found that they [8:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=487s) were incredible for all sorts of things [8:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=489s) like handwritten text in a way that [8:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=490s) traditional OCR just never was. But the [8:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=493s) flip side is that we don't think that [8:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=495s) they're a one-sizefits-all solution. And [8:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=497s) if you're solving this problem at scale, [8:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=499s) if you're a company dealing with [8:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=500s) hundreds of millions of documents, there [8:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=502s) are all sorts of secondary [8:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=503s) considerations like determinism. you [8:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=506s) care a lot about the efficiency of your [8:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=507s) processing. Uh, and so we find that [8:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=510s) there are some things where traditional [8:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=512s) CV is actually still really really [8:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=513s) strong. And this has been really [8:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=515s) underappreciated as autonomous vehicle [8:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=517s) research has gotten better. Techniques [8:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=518s) like object detection are more [8:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=520s) sophisticated than they were a decade [8:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=522s) ago. And so we find that subund million [8:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=524s) parameter models are really effective [8:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=526s) for things like detecting the layout of [8:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=528s) a document. Um, you can go really really [8:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=530s) far without even needing a large [8:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=532s) foundation model. And these are models [8:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=534s) that can actually run on CPU. You can [8:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=535s) run them at scale. You can make sure [8:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=537s) that you understand on a region level [8:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=539s) what are the hard things for you to [9:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=540s) process. And VLMs introduce this notion [9:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=543s) of semantics. You can go through and [9:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=544s) actually identify and make correct the [9:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=546s) sorts of mistakes that you were likely [9:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=547s) to have in your pipeline. [9:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=550s) And off of that idea of semantics, uh, [9:14](https://www.youtube.com/watch?v=0I07YAuF8xA&t=554s) when you're deconstructing this problem [9:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=555s) and you have a clear sense of where the [9:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=557s) more nuanced things are, if you've [9:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=559s) segmented the text on your page and you [9:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=560s) understand where the handwriting is, you [9:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=562s) can introduce this notion of a sort of [9:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=564s) agent in the loop. Whereas historically, [9:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=566s) you would have had a human review team [9:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=568s) go through and annotate and correct [9:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=569s) mistakes. Uh, VLMs can now present this [9:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=572s) sort of idea of what we call agentic [9:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=573s) OCR. Really for us what that looks like [9:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=576s) is if you've ever used you know a tool [9:39](https://www.youtube.com/watch?v=0I07YAuF8xA&t=579s) like cursor that's applying fast edits [9:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=580s) in your IDE there's this notion of [9:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=582s) speculative decoding where you're [9:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=584s) applying token level edits to your [9:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=585s) output. You can apply a similar sort of [9:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=587s) principle here uh that is not just you [9:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=589s) know sending OCR to Gemini writing a [9:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=592s) really pretty prompt asking it nicely to [9:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=595s) not deviate too much from the original [9:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=596s) because we find that when you're doing [9:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=598s) that sort of next token prediction you [9:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=599s) introduce net new loss cases where [10:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=601s) models that are really intelligent will [10:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=603s) start actually correcting things not [10:05](https://www.youtube.com/watch?v=0I07YAuF8xA&t=605s) faithfully to what was in the document. [10:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=606s) They'll see the word total and if the [10:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=608s) human made a mistake in that table [10:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=609s) models will actually sometimes go [10:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=610s) through and add up the values in the [10:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=613s) table themselves. What you want is to [10:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=615s) correct the token level edits that you [10:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=616s) want. Maybe you messed up a period [10:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=618s) versus a comma, a zero versus an O. [10:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=620s) Those sorts of details really, really [10:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=622s) matter. And that's a question of how do [10:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=624s) we actually represent what a human would [10:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=625s) have seen if they had read that [10:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=626s) document. [10:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=629s) So the way that we see this is agentic [10:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=631s) OCR is almost like that human loop [10:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=634s) analogy where you have the first inputs [10:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=637s) uh go through with a CD plus VLM parse [10:39](https://www.youtube.com/watch?v=0I07YAuF8xA&t=639s) but then you have a verification [10:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=641s) correction layer that ends up leading to [10:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=642s) a high confidence output. [10:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=645s) But I mentioned earlier that we don't [10:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=647s) see the range of problems as purely just [10:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=649s) parsing and extraction. And a lot of [10:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=652s) what this looks like is I think it's [10:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=654s) important to think through the details [10:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=656s) of your pipeline. Even if you have a [10:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=658s) great documents to markdown pipeline and [11:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=661s) a great example of this is if you've [11:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=663s) built any sort of rag platform um you've [11:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=666s) probably had some consideration around [11:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=667s) things like tables. [11:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=669s) There are a lot of things that you can [11:11](https://www.youtube.com/watch?v=0I07YAuF8xA&t=671s) encode well in something like markdown. [11:14](https://www.youtube.com/watch?v=0I07YAuF8xA&t=674s) uh but things like this table where the [11:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=676s) merge cells actually encode a lot of [11:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=678s) meaning. It matters that you're [11:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=680s) preserving that sort of structure and [11:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=681s) it's not a model limitation. LMS are [11:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=683s) incredible at reasoning through like an [11:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=685s) HTML structure of the same table, but [11:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=687s) you're also wasting a lot of tokens and [11:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=688s) that gets expensive quickly. And [11:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=690s) obviously on the other end of the [11:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=691s) spectrum, you probably don't want to go [11:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=694s) through and encode simple tables in HTML [11:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=696s) because then you have a lot of HTML tags [11:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=698s) that are erroneous. And so what we ended [11:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=700s) up doing was looking at this as sort of [11:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=702s) like a dynamic problem of when you have [11:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=703s) a simple table, great, we can [11:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=705s) approximate that data in markdown. When [11:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=708s) you have a more complex table, you may [11:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=710s) want to use something like HTML. But [11:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=712s) it's not only language models that [11:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=714s) should be a consideration in your [11:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=715s) pipeline. If you're doing anything [11:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=717s) related to embedding, you're also going [11:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=719s) to have the secondary problem of [12:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=720s) retrieval of that context. Right? That [12:02](https://www.youtube.com/watch?v=0I07YAuF8xA&t=722s) same table that I showed you earlier, if [12:05](https://www.youtube.com/watch?v=0I07YAuF8xA&t=725s) you look at the HTML representation is [12:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=728s) really really messy. The vast majority [12:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=730s) of that snippets is just HTML tags. It's [12:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=732s) just classifying the structure of the [12:14](https://www.youtube.com/watch?v=0I07YAuF8xA&t=734s) document. And the unfortunate thing is [12:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=736s) whereas in some blanket evals, you might [12:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=738s) have contents that is really trying to [12:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=742s) do the work for the model and say [12:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=744s) exactly what you're looking for. A real [12:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=746s) world person does not enumerate the [12:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=748s) values in the table. They just say how [12:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=750s) did revenue change over time and they [12:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=752s) assume that you're going to retrieve the [12:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=753s) right table when it's relevant. And [12:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=755s) whereas language models can reason [12:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=757s) through that text effectively if you're [12:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=758s) pulling from a large corpus, we find [12:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=760s) that embedding models really struggle to [12:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=762s) correlate that natural language human [12:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=764s) prompt with this messy blob of HTML tags [12:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=767s) and numbers. And so a big thing that you [12:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=769s) can do that actually takes very little [12:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=771s) effort is creating a representation [12:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=773s) that's more so designed for the [12:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=774s) embedding model itself. taking the same [12:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=776s) table, we're creating a natural language [12:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=778s) representation of that table. So you [13:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=780s) have the best of both worlds. When [13:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=781s) you're actually passing this into the [13:02](https://www.youtube.com/watch?v=0I07YAuF8xA&t=782s) model for reasoning, you're using the [13:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=784s) HTML table representation. And when [13:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=786s) you're trying to make sure that you [13:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=787s) retrieve the right snippets, you're [13:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=788s) using that natural language block. [13:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=792s) Off of that, uh there's also this idea [13:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=795s) of there's a lot to do that is not just [13:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=797s) parsing and extraction. Um, and I think [13:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=799s) a lot of the industry's focus has been [13:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=801s) on parsing and extraction historically [13:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=803s) because we do think that there's a [13:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=804s) massive uplift there. And I talked [13:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=807s) earlier about the GDF GDP PDF benchmark [13:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=811s) which I think is a great illustrative [13:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=813s) example of what you can see uh as a [13:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=815s) result of improving your data pipeline. [13:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=818s) So what we found is that if you take the [13:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=820s) same exact benchmark that I mentioned [13:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=822s) earlier, uh, unfortunately we couldn't [13:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=823s) test it on Fable because our access was [13:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=826s) cut. uh but if you test on other models [13:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=828s) and you give it both the original PDF [13:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=830s) but also a structured representation of [13:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=832s) the PDF like the parse results here [13:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=835s) across models whether it's Gemini [13:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=837s) whether it's anthropic or openi um you [14:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=840s) find that you actually improve end LLM [14:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=843s) performance just from better inputs and [14:05](https://www.youtube.com/watch?v=0I07YAuF8xA&t=845s) it's to an extreme where models like GPT [14:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=848s) 5.5 and opus actually outperform [14:11](https://www.youtube.com/watch?v=0I07YAuF8xA&t=851s) something like Fable out of the box not [14:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=853s) just on an accuracy basis but as a [14:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=856s) result of giving better inputs, the [14:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=858s) models end up needing to use fewer [14:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=860s) reasoning tokens as well. Um, they're [14:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=861s) focused less on representing the data [14:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=864s) and more on the actual outputs and as a [14:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=866s) result they end up driving down latency [14:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=868s) and also getting to the correct answer [14:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=870s) more quickly. But even once you have [14:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=873s) that sort of pipeline and you've gone [14:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=875s) through and you've actually inspected [14:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=878s) everything in your parsing layer, a lot [14:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=881s) of human work is going to require [14:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=882s) actually understanding the range of what [14:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=884s) you have in your corpus, routing it to [14:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=886s) the appropriate pipeline and sort of [14:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=888s) decomposing that problem or even at the [14:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=890s) end editing and modifying your document. [14:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=892s) And so what we tried to do is look at [14:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=894s) this as this problem of how do you make [14:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=896s) sure that every interaction that a [14:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=897s) language model has with a document is as [14:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=899s) effective as if a human would have done [15:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=901s) it. Um if you're filling out a form, how [15:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=903s) do you make sure that you have precision [15:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=904s) in where you fill out fields? And a [15:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=906s) really good example of this uh on the [15:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=908s) orchestration side is I think [15:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=910s) classification splitting are a very [15:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=912s) underappreciated way to have an LM do [15:14](https://www.youtube.com/watch?v=0I07YAuF8xA&t=914s) its best work. Obviously, you can just [15:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=916s) go through and dump as much context as [15:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=918s) you want. And if you're doing a sort of [15:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=919s) needle in the haststack test, that might [15:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=921s) be fine. But in practice, there is [15:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=923s) erosion that you find in quality outside [15:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=925s) of just the token economics as a result [15:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=927s) of passing in too much. And instead, [15:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=930s) what we find is you can get a lot of [15:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=933s) headroom by thinking through things like [15:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=935s) how do you classify the right documents [15:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=937s) to the right sort of pipeline? And even [15:39](https://www.youtube.com/watch?v=0I07YAuF8xA&t=939s) for large documents, how do you make [15:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=941s) sure that you're passing in the snippets [15:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=943s) uh that are actually relevant? We see [15:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=945s) use cases where people will have things [15:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=946s) like paper mail. Uh and these paper mail [15:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=949s) packets can be hundreds of pages long. [15:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=951s) You don't necessarily know what is going [15:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=953s) to be contained within it. Um you might [15:55](https://www.youtube.com/watch?v=0I07YAuF8xA&t=955s) have issues like a person interle the [15:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=957s) content and having the model do that [15:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=959s) sort of work is almost like a [16:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=961s) distraction from the work that you're [16:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=963s) actually trying to achieve which might [16:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=964s) be extracting the data from the paper [16:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=966s) mail reasoning on it or making a [16:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=967s) decision. [16:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=970s) Again, uh I mentioned earlier that the [16:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=973s) second half I think is the more [16:14](https://www.youtube.com/watch?v=0I07YAuF8xA&t=974s) interesting piece. Um which is once you [16:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=977s) have the sort of initial classification [16:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=980s) and splitting layer, you've figured out [16:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=982s) sorting. Uh I think the thing that we [16:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=985s) are really excited about as a team is [16:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=987s) agent harnesses have been this really [16:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=989s) really interesting frontier to push past [16:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=991s) what canonically used to be hard [16:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=994s) unsolved problems. Um one good example [16:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=997s) of this that I'll talk about in a second [16:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=998s) is things like line charts. uh we work [16:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1000s) with many of the largest hedge funds in [16:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1002s) the world and things like line charts [16:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1004s) historically have been really really [16:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1005s) difficult because one they're an imaged [16:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1006s) format but two there's a lot of pixel [16:48](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1008s) level granularity that if you're doing [16:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1010s) anything with a traditional vision [16:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1012s) encoder you're probably going to lose [16:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1014s) you're going to get a rough plot of how [16:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1016s) revenue trended but you're not going to [16:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1017s) get the individual data points and so [16:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1019s) we've been thinking through how do we [17:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1020s) give agents the ability to have the [17:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1021s) right tools to solve for the specific [17:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1024s) type of problem that you're looking at [17:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1026s) in the case of chart extraction the [17:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1028s) chart on the left encodes codes a [17:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1030s) massive table of data. If you actually [17:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1032s) went through and tried to plot every [17:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1033s) single pixel, it would be really really [17:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1035s) difficult. U but it would also be hard [17:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1038s) for a model to even approximate the [17:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1040s) intricacies of the lines in between. And [17:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1042s) there's no model that out of the box can [17:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1044s) do this as a singleshot problem. What [17:26](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1046s) you're seeing on the right is a [17:28](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1048s) reconstruction of the markdown table [17:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1050s) that we're able to generate off of the [17:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1051s) initial line chart. And the only way [17:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1053s) that we were able to get there was to [17:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1055s) have an agent with all sorts of tools. [17:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1057s) It has its own code interpreter. It has [17:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1058s) the ability to visualize the chart that [17:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1060s) it's generating. And it's iteratively [17:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1062s) going through. It's finding mistakes in [17:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1063s) the line chart again and again and again [17:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1066s) until it's able to get to the final [17:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1067s) output. That applies for problems like [17:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1070s) structured extraction as well where for [17:52](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1072s) a while we've had this documents to [17:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1074s) structured output feature. Uh but you [17:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1076s) can really take it a step further by [17:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1078s) having an agent harness around that same [18:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1080s) sort of task. You can have a parent [18:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1081s) agent go through set validation criteria [18:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1084s) for sub agents to follow. Uh and this [18:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1087s) means that if you have something like a [18:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1089s) CBP form with tens of thousands of [18:11](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1091s) fields, that's the sort of problem where [18:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1093s) you end up finding a lot of issues that [18:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1095s) are silent in nature like you drop [18:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1097s) content, you drop rows and micro one [18:19](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1099s) actually released a really good [18:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1100s) benchmark in the space this morning [18:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1102s) where there's this like bifurcation in [18:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1105s) the market. Uh Frontier models with Max [18:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1107s) Reasoning are really really precise. [18:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1109s) Like provided that they extracted a row, [18:30](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1110s) odds are it's not a hallucinated row [18:32](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1112s) like they actually got it correct. but [18:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1114s) they silently drop a lot of the contents [18:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1116s) across the benchmark. Uh recall really [18:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1118s) really struggles. On the flip side, a [18:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1120s) lot of dedicated document processing [18:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1122s) services are actually behind frontier [18:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1124s) models from a precision perspective, but [18:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1127s) close that gap on a recall perspective. [18:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1129s) And so there's always been this sort of [18:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1131s) trade-off and it was only with an agent [18:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1133s) harness that we were able to find that [18:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1134s) sort of local maximum of both precision [18:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1137s) and also recall for this sort of task. [19:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1141s) The last thing and maybe the most [19:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1143s) important thing from this talk uh is [19:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1146s) that I think at the end of the day eval [19:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1149s) should underpin all of your decisions [19:11](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1151s) and it's been a big part of how we think [19:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1152s) about our product. Um that applies both [19:14](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1154s) to off-the-shelf data sets that you eval [19:17](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1157s) against but also to things like [19:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1158s) real-time production monitoring because [19:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1160s) your production data is going to differ [19:22](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1162s) from whatever else you have in your [19:24](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1164s) contrived set. And I really think it's [19:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1167s) important to think of eval not as just [19:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1169s) this like macrolevel view, but also the [19:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1173s) best teams that we work with look at [19:34](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1174s) eval on a granular level for each step [19:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1176s) of their pipeline. Uh the first thing [19:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1178s) might be that you want to make sure that [19:39](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1179s) the inputs to your pipeline are great. [19:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1181s) And of course, you should eval things [19:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1183s) like your parsing pipeline. But even [19:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1185s) perfect parsing with a horrible [19:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1187s) retrieval pipeline is not going to help [19:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1189s) if you're not passing the right context. [19:50](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1190s) And so it's important that you're [19:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1191s) thinking through details like your [19:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1193s) retrieval pipeline, your formatting at [19:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1196s) the end of the pipeline, and also [19:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1197s) ultimately the most important thing is [19:59](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1199s) are you able to improve end agent [20:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1201s) performance. [20:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1203s) I'll close off just with a a sense of [20:06](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1206s) where we are headed and where we've seen [20:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1207s) the industry head. Uh the most important [20:10](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1210s) thing I think is as agents get better [20:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1212s) and better, you can deviate from the [20:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1215s) sort of deterministic pipeline that you [20:16](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1216s) would have had a few years ago. Um, a [20:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1218s) lot of our customers will actually [20:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1220s) create effectively a file system for [20:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1221s) their agent to go through and navigate [20:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1223s) and let the agent decide what sorts of [20:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1225s) tools it wants to use. So, we create a [20:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1227s) CLI where instead of people creating a [20:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1229s) endto-end pipeline where documents [20:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1231s) always follow one specific flow, um, the [20:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1233s) agent will decide if it needs to read a [20:35](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1235s) certain type of document and they'll [20:37](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1237s) split that into two sets. Uh, one is a [20:39](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1239s) content field which the agent can read [20:41](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1241s) as it needs to. Uh, but the other is all [20:44](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1244s) the metadata that you would have wanted. [20:45](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1245s) If you're doing things like citations, [20:47](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1247s) you may want bounding boxes, so on and [20:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1249s) so forth. [20:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1251s) Um, I'll skip this part on editing. I [20:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1253s) think there's a lot of interesting work [20:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1254s) being done here. We've already released [20:56](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1256s) some of it, but in the next few months, [20:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1258s) you will see us look more and more [21:00](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1260s) towards things like document generation. [21:02](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1262s) But the recap for today, and I really [21:04](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1264s) appreciate your time, is one, I highly [21:07](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1267s) recommend that you decompose the parsing [21:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1269s) problem. To the extent possible, you [21:11](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1271s) should think of it as the right tool for [21:12](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1272s) the right task so that you can hit that [21:13](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1273s) perimeter frontier of accuracy, cost, [21:15](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1275s) and latency. Two, I think agentic [21:18](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1278s) verification is the biggest step change [21:20](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1280s) that the industry has had for a while [21:21](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1281s) and it's a really good opportunity for [21:23](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1283s) you to make sure that you're building [21:25](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1285s) pipelines that work in production. [21:27](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1287s) Three, I think it takes very little [21:29](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1289s) effort, but there's a lot of headroom [21:31](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1291s) from details like formatting the data [21:33](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1293s) for its consumer. or similar vein, I [21:36](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1296s) think it's really really important to [21:38](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1298s) think about not just the data processing [21:40](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1300s) but the the data orchestration and so [21:42](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1302s) you should always think about tools like [21:43](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1303s) classify and split as a way to augment [21:46](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1306s) your pipeline. Five uh make sure that [21:49](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1309s) you eval at every stage and six think [21:51](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1311s) through what that next frontier looks [21:53](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1313s) like for you because I think most [21:54](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1314s) successful companies in today's era have [21:57](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1317s) deviated a lot from what we used to do [21:58](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1318s) two three years ago. But if you have any [22:01](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1321s) questions uh please feel free to reach [22:03](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1323s) out at any point. My email is just first [22:05](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1325s) namered reductto.ai. Uh and you can also [22:08](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1328s) reach out on our website if we can be [22:09](https://www.youtube.com/watch?v=0I07YAuF8xA&t=1329s) helpful for your use case. Thank you.
Q Rev
USEU