From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto

A bad document parse can spoil a chatbot answer. In an agent workflow, the same mistake can travel through several decisions and into the finished work. That is the practical problem behind Adit Abraham’s talk. As Reducto’s co-founder and CEO, he draws on processing many billions of customer documents, including historical material that enterprises have struggled to use beyond a demo.

The earlier retrieval-augmented generation pattern mostly synthesized information: retrieve context, then answer a question through enterprise search or a chatbot. Agents expand the job. They make decisions, produce work, and generate or modify documents. Input quality matters throughout that longer sequence because each step can carry forward an earlier misunderstanding while adding information from another source.

Enterprise information also arrives without a tidy inventory. Different teams put files in Google Drive, Box, and other repositories; the contents are scattered, unstructured, and multimodal. A useful document system therefore has several jobs: recover the contents accurately, retrieve the relevant context, and support the interactions or modifications the task requires. Parsing is the first problem, but it does not settle the others.

PDFs were designed to reproduce a document’s appearance faithfully and make it printable. The present task asks for something different: a representation an agent can reason over. Abraham introduces the difficulty with a document-decision benchmark on which a frontier model scored about 30%. That figure describes the benchmark he cites, rather than a general PDF accuracy rate.

Humans encode meaning visually. A financial deck’s author is thinking about readers, rather than whether an agent can consume the slide. Abraham’s memorable example is SoftBank’s imagery of a goose laying eggs: a picture can carry part of the explanation. More routinely, merged cells establish relationships in tables, lines encode numerical trends, and handwriting holds information that even a human may struggle to read. Extracting characters alone can leave those relationships behind.

A traditional document pipeline ran OCR and then processed the resulting text. That approach could work well when the layout stayed consistent: a W-2 presents a constrained problem that templates can address. Vision-language models (VLMs) widened the possibilities by reading visual content across unfamiliar layouts, with handwriting as a particularly useful case. Their generality makes the long tail more approachable.

At hundreds of millions of documents, however, efficiency and determinism also shape the design. The work can be divided between complementary capabilities:

• Layout detection: Small computer-vision models identify document regions and can run on CPUs at scale. They help locate the difficult parts before more expensive processing.
• Semantic interpretation: VLMs handle challenging visual content and help identify mistakes that simpler processing leaves behind.

This division gives each task a suitable tool instead of making every region depend on a large foundation model.

Once the page has been segmented and difficult regions located, a verification layer can play a role similar to a human reviewer. Reducto calls this agentic OCR. The important decision is to apply targeted token edits to an initial transcription, rather than ask a model to regenerate the whole page. Abraham compares the approach to fast edits in an IDE: preserve the existing output and change the small pieces that need correction.

Why restrict an intelligent model’s freedom? Consider a table with a total that its human author calculated incorrectly. A model rewriting the OCR can recognize the word “total,” add the entries, and replace the printed value with the correct arithmetic result. The output becomes less faithful precisely because the model understood the table. The desired correction is narrower: distinguish 0 from O, or a period from a comma, while retaining the source’s own mistake. Reading a document and checking its arithmetic are separate jobs.

A correct parse still needs a useful representation. Simple tables fit naturally in Markdown. Complex tables may need HTML because merged cells encode relationships that must survive conversion. HTML preserves that structure, but its tags consume tokens; applying it to every simple table adds overhead without a corresponding benefit. Reducto chooses the representation according to the table’s complexity.

Now follow that complex table into retrieval. A user asks, “how did revenue change over time,” without enumerating the table’s values. The reasoning model may understand the HTML perfectly once it receives it. The embedding model first has to find it among many documents, though, and Reducto finds that matching an ordinary language query to a mass of tags and numbers is difficult. A representation that serves reasoning can therefore obstruct the step that supplies the reasoning context.

The change is to create a second, natural-language rendering of the same table. Retrieval uses that block; reasoning uses the structured HTML. What does each consumer receive? The diagram makes the division visible: the retrieval-friendly text helps select the relevant table, while the reasoning-friendly form preserves its relationships. This addresses the matching problem without requiring the reasoning model to work from the natural-language rendering alone.

Returning to the document-decision benchmark, Abraham reports that supplying structured parse results alongside the original PDF improved answers across models from Google, Anthropic, and OpenAI. Some tested models surpassed the earlier frontier model’s out-of-the-box result. The comparison does not establish what that earlier model would do with the same improved inputs: Reducto could not test it after losing access.

The reported benefit also included fewer reasoning tokens and lower latency. Structured inputs let the model spend less effort reconstructing the data and more effort producing the answer. Improving the input pipeline can thus change both the quality and the amount of downstream work, without changing the model itself.

The next improvement concerns which documents enter the task. A large context window does not make every page helpful; Abraham describes quality erosion from excessive context as well as token cost. Two operations address different parts of that problem:

• Classification: Route the right document to the appropriate processing pipeline.
• Splitting: Select the relevant portions of a large document instead of passing the entire packet.

A paper-mail packet can run to hundreds of pages with uncertain contents. Making the final decision model sort through that packet adds a separate job that distracts from extracting information, reasoning over it, or making the intended decision.

Line charts expose a harder extraction problem. A vision encoder may retain the broad trend in revenue while losing the pixel-level detail needed to recover individual data points. The chart contains a potentially large table, but a rough visual reading does not recover that table. Abraham presents a case his team could not solve with a single model response.

The agent harness changes the task from a one-shot reading into an iterative reconstruction. The agent has a code interpreter and can visualize the chart it generates from extracted data. It examines that reconstruction, finds mistakes, and revises the extraction repeatedly. The reported result is a Markdown table recovered from the original line chart, with a plotted reconstruction shown in the presentation. The talk does not give a numerical error tolerance for the recovered points, so the example demonstrates the method rather than establishing exact numerical recovery.

Where does the improvement come from? The loop below makes the generated chart inspectable before the table is accepted. Tools give the agent a way to expose errors in its proposed data, and repeated checks let it change that proposal. The useful capability is the whole loop—extraction, execution, visualization, and correction.

Structured extraction can use a similar harness. A parent agent sets validation criteria for subagents, which matters when a form contains tens of thousands of fields and omissions can remain silent. Abraham describes a benchmark tradeoff: frontier models at maximum reasoning were precise about the rows they returned, yet dropped substantial content; dedicated document services recovered more content while making more errors in what they extracted. Precision asks whether returned content is correct. Recall asks how much required content was recovered. Reducto reports that its harness improved both for this task, though the talk supplies no scores to quantify the gain.

Evaluations should guide these design choices at two levels. Off-the-shelf datasets provide a test bed, while production monitoring catches the differences between a constructed dataset and the documents users actually submit. Within the pipeline, parsing, retrieval, and final formatting each need attention. Perfect parsing cannot help an agent that receives the wrong context. The final question remains whether the changes improve the agent’s completed work.

The closing architecture moves some orchestration into the agent itself. Abraham describes customers giving agents a file system to navigate and a CLI through which they can choose document tools as needed. A document no longer has to follow one predetermined end-to-end flow. The agent decides whether it needs to read a particular document and which operation to apply.

That interface separates readable content from metadata. The content field supplies what the agent needs to read; metadata retains information such as bounding boxes for citations. This gives the agent useful text while preserving a way to locate the supporting material in the original document. Editing receives only a brief mention, and document generation is presented as a direction for the coming months rather than an explained capability in this talk.

Abraham’s final recommendation is to reconsider the workflow as capabilities change. Decompose parsing to balance accuracy, cost, and latency; use verification to catch errors; format data for its consumer; classify and split before downstream work; and evaluate every stage. The file-system example extends those choices: tools and document representations can remain well defined even when the agent chooses the sequence in which to use them.