Building AI Agents that actually automate Knowledge Work

Increasing efficiency and improving decisions sound useful, but what does an agent actually do to deliver them? Does automating knowledge work mean building a RAG chatbot, or can it carry out the work that follows an answer? That is Jerry Liu’s opening question.

The concrete setting is a worker reading documents to make decisions and take actions. Liu estimates that 90% of enterprise data lives in documents. Those documents include PDFs, PowerPoints, Word files, and spreadsheets: the material an investment banker or customer support worker must interpret before doing anything useful. Agents extend that sequence by analyzing large amounts of unstructured context, researching questions, synthesizing findings, and taking actions.

Two kinds of products follow from that possibility:

• Assistive agents help a human get information faster, usually through a chat interface.
• Automation agents perform routine tasks, potentially in the background, with less continuous human involvement.

Both work over knowledge, but they ask different things of the person using them.

Either category needs two components: useful tools and an agent architecture. Tools connect the agent to external context and actions—the interface problem Liu associates with MCP and A2A. Orchestration determines how those capabilities serve the task. A general reasoning loop gives the model room to choose its next action; a constrained workflow expresses more of the business logic in explicit control flow.

The document toolbox and the workflow solve different problems. First, documents must become accessible through reliable operations. Then an architecture must coordinate those operations. The applications emerge from combining the two.

A general interface such as Claude or ChatGPT still needs tools to interact with enterprise documents. Liu’s baseline here is naive RAG: retrieval followed by one-shot synthesis. A document agent needs that capability, but it may also need to locate a particular file, inspect its metadata, or manipulate it. Retrieval is one operation in a larger toolbox.

That toolbox depends on a preprocessing pipeline:

1. Connect and synchronize. Bring data from sources such as SharePoint, Google Drive, S3, and Confluence into an accessible representation. Synchronize permissions and metadata along with the content.
2. Parse and extract. Recover the meaning and structure of documents, including tables and charts.
3. Index for the intended operations. Store the resulting representations in a vector index, SQL tables, a graph database, or another suitable system.

The storage choice follows what the agent needs to do; a vector index is not the only possible destination.

Liu calls the resulting collection of interfaces a Document MCP server. Its purpose is broader than exposing a single retrieval endpoint:

Processing and structuring the source material is what makes these interfaces useful. An agent cannot obtain dependable aggregate results merely because a tool has been given a convenient name.

Complex documents are designed for human consumption. Embedded tables, charts, images, irregular layouts, headers, and footers all carry information that can disappear or become confused during parsing. If preprocessing damages that information, a stronger downstream model cannot reliably recover it. Document understanding is part of the agent’s reasoning infrastructure.

Traditional approaches often use task-specific models tuned for particular document classes. Language and vision models offer a more general starting point: even sending a screenshot of a PDF to a model can recover useful information. But Liu treats screenshot prompting as a baseline, not the complete solution. His approach interleaves traditional parsing with language and vision models, then spends additional test-time tokens on agentic validation and reasoning.

The parsing service described in the talk adapts models including Claude Sonnet 3.5, Claude Sonnet 4, Gemini 2.5 Pro, and OpenAI GPT-4.1. Liu claims its parsing modes outperform existing open-source and proprietary alternatives, but supplies no benchmark protocol or numerical results in this passage. The managed platform packages these capabilities as parsing, extraction, and indexing operations for the document toolbox.

The Spreadsheet Agent extends the toolbox to a familiar but difficult document format. Excel, Google Sheets, and Numbers contain much of the work people already do, yet a sheet need not be a clean two-dimensional table. The example in the talk has gaps in both rows and columns. Treating that arrangement as ordinary tabular data misses the structure that gives its cells meaning.

The agent transforms an unnormalized workbook into a normalized two-dimensional format and supports question answering over both the original and normalized versions. Described at the time as an early preview, it complements parsing, extraction, and indexing. The demonstration uploads a synthetic dataset, transforms it into a table, and then asks questions to obtain insights. The key work is understanding the spreadsheet’s semantic structure and making that understanding available through specialized tools.

For this task, Liu identifies an LLM with a code interpreter as the meaningful baseline, rather than RAG or text-to-CSV techniques. He reports the following approximate accuracy figures in the context of a private evaluation he describes as using synthetic Excel sheets:

These are reported evaluation results, not a general guarantee of spreadsheet performance. The talk does not give a sample size, a scoring definition, or a task breakdown that would isolate normalization accuracy from question answering.

The mechanism begins with structure understanding. Liu describes using reinforcement learning and adapting to the document’s particular format to learn a semantic map of the sheet. He does not provide a training recipe or establish that the model learns new weights while processing each workbook. The useful architectural point is the intermediate representation: understand how the sheet is organized before asking an agent to operate on it.

That semantic map becomes a set of specialized tools supplied to the agent. Instead of discovering the entire structure and writing every operation from scratch, the agent reasons through capabilities derived from the map. General code generation remains a baseline that Liu expects to improve as models improve; the specialized tools help it handle spreadsheet structure in the meantime.

Once the document tools are available, the next decision is how much control flow to specify. At the constrained end, the application explicitly defines the steps. At the more open-ended end, it gives the model tools and lets it choose how to proceed. Liu places ReAct, function calling, CodeAct, and deep research in this latter family.

An assistant interface usually accepts natural-language input and either surfaces information or produces a unit of knowledge work. A flexible loop fits that interaction because the user can guide, correct, and coax the agent through the task. More freedom for the agent can coexist with more human involvement. The chat interface keeps the person available to steer execution, generalizing the familiar RAG chatbot into a broader working assistant.

Automation changes the interaction contract. Instead of helping a person obtain information one exchange at a time, the system processes a routine task through multiple steps. It can accept a batch of inputs, run in the background, or start when a person triggers it. Because the person is no longer steering every step, the architecture is usually more constrained.

Reduced involvement during execution does not mean removing review. Liu describes batch review near the end: the workflow produces structured results or decisions, and approval precedes routing to downstream systems through API integrations. That boundary separates a prepared result from an authorized external action.

Financial data normalization, datasheet extraction, invoice reconciliation, and contract review fit this pattern. Liu also points to Laurie Voss’s Invoice Reconciler as a community example, though he skips its video rather than demonstrating its implementation.

The two interfaces can form layers of the same application. Automation runs in the background, performing extraction, transformation, structuring, and decisions, with human review still present. An assistant provides the foreground interaction over the resulting information.

This creates a useful composition: an automation pipeline processes unstructured context and exposes tools that a research assistant can use. The pipeline may itself need agentic reasoning to handle varied documents. But not every tool requires an agent; ordinary deterministic operations can remain ordinary tools. What matters is that the assistant receives usable capabilities over prepared data.

Liu’s Carlyle example combines both interfaces in a leveraged-buyout agent. The automation side ingests public and private financial material across Excel sheets, PDFs, and PowerPoints. Bespoke extraction algorithms turn those documents into structured information, with human review in the extraction process.

Only after that preparation does the analyst-facing copilot become useful. It works over the structured data to help teams obtain insights and generate reports. In this customer example, the assistant’s capabilities depend on the ingestion and review work behind it; the chat interface is the visible end of a larger document workflow.

Enterprise search sits primarily on the assistant side. Define collections for different data sources, then provide task-specific agents over those collections. The reasoning layer expands retrieval into a process: break down the user’s query, research the relevant information, and synthesize an answer. The product still looks like a search or chat interface, but the work behind the response can involve several steps.

The final example returns to automation: technical datasheet ingestion for an unnamed global electronics company. Historically, people had to process and review large numbers of these documents. The workflow encodes business-specific logic to parse each datasheet, extract the required information, match it against rules, and write structured results into SQL. The accompanying diagram connects datasheets to a LlamaCloud attribute extraction agent, a SQL tool, and a LlamaIndex design copilot.

Human review remains part of the workflow. Liu describes the potential to replace weeks of technical-writer work if the process can run end to end; he does not report a measured speedup. The substantive change is a complete path from source documents through business rules to structured output, with people reviewing the work instead of manually performing every extraction.