# Building AI Agents that actually automate Knowledge Work

Jerry Liu

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/jVGCulhBRZI-building-ai-agents-that-actually-automate) · [Video](https://www.youtube.com/watch?v=jVGCulhBRZI)

## Building AI Agents That Automate Knowledge Work

Document automation needs more than retrieval: it needs tools that preserve document structure and workflows that connect analysis, human review, and action.

Familiarity with retrieval-augmented generation (RAG) and LLM tool calling will help you follow the architecture discussion.

### Does knowledge work automation just mean RAG chatbots?

[0:26](https://www.youtube.com/watch?v=jVGCulhBRZI&t=26s)

Increasing efficiency and improving decisions sound useful, but what does an agent actually do to deliver them? Does automating knowledge work mean building a RAG chatbot, or can it carry out the work that follows an answer? That is Jerry Liu’s opening question.

![Slide listing efficiency promises beside four vendor website screenshots and asking whether knowledge-work automation just means RAG chatbots.](https://ai.engineer/api/talk-essay-media/a1350972ca02ffc50e551c61f5f460c69165b6bc06e022f9129b0759d885e439)

Does knowledge-work automation just mean RAG chatbots?

The concrete setting is a worker reading documents to make decisions and take actions. Liu estimates that 90% of enterprise data lives in documents. Those documents include PDFs, PowerPoints, Word files, and spreadsheets: the material an investment banker or customer support worker must interpret before doing anything useful. Agents extend that sequence by analyzing large amounts of unstructured context, researching questions, synthesizing findings, and taking actions.

Two kinds of products follow from that possibility:

- **Assistive agents** help a human get information faster, usually through a chat interface.
- **Automation agents** perform routine tasks, potentially in the background, with less continuous human involvement.

Both work over knowledge, but they ask different things of the person using them.

### Tools expose capabilities; orchestration encodes the task

[2:20](https://www.youtube.com/watch?v=jVGCulhBRZI&t=140s)

Either category needs two components: useful tools and an agent architecture. Tools connect the agent to external context and actions—the interface problem Liu associates with MCP and A2A. Orchestration determines how those capabilities serve the task. A general reasoning loop gives the model room to choose its next action; a constrained workflow expresses more of the business logic in explicit control flow.

**The document toolbox and the workflow solve different problems.** First, documents must become accessible through reliable operations. Then an architecture must coordinate those operations. The applications emerge from combining the two.

### Build the preprocessing layer behind the tools

[3:27](https://www.youtube.com/watch?v=jVGCulhBRZI&t=207s)

A general interface such as Claude or ChatGPT still needs tools to interact with enterprise documents. Liu’s baseline here is **naive RAG**: retrieval followed by one-shot synthesis. A document agent needs that capability, but it may also need to locate a particular file, inspect its metadata, or manipulate it. Retrieval is one operation in a larger toolbox.

That toolbox depends on a preprocessing pipeline:

1. **Connect and synchronize.** Bring data from sources such as SharePoint, Google Drive, S3, and Confluence into an accessible representation. Synchronize permissions and metadata along with the content.
2. **Parse and extract.** Recover the meaning and structure of documents, including tables and charts.
3. **Index for the intended operations.** Store the resulting representations in a vector index, SQL tables, a graph database, or another suitable system.

The storage choice follows what the agent needs to do; a vector index is not the only possible destination.

Liu calls the resulting collection of interfaces a **Document MCP server**. Its purpose is broader than exposing a single retrieval endpoint:

| Operation | What it provides |
| --- | --- |
| Semantic search | Relevant content through fuzzy matching |
| File lookup | A file and its metadata |
| File manipulation | Operations on the files themselves |
| Structured querying | Aggregate insights over extracted data |

Processing and structuring the source material is what makes these interfaces useful. An agent cannot obtain dependable aggregate results merely because a tool has been given a convenient name.

### Preserve the structure that reasoning depends on

[5:56](https://www.youtube.com/watch?v=jVGCulhBRZI&t=356s)

Complex documents are designed for human consumption. Embedded tables, charts, images, irregular layouts, headers, and footers all carry information that can disappear or become confused during parsing. If preprocessing damages that information, a stronger downstream model cannot reliably recover it. **Document understanding is part of the agent’s reasoning infrastructure.**

Traditional approaches often use task-specific models tuned for particular document classes. Language and vision models offer a more general starting point: even sending a screenshot of a PDF to a model can recover useful information. But Liu treats screenshot prompting as a baseline, not the complete solution. His approach interleaves traditional parsing with language and vision models, then spends additional test-time tokens on agentic validation and reasoning.

The parsing service described in the talk adapts models including **Claude Sonnet 3.5**, **Claude Sonnet 4**, **Gemini 2.5 Pro**, and **OpenAI GPT-4.1**. Liu claims its parsing modes outperform existing open-source and proprietary alternatives, but supplies no benchmark protocol or numerical results in this passage. The managed platform packages these capabilities as parsing, extraction, and indexing operations for the document toolbox.

### A spreadsheet is not necessarily a table

[8:15](https://www.youtube.com/watch?v=jVGCulhBRZI&t=495s)

The [Spreadsheet Agent](https://www.llamaindex.ai/blog/introducing-the-spreadsheet-agent-in-private-preview) extends the toolbox to a familiar but difficult document format. Excel, Google Sheets, and Numbers contain much of the work people already do, yet a sheet need not be a clean two-dimensional table. The example in the talk has gaps in both rows and columns. Treating that arrangement as ordinary tabular data misses the structure that gives its cells meaning.

The agent transforms an unnormalized workbook into a normalized two-dimensional format and supports question answering over both the original and normalized versions. Described at the time as an early preview, it complements parsing, extraction, and indexing. The demonstration uploads a synthetic dataset, transforms it into a table, and then asks questions to obtain insights. The key work is understanding the spreadsheet’s semantic structure and making that understanding available through specialized tools.

For this task, Liu identifies an LLM with a code interpreter as the meaningful baseline, rather than RAG or text-to-CSV techniques. He reports the following approximate accuracy figures in the context of a private evaluation he describes as using synthetic Excel sheets:

| Approach | Reported accuracy |
| --- | --- |
| LLM with a code interpreter | Roughly 70–75% |
| LlamaIndex Excel agent | 95% |
| Human manual transformation | 90% |

These are reported evaluation results, not a general guarantee of spreadsheet performance. The talk does not give a sample size, a scoring definition, or a task breakdown that would isolate normalization accuracy from question answering.

![Bar chart comparing two LlamaIndex Excel agents with two OpenAI code interpreter configurations, with confidence intervals and a dashed human baseline at 90%.](https://ai.engineer/api/talk-essay-media/b612749b2b29e0afd59a97abfaa2401287e303cfdc105d5b7d6248a517ac34f1)

Excel agent and code interpreter benchmark results, with a human baseline.

### Turn spreadsheet structure into specialized tools

[10:17](https://www.youtube.com/watch?v=jVGCulhBRZI&t=617s)

The mechanism begins with structure understanding. Liu describes using reinforcement learning and adapting to the document’s particular format to learn a **semantic map** of the sheet. He does not provide a training recipe or establish that the model learns new weights while processing each workbook. The useful architectural point is the intermediate representation: understand how the sheet is organized before asking an agent to operate on it.

That semantic map becomes a set of specialized tools supplied to the agent. Instead of discovering the entire structure and writing every operation from scratch, the agent reasons through capabilities derived from the map. General code generation remains a baseline that Liu expects to improve as models improve; the specialized tools help it handle spreadsheet structure in the meantime.

### Flexible assistants rely on human guidance

[11:10](https://www.youtube.com/watch?v=jVGCulhBRZI&t=670s)

Once the document tools are available, the next decision is how much control flow to specify. At the constrained end, the application explicitly defines the steps. At the more open-ended end, it gives the model tools and lets it choose how to proceed. Liu places [ReAct](https://arxiv.org/abs/2210.03629), function calling, [CodeAct](https://arxiv.org/abs/2402.01030), and deep research in this latter family.

An assistant interface usually accepts natural-language input and either surfaces information or produces a unit of knowledge work. A flexible loop fits that interaction because the user can guide, correct, and coax the agent through the task. **More freedom for the agent can coexist with more human involvement.** The chat interface keeps the person available to steer execution, generalizing the familiar RAG chatbot into a broader working assistant.

### Constrain automation and preserve an approval boundary

[12:41](https://www.youtube.com/watch?v=jVGCulhBRZI&t=761s)

Automation changes the interaction contract. Instead of helping a person obtain information one exchange at a time, the system processes a routine task through multiple steps. It can accept a batch of inputs, run in the background, or start when a person triggers it. Because the person is no longer steering every step, the architecture is usually more constrained.

Reduced involvement during execution does not mean removing review. Liu describes batch review near the end: the workflow produces structured results or decisions, and approval precedes routing to downstream systems through API integrations. That boundary separates a prepared result from an authorized external action.

Financial data normalization, datasheet extraction, invoice reconciliation, and contract review fit this pattern. Liu also points to Laurie Voss’s [Invoice Reconciler](https://github.com/run-llama/invoice-reconciler) as a community example, though he skips its video rather than demonstrating its implementation.

### Background automation can supply foreground assistants

[14:01](https://www.youtube.com/watch?v=jVGCulhBRZI&t=841s)

The two interfaces can form layers of the same application. Automation runs in the background, performing extraction, transformation, structuring, and decisions, with human review still present. An assistant provides the foreground interaction over the resulting information.

This creates a useful composition: an automation pipeline processes unstructured context and exposes tools that a research assistant can use. The pipeline may itself need agentic reasoning to handle varied documents. But not every tool requires an agent; ordinary deterministic operations can remain ordinary tools. What matters is that the assistant receives usable capabilities over prepared data.

### Financial due diligence combines ingestion and a copilot

[15:13](https://www.youtube.com/watch?v=jVGCulhBRZI&t=913s)

Liu’s Carlyle example combines both interfaces in a leveraged-buyout agent. The automation side ingests public and private financial material across Excel sheets, PDFs, and PowerPoints. Bespoke extraction algorithms turn those documents into structured information, with human review in the extraction process.

Only after that preparation does the analyst-facing copilot become useful. It works over the structured data to help teams obtain insights and generate reports. In this customer example, the assistant’s capabilities depend on the ingestion and review work behind it; the chat interface is the visible end of a larger document workflow.

### Enterprise search adds reasoning over collections

[15:59](https://www.youtube.com/watch?v=jVGCulhBRZI&t=959s)

Enterprise search sits primarily on the assistant side. Define collections for different data sources, then provide task-specific agents over those collections. The reasoning layer expands retrieval into a process: break down the user’s query, research the relevant information, and synthesize an answer. The product still looks like a search or chat interface, but the work behind the response can involve several steps.

### Datasheet ingestion turns business rules into structured output

[16:30](https://www.youtube.com/watch?v=jVGCulhBRZI&t=990s)

The final example returns to automation: technical datasheet ingestion for an unnamed global electronics company. Historically, people had to process and review large numbers of these documents. The workflow encodes business-specific logic to parse each datasheet, extract the required information, match it against rules, and write structured results into SQL. The accompanying diagram connects datasheets to a LlamaCloud attribute extraction agent, a SQL tool, and a LlamaIndex design copilot.

![Workflow diagram for a global electronics company: datasheets flow into a LlamaCloud attribute extraction agent, then a SQL tool, and finally a LlamaIndex design copilot.](https://ai.engineer/api/talk-essay-media/06448c0c74e820fec55b8a629c0c195c4b6df824c16df5de9ec4fd2299074e43)

Technical datasheet ingestion connects attribute extraction, a SQL tool, and a design copilot.

Human review remains part of the workflow. Liu describes the potential to replace weeks of technical-writer work if the process can run end to end; he does not report a measured speedup. The substantive change is a complete path from source documents through business rules to structured output, with people reviewing the work instead of manually performing every extraction.

## Resources

- [Introducing the Spreadsheet Agent](https://www.llamaindex.ai/blog/introducing-the-spreadsheet-agent-in-private-preview) — Launch article explaining spreadsheet normalization, semantic maps, specialized tools and the reported private evaluation.
- [LlamaIndex Invoice Reconciler](https://github.com/run-llama/invoice-reconciler) — Example application that matches invoices to contracts and flags discrepancies for review. Includes setup instructions and a warning about insecure demonstration credential storage.
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629) — Original research on interleaving reasoning with actions and observations.
- [Executable Code Actions Elicit Better LLM Agents](https://arxiv.org/abs/2402.01030) — Introduces CodeAct, using executable Python as an agent action space with interpreter feedback.
- [LLM APIs Are Not Complete Document Parsers](https://www.llamaindex.ai/blog/llm-apis-are-not-complete-document-parsers) — Liu's later explanation of hybrid document parsing, metadata preservation and operational requirements beyond screenshot prompting.

## Slides

### AI Agents Can Automate Knowledge Work

B2B SaaS vendors promise lower costs and better decisions from AI agents. The speaker asks what knowledge work automation means in practice, and whether it amounts to RAG chatbots.

[0:26](https://www.youtube.com/watch?v=jVGCulhBRZI&t=26s) [Benefits of AI agents for knowledge workers alongside vendor screenshots and a question about RAG chatbots.](https://assets.ai.engineer/sites/aiecode2025/sha256/5886e82c02f2bb6924b0b2ebe6343bff2a1c334b2037db61aea83a2010727590.webp)

AI Agents can “Automate Knowledge Work”

A big promise of AI agents is making knowledge workers more efficient:
• Lower cost, save time
• More data, better decision-making

Every b2b SaaS is talking about it.

But does “knowledge work automation” just mean RAG chatbots?

SAP Business AI
Create transformative impact with the most powerful AI and agents fueled by the context of all your business data.

Put AI Agents and Robots to work on your processes
Bring the power of Agentic AI and RPA to your enterprise workflows—securely, reliably, and at scale

Build your foundation for actionable AI.
Integration is the key to unlocking the potential of AI agents. Use MuleSoft’s all-in-one platform to build your foundation for agent action by connecting your systems, models and vector stores, orchestrating complex workflows, and securing mission critical APIs.

Work AI for all.
Give every employee an AI Assistant and Agents that put your company’s knowledge to work.

2

Interface text:
SAP; UiPath; MuleSoft; glean
Request a demo; Try UiPath Free; Explore the platform; Watch now; Talk to an expert; Get a demo; Watch video
Engineering; Sales; Customer Support

### The Alpha Is in Unstructured Data

The slide claims 90% of enterprise data lives in documents. It contrasts humans reading and writing those documents with AI agents reasoning and acting over unstructured context.

[1:00](https://www.youtube.com/watch?v=jVGCulhBRZI&t=60s) [A diagram compares humans and agents working with documents, alongside a claim about enterprise data.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a1cbf6fc6336b8c4f4274cf4387f3882d0c3c74e151135fdfdb24c414f0838c.webp)

The Alpha is in Unstructured Data

90% of Enterprise Data Lives in Documents*

Humans historically needed to read/write these documents.

For the first time, AI agents can reason and act over massive amounts of unstructured context tokens.

Human
Agent
XLS
W

[illegible]ped Value: What Every Executive Needs to Know About Unstructured Data [White paper]. IDC https://www.box.com/resources/unstructured-data-paper

3

### The Unlock Is in the Tools and Architecture

Assistive agents help people get information faster; automation agents handle routine tasks in the background. Both need tools for context and external actions, plus architecture that encodes business logic.

[1:49](https://www.youtube.com/watch?v=jVGCulhBRZI&t=109s) [Assistive and automation agents are contrasted above boxes describing tools and architecture.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b932113148926859d7ea302bd6b313cf90213ae3f25bb8d42d7cdf81d2f28b2.webp)

The “Unlock” is in the Tools and Architecture

Helping Humans with Document Work

Assistive Agent
Help humans get more information, faster.

Automation Agent
Automate routine tasks in the background, take actions on behalf of the user.

Necessary ingredients for automation

Tools
Surface relevant context. Let agent take external actions.
Connect through MCP/A2A

+

Architecture
Encode business logic through agent reasoning.

4

### Building Advanced Document Agents

The talk moves from a document toolbox for interacting with unstructured documents to agent designs spanning assistance and automation, then brings them together in document agent use cases.

[2:58](https://www.youtube.com/watch?v=jVGCulhBRZI&t=178s) [Three topics: document toolboxes, agent design patterns, and document agent use cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/41d0faa2d168f57719d623b66ee1899b38eed61ed99e8d94c4f14ca36a288ba1.webp)

Building Advanced Document Agents

Building a document toolbox
(MCP Server)

Agent Design Patterns: From
Assistant to Automation UXs

Document Agent
Use Cases

6

### Building a Document Toolbox

The first section looks at tools that let AI agents interact with large amounts of unstructured documents.

[3:25](https://www.youtube.com/watch?v=jVGCulhBRZI&t=205s) [Section title with a server and toolbox illustration, shown above a small stage view.](https://assets.ai.engineer/sites/aiecode2025/sha256/a09875c03b82b8edd50aa82314969915c55cea87d642ab955c1b049e95896449.webp)

Building a Document Toolbox

AIE
aws

### Building a Document Toolbox

Connectors, parsing, extraction and indexing prepare documents for an MCP server, which gives an agent SQL, vector retrieval and file tools.

[3:30](https://www.youtube.com/watch?v=jVGCulhBRZI&t=210s) [Diagram showing data connectors, parsing, extraction and indexing feeding document tools for an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ec162b625d3459d0d6236cf0a9af4c78382e3057fad0e1e0a21ce2f337cce56.webp)

Building a Document Toolbox

Agents need the right tool interfaces over your documents.

Data

Data Connectors
• Sharepoint, GDrive, S3, Confluence
• Access Control Syncing
• Incremental Data Updates

Parsing
• Parse any doc type (.pdf, .docx, .pptx)
• Handles tables, charts, scans, images, fonts, multi-column layouts

Extraction
• Transform documents into clean, structured data

Indexing
• Multimodal chunking, embedding, indexing, advanced retrieval

Document MCP Server
SQL Tool
Vector Retrieval Tool
File Tool

Agent

To ensure the data is high quality, agents need a layer to process and structure documents into highly accurate data

8

### Tool Interfaces over Documents

A Document MCP server equips agents with file lookup, semantic retrieval, structured queries for aggregate insights, and functions for manipulating files.

[5:07](https://www.youtube.com/watch?v=jVGCulhBRZI&t=307s) [Document MCP server interfaces for file lookup, retrieval, structured querying and manipulation.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9c2eb42c9f5a435db8c37082373b7d3ff4a6ea5af75922ba79bfbf0b8985de9.webp)

Tool Interfaces over Documents

Document MCP Server

File Lookup: Query an API interface with precise parameters to look up files/metadata on the files.

Retrieval: Do semantic search to fuzzy-find relevant source of data (this is just RAG!)

Structured querying: Query a structured database to get aggregate insights.

Manipulation: Call a set of functions exposed by the file type to manipulate file

9

### Parse Performance

The chart places 14 highlighted LlamaParse configurations above Mistral OCR and the other listed parsers on general score. Other LlamaParse configurations appear further down the ranking.

[7:21](https://www.youtube.com/watch?v=jVGCulhBRZI&t=441s) [Bar chart comparing general scores for document parsers, with several LlamaParse modes highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/2320f8065f79325c737120d9b865ee1bf9901d6bd15bdbe2aa482e09ce306b2d.webp)

Parse Performance

Y-axis: General Score
Y-axis ticks: 0.0, 2.0, 4.0, 6.0, 8.0
X-axis: Parser
LlamaParse - parse_page_with_agent - anthropic-sonnet-3.5
LlamaParse - parse_page_with_agent - anthropic-sonnet-4.0
LlamaParse - parse_page_with_agent - gemini-2.5-pro
LlamaParse - parse_page_with_agent - anthropic-sonnet-3.7
LlamaParse - parse_page_with_lvm - anthropic-sonnet-3.7
LlamaParse - parse_page_with_lvm - anthropic-sonnet-3.5
LlamaParse - parse_page_with_lvm - gemini-1.5-pro
LlamaParse - parse_page_with_lvm - gemini-2.5-pro
LlamaParse - parse_page_with_layout_agent
LlamaParse - parse_page_with_lvm - gemini-1.5-flash
LlamaParse - parse_page_with_agent - gemini-2.0-flash-001
LlamaParse - parse_page_with_lvm - anthropic-sonnet-4.0
LlamaParse - parse_page_with_lvm - gemini-2.0-flash-001
LlamaParse - parse_page_with_lvm - openai-gpt-4-1
Mistral OCR
Marker
MinerU
LlamaParse - parse_page_with_lvm - openai-gpt4o
LlamaParse - parse_page_with_llm
azureDocIntelligence
Docling
AWS Textract
LlamaParse - parse_page_with_lvm - openai-gpt-4o-mini
pymupdf4llm
unstructured-highres
unstructured-fast

[illegible]rse Modes
12

### Excel: The Spreadsheet Challenge

The profit-and-loss sheet uses visual grouping and gaps rather than a regular 2D table. The speaker argues that this structure defeats text-to-CSV and retrieval baselines.

[8:14](https://www.youtube.com/watch?v=jVGCulhBRZI&t=494s) [Slide describing the difficulty of interpreting visually structured Excel sheets, alongside a synthetic profit-and-loss table.](https://assets.ai.engineer/sites/aiecode2025/sha256/505fa1198ea3a36dfb18f3947c0aaac2bae9f9b0de4946ce75df889b4416e9ea.webp)

Special Release: Excel!

A lot of knowledge work happens in Microsoft Excel, but it is unsolved by LLMs.

Excel sheets are visually structured for humans, not for machine readability

Baselines:
- Text-to-CSV
- Vector retrieval
- Dumping tokens all into the prompt

Profit and Loss
Acme Corp
For the month ended 30 June 2023

Account | Jun 2023 | May 2023 | Apr 2023 | Mar 2023 | Feb 2023 | Jan 2023 | Dec 2022

Income
Other Revenue | (411.60) | 4,067.33 | 2,574.08 | 9,587.45 | 26,570.28 | 28,598.44 | 51,359.65
Sales - SolarMate Panels | 1,634,877.72 | 1,824,809.19 | 1,431,798.08 | 1,692,396.14 | 1,407,866.93 | 1,939,293.79 | 1,534,068.19
Sales - GlowCharge Lamps | 12,941.20 | 44,630.76 | 13,153.71 | 59,880.85 | 71,652.49 | 93,975.01 | 17,122.30
Sales - EcoFlow Water Bottles | 353,170.82 | 64,870.56 | 74,317.12 | 149,776.46 | 21,767.12 | 35,449.07 | 23,788.46
Sales - AeroGrip Yoga Mats | 103,129.22 | 0.00 | 73,845.88 | 0.00 | 90,941.75 | 105,581.58 | 0.00
Sales - ThermaTech Jackets | 2,390,663.81 | 1,338,386.79 | 1,719,429.14 | 1,309,466.44 | 1,165,902.34 | 2,433,383.87 | 2,503,217.53
Total Income | 4,494,371.17 | 3,276,764.63 | 3,315,118.01 | 3,221,107.34 | 2,784,700.91 | 4,636,281.76 | 4,129,556.13

Cost of Sales
Damaged Goods | 113,397.91 | 116,956.73 | 111,092.98 | 110,125.12 | 210,269.27 | 311,074.34 | 17,230.41
AeroGrip Yoga Mats | 66,075.84 | 875.28 | 50,172.20 | 0.00 | 61,609.60 | 71,781.52 | 0.00
SolarMate Panels | 556,369.31 | 577,748.85 | 291,114.24 | 474,118.52 | 278,309.73 | 653,710.63 | 360,327.02
GlowCharge Lamps | 205,144.17 | 172,639.00 | 127,673.90 | 96,553.03 | 112,410.32 | 123,562.98 | 48,446.21
EcoFlow Water Bottles | 35,000.00 | 35,000.00 | 35,000.00 | 35,000.00 | 35,000.00 | 35,000.00 | 35,000.00
ThermaTech Jackets | 8,085.37 | 309,610.36 | 269,950.11 | 130,755.88 | 66,092.67 | 194,684.54 | 231,615.80
Freight (COGS) | 45,333.42 | 44,711.16 | 27,243.87 | 39,762.66 | 35,035.13 | 32,070.17 | 57,656.41
Inventory Adjustment | 17,610.50 | 780.66 | 0.00 | (196.43) | 0.00 | 52.52 | 85.45
Other COGS | 490.00 | 574.01 | 0.00 | 1,098.20 | 2,547.00 | 3,293.00 | 4,477.27
Packaging - GlowCharge Lamps | 91,174.32 | 27,941.20 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Packaging - SolarMate Panels | 14,000.00 | 14,000.00 | 14,000.00 | 14,000.00 | 14,000.00 | 14,000.00 | 14,000.00
Packaging - EcoFlow Water Bottles | 10,000.00 | 21,971.36 | 0.00 | 0.00 | 10,000.00 | 0.00 | 0.00
Packaging - ThermaTech Jackets | 53,204.84 | 34,364.72 | 4,417.20 | 10,000.00 | 6,000.00 | 6,000.00 | 6,000.00
Promotions - AeroGrip Yoga Mats | 8,418.50 | 2,722.91 | 26,756.07 | 203.65 | 3,592.42 | 3,351.62 | 9,754.25
Promotions - SolarMate Panels | 55,394.19 | 70,551.17 | 135,155.19 | 2,290.42 | 71,903.89 | 59,587.38 | 243,519.86
Promotions - EcoFlow Water Bottles | 62,965.10 | 239,907.67 | 182,749.23 | 12,975.87 | 74,690.81 | 52,078.24 | 77,198.61
Promotions - GlowCharge Lamps | 11,136.93 | 16,264.30 | 26,586.91 | 7,884.24 | 10,223.55 | 8,370.97 | 6,262.86
Promotions - ThermaTech Jackets | 24,232.39 | 35,574.30 | 11,269.12 | 13,621.76 | 87,408.54 | 23,378.01 | 90,975.09
Sales Rebates - AeroGrip Yoga Mats | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 2,637.45 | 0.00
Sales Rebates - SolarMate Panels | 0.00 | 0.00 | 34,291.00 | 0.00 | 0.00 | 50,452.04 | 0.00
Sales Rebates - EcoFlow Water Bottles | 121,953.83 | 29,368.07 | 110,797.36 | 788.44 | 8,409.07 | 32,164.38 | 25,804.68
Sales Rebates - GlowCharge Lamps | 8,465.98 | 27,327.93 | 20,195.10 | 788.44 | 7,420.62 | 14,629.32 | 0.00
Sales Rebates - ThermaTech Jackets | 14,882.03 | 29,368.08 | 121,670.93 | 788.44 | 7,790.65 | 21,757.12 | 25,804.70
Warehousing Charges | 158,174.87 | 242,732.94 | 239,272.72 | 248,673.67 | 239,396.11 | 169,285.81 | 252,040.02
Total Cost of Sales | 1,681,509.50 | 2,050,990.70 | 1,839,408.13 | 1,199,231.91 | 1,342,109.38 | 1,882,922.04 | 1,506,198.64

Gross Profit | 2,812,861.67 | 1,225,773.93 | 1,475,709.88 | 2,021,875.43 | 1,442,591.53 | 2,753,359.72 | 2,623,357.49

### Excel Agent: Capabilities and Preview

The Excel agent converts sheets into a normalized 2D format and answers questions over both original and normalized spreadsheets. The early-preview demo opens on a file-upload screen.

[8:54](https://www.youtube.com/watch?v=jVGCulhBRZI&t=534s) [Slide lists spreadsheet normalization and agentic question answering, with a preview QR code and an upload screen.](https://assets.ai.engineer/sites/aiecode2025/sha256/77c1c95635ccb87c53180b6c5e4b8424bd010dc0d7b7556f6a26152e4fa3e7a7.webp)

Special Release: Excel!

We built an Excel agent capable of:
1. Data Transformation: Transforming each sheet into a normalized 2D format.
2. Agentic QA: Answer questions over cells in any of the sheets.

This is available in early preview.

Interface text:
Back to Home
Financial Data Processor
Transform raw Excel data from your ERP systems into structured formats. Upload your file to begin analysis.
Upload File
Import your Excel document
0 steps completed
0/3
1 Upload file
2 Process data
3 Download results
Drop Excel file here
or
Browse Files
Data Preview
View and analyze processed data
No file uploaded
Upload an Excel file to get started

### Excel Demo: Source Spreadsheet

The demo shows a source profit-and-loss spreadsheet. The Excel agent transforms unnormalized sheets into a 2D format and supports questions over both the original and normalized versions; it is available in early preview.

[9:02](https://www.youtube.com/watch?v=jVGCulhBRZI&t=542s) [The Excel agent slide accompanies a demo showing the source profit-and-loss spreadsheet.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e33aee5976aa7ed9edca3e45616119b42ec9003338b83793d7582974bd7aa9a.webp)

Special Release: Excel!

We built an Excel agent capable of:
1. Data Transformation: Transforming each sheet into a normalized 2D format.
2. Agentic QA: Answer questions over cells in any of the sheets.

This is available in early preview.

Spreadsheet:
Profit and Loss
Marketing and Advertising
Other Operating Expenses
Operating Profit
Non-Operating Income
Non-Operating Expenses
[Spreadsheet values and other rows illegible]

Interface text:
Upload File
0 steps completed
Upload file
Process data
Download results
Sheet Name
Profit and Loss
Background
Duplicate Sheet
Delete Sheet

### Excel Demo: Normalized Output

The demo transforms a synthetic spreadsheet into a 2D table and offers a CSV download. The early-preview Excel agent also supports questions over spreadsheet cells.

[9:09](https://www.youtube.com/watch?v=jVGCulhBRZI&t=549s) [The Excel agent slide accompanies a demo showing processed rows and a CSV download option.](https://assets.ai.engineer/sites/aiecode2025/sha256/56639e7cbfaa9f6e93ef059e82a4aeeaf9acaeb2ab1e0067ff86e0b60e3a3b2c.webp)

Special Release: Excel!

We built an Excel agent capable of:
1. Data Transformation: Transforming each sheet into a normalized 2D format.
2. Agentic QA: Answer questions over cells in any of the sheets.

This is available in early preview.

Interface text:
2 steps completed
2/3
Upload file
Process data
Download results
[illegible]
File Settings
Configure processing options
Select Sheet
Profit and Loss
Generate Flat File
Delete File
Export Data
Download processed results
Download CSV
Compare
Flat Data Only
flat_file_output.csv

155 Jul 2022 EcoFlow Water Bottles Cost of Sales
156 Total EcoFlow Water Bottles Cost of Sales
157 Jun 2023 ThermaTech Jackets Cost of Sales
158 May 2023 ThermaTech Jackets Cost of Sales
159 Apr 2023 ThermaTech Jackets Cost of Sales
160 Mar 2023 ThermaTech Jackets Cost of Sales
161 Feb 2023 ThermaTech Jackets Cost of Sales
162 Jan 2023 ThermaTech Jackets Cost of Sales
163 Dec 2022 ThermaTech Jackets Cost of Sales
164 Nov 2022 ThermaTech Jackets Cost of Sales
165 Oct 2022 ThermaTech Jackets Cost of Sales
166 Sep 2022 ThermaTech Jackets Cost of Sales

### Excel QA Benchmark

On a private dataset of synthetic Excel sheets, LlamaIndex’s agents scored around 95%, above the ~90% human baseline and both code interpreter comparisons.

[9:48](https://www.youtube.com/watch?v=jVGCulhBRZI&t=588s) [Bar chart compares Excel agent and code interpreter accuracy on complex financial sheets, with a human baseline near 90%.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c9d997f66bbd5fd3e308dc10cc96ef8a3d52848a8d0b2bd75eafa16012d179e.webp)

LlamaIndex QA benchmark - complex complex financial sheets

Excel Agentic Understanding Comparison with 0.99 Confidence Intervals

Y-axis: Mean Accuracy
Y-axis ticks: 0.2, 0.4, 0.6, 0.8, 1.0
Human Baseline ~90%

[illegible]dex Excel Agent - GPT-4.1: 94.4%, 96.1%
LlamaIndex Excel Agent - GPT-4o: 92.6%, 95.1%
OpenAI Code Interpreter GPT-4.1: 72.2%, 75.3%
OpenAI Code Interpreter GPT-4o: 63.7%, 66.8%

### Special Release: Excel!

The Excel pipeline uses reinforcement learning to adapt to a sheet’s format and build a semantic map. Specialized agents then use tools over that map for arithmetic and data aggregation.

[10:15](https://www.youtube.com/watch?v=jVGCulhBRZI&t=615s) [Excel pipeline uses reinforcement learning to understand sheet structure and specialized agents and tools for calculations.](https://assets.ai.engineer/sites/aiecode2025/sha256/f61096d63541725c957cbedac1e19e27d29b2540e06a51fc372cea215110a756.webp)

Special Release: Excel!

Our pipeline combines RL-based data modeling with specialized agents tools:

1. RL-based structure understanding → Learn a semantic map of the sheet, use good-ol reinforcement learning to adapt to a given format.
2. Uses specialized sub-agents + tools → Agents that can reason using tailored tools for arithmetic and data aggregation.

Messy Excel sheets
[Embedded spreadsheet text illegible]
Parsing Algorithms
Semantic Representation
Serves as a 'map' to understand the data
Specialized Agents
Tailored tools for exact calculations

Generic Parser + Data (RL Environment) + Concrete Decision Logic (RL Policy Learning) → Structure Parsing Program (RL Agent)

### Agent Design Patterns

Agent architectures range from explicitly defined control flows to unconstrained loops that receive tools and run.

[11:17](https://www.youtube.com/watch?v=jVGCulhBRZI&t=677s) [Title slide for Agent Design Patterns: From Assistant to Automation UX, with an agent and interface illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/0529e5fd23bb424e412df3bb5503f8b4e70c6fc1b9838388ddd5e44bad8f0e11.webp)

Agent Design Patterns
From Assistant to Automation UX

AIE
Microsoft
smol ai
World’s Fair

### Assistant UX

A human guides the agent through steps while the agent cycles through tools, producing synthesized information or a knowledge artifact.

[11:46](https://www.youtube.com/watch?v=jVGCulhBRZI&t=706s) [Assistant UX uses chat-oriented interaction, natural language or files, and human guidance to produce information or a knowledge artifact.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0063f897742eb7818380a2c6a2ade244329b55358ded5bd404393ef44e55efe.webp)

Assistant UX

Goal
Surfacing information and/or producing a single unit of knowledge work.

Properties
• UX: Chat-oriented
• Input: Natural language or file
• Architecture: Unconstrained, more tools, high HITL
• Output: Output is synthesized information and/or a knowledge artifact

Step 1
Human
Agents
Tools
…
Step N
Human
Agents
Tools
Output

22

### Assistant UX Use Cases

Assistant use cases include financial due diligence, customer support, and workplace search. The demo shows a user entering a request for a report about Tesla finance.

[12:32](https://www.youtube.com/watch?v=jVGCulhBRZI&t=752s) [Assistant UX use cases include financial due diligence, customer support, and workplace search, alongside a report-generation demonstration.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f90c2f229ffb438146ea47669a159d32136ec5dcdb7c4a0501cbb5d15b731b8.webp)

Assistant UX

Use Cases
• Financial due diligence
• Customer support
• Workplace search

Create a report about the Tesla finance

Interface text:
0:02 / 0:14
CC

### Automation UX: Agentic Document Workflows

Document batches go through extraction and human review to produce structured results. Agents interact with tools during processing, with less human involvement at each step.

[12:40](https://www.youtube.com/watch?v=jVGCulhBRZI&t=760s) [Properties of batch document automation alongside a flow from documents through extraction, agents, tools, human review, and output.](https://assets.ai.engineer/sites/aiecode2025/sha256/18c6ba1f587324997f2bc4bc48946b09e73b2bf2c8d500c77d83fcc07c01a48b.webp)

Automation UX

Agentic Document Workflows

Goal: Process routine tasks and processes to increase efficiency

Properties:
• UX: Batch processing of jobs, review results, wait until done
• Input: Batch of inputs (e.g. files)
• Architecture: Constrained, multi-step, lower HITL
• Output: Structured results, integration with API

Document
Extraction
Agents
Tools
Human Review
Output

24

## Full transcript

[0:00](https://www.youtube.com/watch?v=jVGCulhBRZI&t=0s) [upbeat music] Okay.

[0:15](https://www.youtube.com/watch?v=jVGCulhBRZI&t=15s) Hey, everyone. Uh, I'm Jerry, co-founder, CEO of LlamaIndex. It's great to be here. Um, and today my topic, uh, the talk ti-title is Building AI Agents that Actually Automate Knowledge Work.

[0:26](https://www.youtube.com/watch?v=jVGCulhBRZI&t=26s) So, basically, a big promise of AI agents these days is making knowledge workers more efficient. Um, I'm, I'm sure you've heard the, like, high-level business speak of this, um, and I copy and pasted a bunch of B2B SaaS vendors on the, on the right in terms of screenshots.

[0:41](https://www.youtube.com/watch?v=jVGCulhBRZI&t=41s) You know, increase operational efficiency, better decision-making through more data. But what does this actually mean, right? Does knowledge work automation actually just mean building RAG chatbots? And if not, what is the stack and what are the use cases that AI agents can actually do, um, in terms of automating knowledge work?

[0:59](https://www.youtube.com/watch?v=jVGCulhBRZI&t=59s) So, for us, a lot of our use cases and a lot of our core focus areas is basically automating knowledge work over unstructured data. Um, 90% of enterprise data lives within the form of documents, whether it is PDFs, PowerPoints, um, Word, and, uh, you know, as you'll soon see, Excel.

[1:17](https://www.youtube.com/watch?v=jVGCulhBRZI&t=77s) But humans, uh, have historically needed to basically read and write these types of docs, right? You have, you know, investment banker or someone, you know, kind of on the customer support side reviewing a lot of just unstructured data and using that documentation to basically make decisions and take actions.

[1:35](https://www.youtube.com/watch?v=jVGCulhBRZI&t=95s) For the first time, AI agents can actually reason and act over massive amounts of unstructured context tokens and, you know, do analysis, do research, syn-synthesize these insights, and actually take actions end to end.

[1:50](https://www.youtube.com/watch?v=jVGCulhBRZI&t=110s) And so for us, when we think about the use cases and the types of agents for automating knowledge work, they really fall into two main categories. There's what we call assistive agents, um, so those that are kind of more like a standard chat interface.

[2:03](https://www.youtube.com/watch?v=jVGCulhBRZI&t=123s) They help humans get more information faster. And then there's automation-type agents, agents that automate routine tasks, uh, can run in the background, maybe require a little bit less human-in-the-loop, and can take actions, um, that automate the routine operational stuff.

[2:20](https://www.youtube.com/watch?v=jVGCulhBRZI&t=140s) When we think about the stack that's required to actually build either the assistive or automation-type agents, there's two main components. There's really, really nice tools, um, and then there's a really nice agent architecture.

[2:32](https://www.youtube.com/watch?v=jVGCulhBRZI&t=152s) Uh, with MCP A2A these days, a lot of people are thinking about, "How do I build really nice tools that allow agents to interface with the external world to basically surface relevant context and let the agent take, uh, external actions?"

[2:45](https://www.youtube.com/watch?v=jVGCulhBRZI&t=165s) And a lot of the agent architecture, you know, there's very general reasoning loops as well, uh, as well as more constrained loops. It's basically, how do I encode the business logic, um, through an agentic workflow to help achieve the task?

[2:59](https://www.youtube.com/watch?v=jVGCulhBRZI&t=179s) So, for the purposes of this talk, we'll talk about three main things. Um, a lot of stuff to cover, so I'll probably, uh, pick up my clock speed a little bit.

[3:06](https://www.youtube.com/watch?v=jVGCulhBRZI&t=186s) But basically, there's building a document toolbox, um, which is how do I build really nice tools that allow, uh, you know, AI agents to interact with massive amounts of unstructured documents?

[3:15](https://www.youtube.com/watch?v=jVGCulhBRZI&t=195s) Um, two is agent design patterns. Um, so thinking about just at a high level the two categories of agents from assistance to automation. And three is bringing it together in terms of document agent use cases.

[3:27](https://www.youtube.com/watch?v=jVGCulhBRZI&t=207s) So first step is on building a document toolbox.

[3:31](https://www.youtube.com/watch?v=jVGCulhBRZI&t=211s) Basically, if you think about agents interacting with tools, um, and as LLMs get better, you're gonna have these very general front-end interfaces like Claude or ChatGPT. Um, agents need access to the right tools to basically interface with the external world.

[3:46](https://www.youtube.com/watch?v=jVGCulhBRZI&t=226s) And for the purposes of, you know, massive amounts of unstructured enterprise data, um, they basically need, uh, the right toolbox to interact with this data. It's basically a generalization beyond, uh, naive RAG, right?

[3:57](https://www.youtube.com/watch?v=jVGCulhBRZI&t=237s) RAG is just, uh, retrieval. I know this is a RAG workshop, um, but NAG-- RAG is just like retrieval and then one-shot synthesis. Um, a lot of what agents can do over your documents includes retrieval, but also includes other operations like file-based search, uh, manipulation, and more.

[4:14](https://www.youtube.com/watch?v=jVGCulhBRZI&t=254s) And one of the points I'm trying to make is that to basically create these tool interfaces in the first place, you need a really nice pre-processing layer. Um, so you need, you know, actual data connectors to your data sources, um, that basically sync data from your data source into a format that your agents can access.

[4:30](https://www.youtube.com/watch?v=jVGCulhBRZI&t=270s) Um, you know, could be SharePoint, Google Drive, S3, Confluence. It needs to sync permissions too, and the right metadata. You need the right document parsing and extraction piece. More on this in just a bit.

[4:41](https://www.youtube.com/watch?v=jVGCulhBRZI&t=281s) But you basically need actual, actually good understanding over your documents, over tables, charts, and more. And of course, you know, if you have a large collection of docs, you need it to index it in some way.

[4:52](https://www.youtube.com/watch?v=jVGCulhBRZI&t=292s) Um, it could be vector indexing into, you know, vector search. It could also be, uh, indexing into a SQL table. Uh, it could be GraphDBs. Uh, it could be anything.

[5:01](https://www.youtube.com/watch?v=jVGCulhBRZI&t=301s) So basically, uh, to ensure the data is high quality, you need this layer to actually process and stru-structure your documents and expose the right tool interfaces. In terms of the right tool interfaces, this is what I want to kind of, uh, define a term.

[5:14](https://www.youtube.com/watch?v=jVGCulhBRZI&t=314s) It's basically called like a Document MCP server. Um, again, it's like a generalization of this idea of RAG, right? If RAG is just one-shot vector retrieval, you kind of need like a set of tools, um, to basically equip an AI agent with, uh, to basically, uh, understand and manipulate different types of documents.

[5:33](https://www.youtube.com/watch?v=jVGCulhBRZI&t=333s) It could be, you know, doing semantic search to fuzzy find the relevant source of data. It could be file lookup to basically look up the right file metadata. Um, it could be manipulation to actually do operations on top of the files, and it could be structured querying, right?

[5:47](https://www.youtube.com/watch?v=jVGCulhBRZI&t=347s) Querying a, a more structured database to get aggregate insights over the types of data, um, that, that you've extracted out.

[5:56](https://www.youtube.com/watch?v=jVGCulhBRZI&t=356s) One, you know, top consideration, uh, when actually building this type of toolbox is, uh, complex documents. Uh, for those of you who follow our socials, we talk a lot about this type of issue where a lot of human knowledge is in the form of like really complicated PDFs, uh, and other formats too.

[6:12](https://www.youtube.com/watch?v=jVGCulhBRZI&t=372s) Embedded tables, charts, images, irregular layouts, headers, footers. This is typically stuff that's designed for human consumption and not machine consumption. And so, you know, if the documents are not processed correctly, no matter how good your LLM is, um, it will fail.

[6:26](https://www.youtube.com/watch?v=jVGCulhBRZI&t=386s) So we were probably one of the first people to actually realize that LLMs and LVMs could be used for document understanding. Um, if, uh, in contrast to more traditional techniques where you use kind of like hand-tuned and task-specific ML models to achieve, uh, kind of like document parsing over a specific class of documents, LLMs actually have a

[6:46](https://www.youtube.com/watch?v=jVGCulhBRZI&t=406s) much general layer of accuracy, um, that you can use to your advantage in just like understanding and inhaling any type of document with compli- uh, any type of complexity.

[6:56](https://www.youtube.com/watch?v=jVGCulhBRZI&t=416s) Um, ob-obviously, the baseline these days is you can just screenshot a PDF, feed it into ChatGPT or Claude. Um, it doesn't actually give you amazing accuracy, but it's a good start.

[7:06](https://www.youtube.com/watch?v=jVGCulhBRZI&t=426s) And so one of the kind of secret sauce like, uh, magic tricks we found was figuring out how to interleave LLMs and LVMs with more traditional parsing techniques and adding kind of test time tokens in terms of agentic validation and reasoning to really get a higher level of accuracy.

[7:23](https://www.youtube.com/watch?v=jVGCulhBRZI&t=443s) Um, and so, you know, we have a cloud service that does document parsing and is a core step of this document toolbox. Uh, we basically benchmarked, uh, our modes where we adapt, uh, you know, Sonnet 3.5, 4.0, uh, Gemini 2.5 Pro, 4.1 from OpenAI, and it basically outperforms all existing parsing benchmarks, um, and, and tools out there

[7:43](https://www.youtube.com/watch?v=jVGCulhBRZI&t=463s) in terms of open source to proprietary. Um, yeah.

[7:48](https://www.youtube.com/watch?v=jVGCulhBRZI&t=468s) So some of you might know us as a RAG framework. That's basically how we started. Um, you know, for those of you who don't know, we have this, uh, managed platform that is basically this gen AI native document toolbox.

[8:00](https://www.youtube.com/watch?v=jVGCulhBRZI&t=480s) Um, contains a lot of operations that you need to do on top of your docs. It could be document parsing, document extraction, uh, uses some of those, you know, kind of capabilities I just mentioned, and allows you to parse, extract, index data for all the set of tools I just mentioned.

[8:15](https://www.youtube.com/watch?v=jVGCulhBRZI&t=495s) One of the special releases that I actually wanna highlight today, um, and we just announced this in a blog post a few hours ago, is Excel capabilities to help complement this document toolbox.

[8:25](https://www.youtube.com/watch?v=jVGCulhBRZI&t=505s) A lot of knowledge work happens in Microsoft Excel and also Google Sheets and, you know, Numbers, and basically it's, uh, spreadsheets, right? But it's been unsolved by LLMs. Um, if you look at the document to the right, uh, neither RAG nor text to CSV techniques will actually work over this because it's not really a structured 2D

[8:44](https://www.youtube.com/watch?v=jVGCulhBRZI&t=524s) table. There's a bunch of gaps in the rows and gaps in the columns.

[8:49](https://www.youtube.com/watch?v=jVGCulhBRZI&t=529s) So we basically built an Excel agent, um, that's capable of taking unnormalized Excel spreadsheets and transforming them, um, into a normalized 2D format, and also allows you to do agentic QA, um, over, uh, both the unnormalized and normalized versions of the Excel spreadsheet.

[9:08](https://www.youtube.com/watch?v=jVGCulhBRZI&t=548s) Um, it's a pretty cool capability. I'll describe, uh, how it kinda works in just a bit. Um, but it's gonna complement our toolbox, right, in terms of, uh, more traditional document parsing, extraction, indexing, and it's available in, uh, early preview.

[9:23](https://www.youtube.com/watch?v=jVGCulhBRZI&t=563s) So if you just, uh, take a look at the video, it's also on our blog post. We basically uploaded that example synthetic dataset, transformed it into a 2D table, and you can also ask questions over it to basically get insights.

[9:35](https://www.youtube.com/watch?v=jVGCulhBRZI&t=575s) And it's really doing the heavy lifting of deeply understanding the semantic structure of the Excel spreadsheet, um, and then using that and plugging that in as specialized tools to an AI agent.

[9:47](https://www.youtube.com/watch?v=jVGCulhBRZI&t=587s) Um, the best baseline is not really RAG or text to CSV. Um, those both suck. Um, it's really just an LLM being able to write code. Um, so, uh, LLM with a code interpreter tool is a reasonable baseline, gets you to seventy, seventy-five percent accuracy.

[10:03](https://www.youtube.com/watch?v=jVGCulhBRZI&t=603s) Um, over like a private dataset of synthetic Excel sheets, uh, we basically were able to get this up to ninety-five percent. Um, it actually surpasses human baselines of ninety percent of a human trying to go and do the data transformation by hand.

[10:17](https://www.youtube.com/watch?v=jVGCulhBRZI&t=617s) Um, a brief note on how it works. Uh, it's a little bit technical, um, but, you know, more details are in the blog post. Um, first, we do some sort of structure understanding of the Excel spreadsheet.

[10:28](https://www.youtube.com/watch?v=jVGCulhBRZI&t=628s) So we do a little bit of RL, reinforcement learning. Um, you know, uh, we actually kind of adapt dynamically to the specific format of the document, um, and learn a semantic map of the sheet.

[10:41](https://www.youtube.com/watch?v=jVGCulhBRZI&t=641s) By learning a semantic map, uh, we can then translate this into, um, kind of a set of specialized tools that you provide to an agent. And so from an abstract perspective, you can kind of think about it as an agent could just write code from scratch.

[10:55](https://www.youtube.com/watch?v=jVGCulhBRZI&t=655s) Um, as LLMs get better, that will certainly become, um, an eas- like a, a kind of higher performing baseline. But in the meantime, we're helping it out by really providing, uh, a set of specialized tools over this semantic map, so it can reason over an Excel spreadsheet.

[11:10](https://www.youtube.com/watch?v=jVGCulhBRZI&t=670s) Great. Um, the next piece here is-- So we talked about a document toolbox. Uh, we talked about a lot of operations basically make this, uh, document toolbox really good and comprehensive.

[11:20](https://www.youtube.com/watch?v=jVGCulhBRZI&t=680s) So now that you plugged it into an agent, what are the different agent architectures and what are the use cases that are implied by them? Um, as many of you probably know from building agents yourselves, agent orchestration ranges from more constrained architectures to unconstrained architectures.

[11:36](https://www.youtube.com/watch?v=jVGCulhBRZI&t=696s) Um, constrained is basically you kind of more explicitly define the control flow. Unconstrained is like a ReAct loop, function calling, CodeAct, uh, whatever. You basically give it a set of tools and let it run.

[11:44](https://www.youtube.com/watch?v=jVGCulhBRZI&t=704s) Um, deep research is kind of the same thing.

[11:47](https://www.youtube.com/watch?v=jVGCulhBRZI&t=707s) Um, for us, we basically noticed there's two main categories of UXs. Um, there's more assistant-based UXs that can basically surface information and, um, help a human surface information or produce some unit of knowledge work through usually a chat-based interface.

[12:03](https://www.youtube.com/watch?v=jVGCulhBRZI&t=723s) It's usually chat-oriented, the input's natural language. Um, the architecture is a little bit more unconstrained. You know, it's basically a ReAct loop over some set of tools. Um, and it's inherently both unconstrained, but also with a higher degree of human-in-the-loop.

[12:18](https://www.youtube.com/watch?v=jVGCulhBRZI&t=738s) So the goal is, or the expectation is that the human is supposed to kind of guide and coax the agent, uh, along the steps of the process to basically achieve the task at hand.

[12:30](https://www.youtube.com/watch?v=jVGCulhBRZI&t=750s) There's a-- I mean, there's-- I'm sure many of you have built these types of use cases, and so this is just a very small subset. Um, but it's basically just, you know, your, uh, generalization of a, of a RAG chatbot.

[12:41](https://www.youtube.com/watch?v=jVGCulhBRZI&t=761s) There's a second category of use cases that I think is interesting, and I think a lot of folks are actually starting to build more into this space, which is, um, this automation interface.

[12:51](https://www.youtube.com/watch?v=jVGCulhBRZI&t=771s) So being able to actually, instead of, uh, providing some assistant or copilot to help a human get more information, um, processing routine tasks in a multi-step end-to-end manner. And usually, the architecture is a little bit different.

[13:05](https://www.youtube.com/watch?v=jVGCulhBRZI&t=785s) Um, it takes in some batch of inputs. Uh, it can run in the background, or it could be triggered ad hoc by the human. Um, the architecture is a little bit more constrained, which kinda makes sense, right?

[13:15](https://www.youtube.com/watch?v=jVGCulhBRZI&t=795s) If you want this thing to run more end-to-end, um, you need it to not just go off the rails. Um, and there's usually a little bit less human-in-the-loop at every step of the process, and usually some sort of like batch review in the end.

[13:27](https://www.youtube.com/watch?v=jVGCulhBRZI&t=807s) And the output is like structured results, integration with APIs, uh, decision-making. After approval, it'll just go route to the downstream systems.

[13:36](https://www.youtube.com/watch?v=jVGCulhBRZI&t=816s) Some of the use cases here include, you know, financial data normalization, datasheet extraction, invoice reconciliation, contract review, and more.

[13:47](https://www.youtube.com/watch?v=jVGCulhBRZI&t=827s) Um, I'll skip this video, but you know, there's some fun example of some community-based open source repos we built in this area, like the invoice reconciler by Laurie Voss.

[14:01](https://www.youtube.com/watch?v=jVGCulhBRZI&t=841s) A kind of general idea that we've em-em-emer-- that has emerged, and we've noticed as a pattern is, you know, oftentimes the automation agents can serve as a back end because it runs in the background, you know, can do the data ETL transformation.

[14:15](https://www.youtube.com/watch?v=jVGCulhBRZI&t=855s) There's still human-in-the-loop, but it's kind of the doing the thing where it needs to process and structure a lot of data, um, and do decisions in the background. And then assistant agents are kind of more front-end facing, right?

[14:27](https://www.youtube.com/watch?v=jVGCulhBRZI&t=867s) And so automation agents can structure, process your data, and provide the right tool interfaces, um, for assistant agents. Not every tool depends on agentic reasoning, but for a lot of these use cases, like for a very generalized data pipeline, um, where you're processing a lot of unstructured context, you might have automation agents go in and process your

[14:47](https://www.youtube.com/watch?v=jVGCulhBRZI&t=887s) data, provide the right tools for some sort of more, uh, research user-facing interface.

[14:56](https://www.youtube.com/watch?v=jVGCulhBRZI&t=896s) So we talked about building a document toolbox. We talked about, you know, the, the, the different categories of agentic architectures. And putting it together, um, here are some real-world use cases of document agents, and these are basically examples of agents that actually help automate different types of knowledge work.

[15:13](https://www.youtube.com/watch?v=jVGCulhBRZI&t=913s) So one of our favorite examples is a combination of both automation and assistant UXs for financial due diligence. Um, Carlyle is one of our, uh, favorite customers and, and partners.

[15:24](https://www.youtube.com/watch?v=jVGCulhBRZI&t=924s) Um, you know, they basically used, uh, some of the core capabilities that we have to build an end-to-end leveraged buyout agent. Um, you know, uh, it requires an automation interface to inhale massive amounts of unstructured public and private financial data, um, Excel sheets, PDFs, PowerPoints, go through some bespoke extraction algorithms with human-in-the-loop review.

[15:47](https://www.youtube.com/watch?v=jVGCulhBRZI&t=947s) And then once that data is actually structured in the right format, providing a copilot interface, uh, for the analyst teams to actually both get insights and generate reports over that data.

[15:59](https://www.youtube.com/watch?v=jVGCulhBRZI&t=959s) If you look at any enterprise search use case, that typically falls within the assistant UX. Um, Semux is one of our favorite, uh, customers in this space, where, you know, just being able to define a lot of different collections to different sources of data and providing more task-specific, specialized agentic RAG chatbots over your data, right?

[16:18](https://www.youtube.com/watch?v=jVGCulhBRZI&t=978s) Um, you know, it's basically a RAG, but you add like an agentic reasoning layer on top so that you can basically break down user queries, do research, and answer the question at hand.

[16:30](https://www.youtube.com/watch?v=jVGCulhBRZI&t=990s) And on the pure automation UX side, uh, we notice a lot of kind of use cases popping up around automa-automation and efficiency. And so one example is actually technical datasheet ingestion.

[16:42](https://www.youtube.com/watch?v=jVGCulhBRZI&t=1002s) Um, you know, we're working with a global electronics company. They have a lot of data sheets, uh, that need to be automatically processed and reviewed, and historically, it's taken a lot of human effort to actually do this.

[16:54](https://www.youtube.com/watch?v=jVGCulhBRZI&t=1014s) Um, so by creating the right end-to-end automation agent, you can basically encode the business-specific logic for parsing these types of documents, extracting out the right pieces of information, matching it against specific rules, and outputting the structured data into SQL.

[17:11](https://www.youtube.com/watch?v=jVGCulhBRZI&t=1031s) There's human-in-the-loop review, um, but if we're actually able to do this end-to-end, it transforms weeks of just like, you know, technical writer work, um, into an automated extraction interface.

[17:24](https://www.youtube.com/watch?v=jVGCulhBRZI&t=1044s) So that's basically it. Um, you know, for those of you who are less familiar, LlamaIndex is, uh, the most accurate, customizable platform for automating your document workflows with agentic AI.

[17:34](https://www.youtube.com/watch?v=jVGCulhBRZI&t=1054s) Um, our mission statement's evolved a little bit since the past few years. We're, uh, we're a very broad horizontal, uh, framework, oftentimes focused on RAG. Um, but if you're interested in some of the capabilities, uh, come talk to us, and then please come check us out at booth G11.

[17:48](https://www.youtube.com/watch?v=jVGCulhBRZI&t=1068s) Thank you. [audience applauding] [upbeat music]
