From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

A legal-ops chatbot works well on around 40 hand-picked files. The pilot parses documents, performs optical character recognition, splits content into chunks, creates embeddings, and loads a vector database and a knowledge graph. Then the team expands to more than 80,000 files. Three questions stall the project: which files belong, where sensitive information lives, and which documents deserve trust. The retrieval pipeline already works; selecting its inputs has become the problem.

Jeff Koss, a staff customer engineer at Deasy Labs, develops this problem through the North River Manufacturing demo scenario. Leo Platzer, Deasy’s former CTO, later explains the retrieval consequences. Deasy supplies the unstructured-data capabilities discussed here following its acquisition by Collibra.

North River has 40,000 employees across 20 countries and millions of accumulated SharePoint documents. Its legal and procurement teams need current answers about supplier termination clauses, agreements expiring this or next quarter, and partnership exclusivity terms. Those questions require more than finding text that resembles a query: the system must find the right agreement and the version that still applies.

The scenario gives several teams different requirements:

• Data and AI: answer accuracy.
• AI engineering: a discoverable set of relevant files without sensitive content.
• Knowledge management: prevention of sensitive-data leaks and redaction where required.
• Legal operations: dependable answers for the business.
• Enterprise transformation: a repeatable process that can support another 15 use cases.

A manually curated pilot satisfies these requirements once. Production needs a way to keep satisfying them as the source collection changes.

Koss brings a structured-data background from Informatica and Databricks, where quality includes dimensions such as completeness, conformity and accuracy. Documents add a practical set of concerns: repeated information, conflicting information and freshness. An expired contract can be perfectly readable and easy to retrieve while still being the wrong basis for an answer.

Metadata connects curation to the chatbot. Tags describe what a document contains, allowing the downstream system to select relevant material and use those descriptions alongside its content. The intended input is therefore a collection of current, relevant, high-quality files enriched with metadata, with sensitive material excluded.

Manual maintenance becomes its own bottleneck. Koss describes a telecommunications company where six to seven subject-matter experts cannot keep up with curating 20,000 HTML files. The consequences extend beyond slow preparation: incorrect answers, sensitive information appearing in chatbots, and a collection that becomes brittle as it changes.

The demonstration starts with 339 ingested files from a mixed SharePoint collection: call transcripts, PDFs, spreadsheets and Word documents. Deasy exposes both file-level information and page- or chunk-level content. The first task is to classify this mixture so the legal material becomes selectable.

A taxonomy is a tagging tree. Its tags can come from several places:

• Manual definitions: create a tag and supply a prompt describing how the model should classify a file.
• Reusable definitions: select tags from a library or import an existing taxonomy and valid values through CSV.
• AI suggestions: ask the system to propose tags and classification values when the categories are not already known.

Classification values are optional. Supplying them lets the team state its existing categories; generating them helps explore a collection whose structure is still unfamiliar.

The tree can make later classification conditional on an earlier result. First identify legal and contract files; then classify contract type within that branch, such as supplier contracts or NDAs. Other tags can apply unconditionally. This expresses a useful dependency: a question about contract type belongs to the contract branch rather than every engineering document or call transcript.

After selecting files and tags and generating metadata, the legal-and-contract category contains 68 files. Clicking that category filters the collection. Applying two filters brings the 339-file collection down to 20 files; the recording does not identify both filter criteria. The observable change is concrete: generated metadata turns an undifferentiated file collection into a smaller set the engineer can select.

Why did a file enter that category? The interface exposes the passage that supported its classification. Thumbs-up and thumbs-down feedback can supply examples for few-shot learning. This puts a review step beside the generated label: a person can inspect the evidence and provide an example rather than accepting the category without explanation.

Sensitivity detection supports both AI-based scanning and patterns interpreted with surrounding context. Koss’s example is a request to detect Finnish national IDs. The system generated a regular expression and added Finnish words that he believed were associated with the identifier. The intended mechanism is to combine a matching character pattern with contextual clues, reducing accidental matches; this anecdote does not establish the resulting false-positive rate.

Quality checks apply several distinct selection rules:

• Duplicates and conflicts: group related material and use AI to suggest which items to retain.
• Freshness: restrict files to a specified date range or number of days.
• Required metadata: identify the files that satisfy the information requirements of the intended data product.

The retained collection becomes a data slice: files selected by explicit criteria for a particular use.

A second demonstration starts with a separate 14-file legal slice. A use-case description and the files’ contents guide suggestions for either a flat taxonomy or a tree with multiple levels. A person chooses which proposed tags belong, and the system also generates classification values. This makes the taxonomy specific to both the business question and the actual collection.

Generated metadata includes sensitivity findings, file- and chunk-level evidence, and model-returned confidence scores. Scores such as 95% or 100% describe the model’s reported confidence; the recording does not establish that they are calibrated probabilities of correctness. Their practical companion is the evidence: a reviewer can see the content behind the classification.

The data slice is maintained on a schedule. Workflows pick up files added to or deleted from SharePoint and update the selected collection. The SDK then makes that data product available to the chatbot, including metadata used when loading the vector database. Freshness therefore depends on recurring source updates reaching the downstream retrieval system; a scheduled refresh leaves an interval between a source change and its propagation.

Where does a source change travel? The flow below separates the maintained collection from its downstream use. Updating SharePoint alone does not update chatbot context: the scheduled slice and SDK delivery connect those stages.

For the North River scenario, Koss reports reducing preparation from four months to days, improving chatbot accuracy through file selection and metadata, and reducing compliance risk by filtering sensitive material out of the slice. These are presented scenario outcomes rather than a documented customer benchmark. The mechanism is nevertheless specific: less manual discovery, additional metadata for retrieval, and exclusion of identified sensitive content.

Platzer turns to the retrieval consequence. A query selects a limited number of results—the top K—from the search collection. Repeated information and conflicting versions can occupy those positions. His Slack example makes the temporal problem clear: someone states one thing today and changes it tomorrow, while both messages remain searchable. Similarity to the question does not by itself establish which statement still applies.

Duplicate handling must distinguish useful additional context from simple redundancy. Deasy retains material that contributes context and removes material considered redundant. That distinction matters when two versions look alike but describe a real change: deleting every near-duplicate could discard the information needed to understand the current state.

The share of poor-quality files in a corpus need not equal their share of retrieved context. Platzer gives a 1,000-file example in which 30% are stale or duplicative, yet up to 80% of an agent’s context can contain stale or redundant information. Retrieval selects relevant-looking results rather than sampling files uniformly, so repeated matches can crowd the limited context. The 80% figure is a reported possible outcome, not a fixed conversion from corpus contamination to context contamination.

On the same multi-hop RAG tasks, Platzer reports roughly doubled recall and a 10–15% improvement in accuracy or task completion after data-quality work. These were chatbot retrieval evaluations, not coding-agent evaluations; the recording does not specify the full protocol or whether the completion gain is relative or in percentage points. The results support testing curation as a retrieval intervention, without establishing the same numerical gain for every application.

Relevance criteria depend on the use case: a legal chatbot and another enterprise application need different files. Duplication and freshness remain concerns across those uses, although deciding what counts as useful history or obsolete information still requires care. Cleaning the collection changes what retrieval can supply before any change to the answering model.

The final technical demonstration uses the SDK to fetch metadata for a SharePoint site, describe each folder, and create an index across folders. Claude Code or Codex can consult a Markdown context file to decide where to look before inspecting individual documents. The same metadata used for curation now supplies a navigation layer for an agent.

One folder contains 23 files. Without the generated description, the agent would need to inspect those files to decide whether the folder is relevant. The context repository mirrors the SharePoint structure and describes document types, key topics, folder purpose, when to use it, and example questions. This changes the first step from opening documents to choosing a promising folder using its summary.

How does metadata save repeated discovery? The diagram shows two stages: metadata first becomes a folder index and descriptions; the agent then uses that map to choose where to inspect content. The map helps route exploration rather than supplying every document’s answer. Platzer proposes improved performance and lower token cost from this approach, without reporting a measured coding-agent result.

Koss closes with an offer of one-week proofs of concept built around a business problem and sample files. The proposed outputs are concrete: metadata for the files, a sensitivity report, a data-quality report, and a taxonomy shaped by the use case and the collection. That gives the initial experiment a broader scope than testing answer generation alone: it also examines whether the source documents can become a usable, maintainable input.