Forget RAG Pipelines—Build Production-Ready AI Agents in 15 Minutes

A RAG demo over ten documents is easy to assemble. What happens when it must answer questions across thousands of diverse enterprise documents, extraction becomes unreliable, and users ask questions differently from the developers who built it? Treating RAG as a managed service moves responsibility for maintaining the pipeline to a provider, much as hosted language models and vector databases already do. This is the practical problem Rajiv Shah and Nina Lopatina work through.

Shah traces Contextual AI’s approach to founders Douwe and Aman’s work on RAG at Meta and their subsequent overlap with him at Hugging Face. The basic mechanism remains retrieval-augmented generation: store representations of enterprise information, retrieve relevant material—using cosine similarity in the simplest example—and pass that context to a language model. The difficulty is making each stage work on real documents and real questions.

The platform offers three levels of adoption:

These options matter because an existing application may need a better parser without needing a replacement for its entire RAG system. As the pipeline grows to include lexical retrieval with BM25, embeddings and reranking, maintaining the orchestration becomes part of its latency, cost and engineering burden.

The workshop materials pair a Python notebook with the Contextual interface. Lopatina softens the fifteen-minute promise immediately: explaining the steps will take longer. The dataset contains NVIDIA financial statements and reports about spurious correlations. The first tests quantitative reasoning over tables; the second tests whether the system follows supplied evidence even when it conflicts with conventional expectations. She uses a general-access account and creates a workspace with a unique name, suggesting a personal name to avoid collisions.

Prepare the notebook and credential in this order:

1. Use File → Save a copy in Drive so notebook changes can be saved.
2. Run the package installations and imports.
3. Create an API key in the Contextual interface.
4. Store its value in notebook secrets as contextual_api_key.

That lowercase name belongs to the recorded notebook; the current workshop README uses CONTEXTUAL_API_KEY. Use the name expected by the notebook version you are following. Copying stalls during the demonstration, so Lopatina continues in the original notebook. The same credential serves the agent workflow and the standalone components.

Next, initialize the client and create a named datastore, reusing an existing datastore when the notebook finds one. Step two downloads the financial and correlation reports, then uploads them. Uploaded files first appear as processing; only after processing completes can the three-dot menu’s Inspect action show their extracted contents. While the new upload runs, Lopatina opens the previous day’s completed ingestion of the same files.

Inspection shows both raw text and a rendered preview. For a table, inspect the relationship between the values and the row and column structure—not just whether the numbers appear somewhere in the output. Lopatina reports that the displayed table preserves both. Figures also produce extracted text, summaries and key data points. This is the first useful debugging boundary: before asking whether an answer is correct, check whether the evidence survived ingestion.

The service takes responsibility for extraction, chunking, reranking and scaling, with a business-user interface and REST endpoints. Shah describes SaaS and VPC deployment options. Inside the pipeline, document understanding extracts information from text, tables and images; chunking prepares it for retrieval; BM25 and an embedding model retrieve candidates; an internally trained reranker selects stronger evidence; and a grounded language model generates the answer. Its intended behavior is to prioritize supplied context over its pretrained knowledge.

Shah says the team evaluates both individual components and the full system using academic benchmarks and customer datasets. The resulting service need not appear as a standalone chatbot. He describes Qualcomm’s website using it for answers, sources and feedback; financial applications calling the APIs to turn unstructured information into spreadsheet-ready outputs; and MCP integrations making enterprise knowledge available inside other clients.

Once ingestion finishes, the notebook can inspect document metadata through the API, including name, creation time and processing status. Lopatina creates an agent named demo-AIE with the default system prompt. It immediately appears in the interface, where she asks for NVIDIA’s annual revenue for fiscal 2022 through 2025. The demonstrated response adds the supplied quarterly figures and explicitly identifies quarterly data as its basis. Numbered citations beside the answer open the source images.

The supporting images come from two separate files without a shared naming convention. The retrieval system must connect their contents rather than depend on a carefully coordinated filename scheme. Notebook step four repeats the query through the API, and a separate request retrieves Base64-encoded source imagery. An application can therefore expose both the generated answer and inspectable evidence.

The next question asks when NVIDIA’s data-center revenue overtook gaming revenue. The demonstration identifies Q1 fiscal 2023 and opens the supporting chart. The spoken revenue units should be billions, not millions: NVIDIA’s official Q1 fiscal 2023 release reports $3.75 billion for Data Center and $3.62 billion for Gaming. This confirms their ordering that quarter; the workshop chart supplies the demonstrated crossover context.

The spurious-correlation reports make a different demand on grounding. Asked about Neptune’s distance from the Sun and US burglary rates, the agent reports the high correlation described in the document, then includes the document’s explanation of data dredging and its statistical caveats. A useful answer must preserve both parts: what the report measured and why that observation does not establish a meaningful causal relationship. Citation loading is interrupted by the workshop’s Wi-Fi problems.

A second report pairs Unilever Group’s global revenue with Google searches for “lost my wallet.” Lopatina compares how different input conditions affect the response:

This is Lopatina’s comparison of those examples, with different access to documents. Its practical lesson is that enterprise information may be unfamiliar or counterintuitive: a model should still represent what a source says accurately.

Grounding also needs boundaries. Asked whether Unilever’s revenue therefore comes from lost wallets, the agent rejects the causal inference. Asked for a relationship between Neptune’s distance and Unilever’s revenue—two subjects appearing in separate reports without a supplied relationship—it declines to answer. Finding both subjects in a datastore does not establish a connection between them.

When an answer is missing, first check which datastores are linked to the agent. The admin Edit screen can attach multiple datastores and change the system prompt. It also exposes query expansion, decomposition, retrieval, reranking, filtering, generation and suggested questions. Multi-turn behavior is disabled for the workshop so participants’ previous questions do not change later responses.

The notebook also introduces the individual components, with links to fuller examples and benchmarks. The parse example displays the first page of Attention Is All You Need after processing completes. Lopatina describes the reranker as the first instruction-following reranker and the generation model as the most grounded on FACTS Grounding; those are the product’s benchmark and novelty claims presented at the workshop. The components reuse the existing API credential, and their detailed examples are not walked through live. LMUnit, a natural-language evaluation model, completes the component set.

Shah divides hallucination defenses into three stages: retrieve clean information, use a generator that respects that information, and check the response while giving users precise attribution. These stages address different failures. A generator cannot reliably repair a table whose cells shifted during extraction, and a hallucinated chart caption can become false evidence for every subsequent answer. Simple PDF tools such as pdfplumber may be useful starting points, but complex tables and multimodal documents demand closer inspection.

His document-understanding diagram is a working mental model of an evolving system. The processing sequence is:

1. Attach document metadata. Preserve information such as filename and creation time.
2. Analyze layout. Identify image-only content needing OCR, figures needing captions and tables needing specialized extraction.
3. Recover hierarchy. Retain section headers and subsections alongside the extracted contents.
4. Build a structured representation. Produce Markdown/JSON that carries the document’s content and organization.
5. Create chunks with context. Include useful hierarchy and metadata in the smaller pieces used for retrieval.
6. Record bounding boxes. Connect evidence to the relevant region of its original page.

Chunking remains useful even as context windows grow, because filling long contexts carries a compute cost. Region-level captioning also makes attribution more precise: a user can inspect the particular chart or table behind a claim instead of searching an entire page.

The parse playground exposes text-only extraction, image captioning and a mode for long tables. These settings can be tried in the interface before being used programmatically through the Python or JavaScript SDKs. The goal at this stage is concrete: produce a faithful representation that downstream retrieval can search.

The query path begins with optional translation and query reformulation. Conversation-aware reformulation resolves references to previous turns. Expansion makes terse company abbreviations more explicit. Decomposition splits a complex question into smaller questions that can be answered separately. These are configurable operations because different users create different retrieval problems.

Each reformulated query then goes through semantic and lexical search. A long question might become three subqueries, each using that hybrid retrieval path. A datastore filter narrows the scope when the target is already known: a question about Apple revenue need not consider Microsoft and Tesla documents. Shah describes a broad candidate pool, mentioning roughly 100 chunks and then 80, followed by a configurable default of 15 reranked results. The stable mechanism is broad retrieval followed by narrower ranking; the spoken candidate count is approximate. An optional filter can remove additional results after reranking.

The instruction-following reranker adds a prompt to the usual relevance decision. For an Apple question, an instruction to prefer recent documents lets the reranker use chunk contents to favor newer material. This separates two operations: filtering determines which documents are eligible, while reranking orders the eligible evidence according to relevance and the additional instruction.

The selected evidence reaches a model fine-tuned to respect supplied context. Contextual also distinguishes factual statements from commentary in the response, allowing a setting that removes commentary when an application wants only factual content. This gives developers a separate control over answer style rather than relying exclusively on a general instruction to be concise.

The groundedness check examines whether claims in the response are supported by the retrieved documents. Shah deliberately prompts the model to invent a claim; the interface highlights the unsupported sentence in yellow. Underneath that display, another model decomposes the generated answer into individual claims and checks each against the retrieved context. Retrieved chunks and groundedness scores are available through the API as well as the interface. The check tests support in the provided evidence, which is a different question from whether the underlying documents are themselves correct.

LMUnit takes a prompt, a response and a natural-language test describing the property to evaluate. Lopatina introduces it as a fine-tuned judge, citing the release-era FLASK and BiGGen Bench leadership claims described in the LMUnit announcement. Because connectivity has failed again, she walks through previously generated results. The notebook retains commented code for generating the dataset, while a saved evaluation CSV contains six prompt-response pairs from the workshop agent.

The evaluation criteria reflect the actual failure modes of the dataset:

LMUnit scores each natural-language criterion on a 1–5 scale. Keeping the criteria separate makes a correct-but-verbose answer distinguishable from a concise answer with an unsupported number.

For the NVIDIA Q4 fiscal 2025 data-center-revenue question, Lopatina reports a score of 2.2 out of 5 for avoiding unnecessary information. She says the answer contains the correct amount and a citation, but adds too much other information. The notebook frame displays the more precise returned value, 2.281. The useful diagnosis is narrow: improve relevance without losing the correct answer or its evidence.

Lopatina warns against drawing conclusions from one answer and notes that even the six-query batch is small. The notebook evaluates every response against every test, then displays per-response scores. It maps the longer criteria to accuracy, causation, synthesis, limitations, evidence and relevance for polar plots. The Neptune/burglary response scores comparatively well on causation, accuracy and limitations, but less well on synthesis. The fuller notebook also supports clustering results to find patterns in a larger evaluation dataset.

The improvement exercise is to revise the agent’s system prompt so it answers the direct question briefly, then update the agent through the API. A small Python preparation step can preserve the existing prompt and make the proposed addition explicit:

from pathlib import Path

original_prompt = Path("system_prompt.txt").read_text(encoding="utf-8")
proposed_prompt = (
    original_prompt.rstrip()
    + "\n\nKeep responses brief. Answer only the direct question."
)

Path("proposed_system_prompt.txt").write_text(
    proposed_prompt,
    encoding="utf-8",
)

This prepares a prompt change; applying it to the agent and reevaluating are subsequent steps. The workshop proposes the update but does not demonstrate an improved rerun. Keep the same questions and criteria when comparing the revised agent so the result measures the change you intended to make.

The notebook’s final integration links lead to the Contextual AI MCP Server and a companion walkthrough. Shah uses a recorded integration demonstration to work around unreliable connectivity. In that workflow, Claude Desktop or Cursor can route a topic-specific question to an existing Contextual agent. The workshop uses a local server; the repository’s newer hosted-server options are separate from this demonstration.

The local integration has three parts:

1. Wrap the agent API. Clone the repository and configure the demonstrated server.py to call the relevant Contextual agent.
2. Describe each tool’s domain. Its docstring tells the client when to select it. Shah’s local server exposes separate tools for technical and financial questions.
3. Configure the client. In Claude Desktop, Developer → Edit Config opens the configuration that points to the local server. Other clients have their own setup.

The tool description is part of the routing mechanism: an accurate description helps the client choose the appropriate source of enterprise knowledge. The same pattern can serve a custom deep-research application.

At workshop time, Shah describes parse, rerank, generate and LMUnit as consumption-priced components, says they are billed by token, and offers a $25 signup credit. Platform consumption pricing is being introduced around document ingestion and query counts. For workloads needing latency or queries-per-second commitments, he describes provisioned throughput backed by dedicated hardware. These are the commercial terms presented in the recording, rather than a current price schedule.

The adoption choice remains modular: use the full managed pipeline or replace an individual component that is underperforming. The Contextual API examples include agent tuning and evaluation integrations such as Ragas, extending the workshop’s initial workflow into repeated improvement.

The audience first asks who should own the evaluation-and-prompt feedback loop. Shah distinguishes business users experimenting with system prompts from developers changing retrieval top_k or query reformulation. His illustrative move from 80% to 90% accuracy requires error analysis: determine whether failures come from retrieval or generation, then change the setting that affects that failure. Exposing a control in the interface does not make its consequences obvious.

A company with thousands of PDFs and Excel files plus an MRP system backed by SQL asks how to package its data and whether custom queries are necessary. Shah offers two paths: self-service APIs, or customer machine learning engineers who help build evaluation datasets and improve the agent. He does not provide a data-type checklist or a concrete consulting package in the answer. The service manages infrastructure, but preparing a useful evaluation and understanding the data still require expertise.

For existing TypeScript or JavaScript agents, Shah points to the APIs and JavaScript SDK. He describes a shift from selling only an end-to-end platform to allowing independent components, and says most customers integrate those capabilities into their own interfaces. Asked how many documents the trial credit buys, he gives no workload estimate and suggests contacting the team if the allowance blocks experimentation.

Data-residency requirements narrow the deployment choices. At the time of the recording, Shah describes Contextual’s own cloud, a Snowflake partnership and VPC installations. Custom on-premises deployments are excluded because of their upfront work. An audience member asks about government environments; Shah says AWS GovCloud has not yet been undertaken, so ordinary VPC availability should not be read as confirmation of government-cloud support.

The scalability answer is similarly bounded. Asked how configuration should change between hundreds and millions of documents, Shah cites Qualcomm as using tens of thousands of documents with hundreds of pages each. He explicitly gives an approximate customer example, then refers detailed questions to the platform engineer. No million-document measurement or configuration recipe is supplied.

Asked whether LMUnit scores are deterministic, the team reports repeated testing, comparisons with trusted metrics and use of a random seed. Their advice is to keep the prompt and natural-language test consistent: changing either can change scores unexpectedly. They point to the LMUnit paper for more detail, but provide no numerical variance or determinism guarantee. The paper’s current landing page includes a later revision, so it should not be confused with the release-era version discussed here.

A separate integration question distinguishes Azure hosting from Microsoft Copilot support. Shah says custom applications can call the APIs and describes Azure as an available custom VPC deployment. He does not confirm an existing Copilot integration or specify what building one would require.

The remaining RAG challenges begin with document understanding: complex tables, charts, long documents, hierarchy and relationships between sections. Shah adds ingestion speed at scale and structured-data questions such as text-to-SQL. These are still difficult even when the pipeline is managed.

The research direction is a dynamic retrieval workflow. Instead of always traversing a fixed pipeline, model routers could decide which tools to use, how many times to retrieve and when to change retrieval configuration. The team describes this as a form of deep research for RAG. At recording time, it is research moving toward the product, not a capability demonstrated by the workshop’s static agent.

A question about searching a company’s public website leads to a distinction between general internet search and ingesting a particular domain. Shah mentions Tavily and Firecrawl, then brings in the integration perspective. Website content can arrive through scraping; application records can arrive through API-based ETL; unstructured files can arrive from data-cloud or blob storage. An ingestion queue handles incoming work. Customers can maintain freshness with scripts, functions or daily cron jobs, while credential-based managed synchronization is described as a roadmap item expected in two to three months from the recording.

Frequently updated content and document permissions are separate requirements. Shah describes a continuous ingestion pipeline for changes, then uses HR salaries indexed alongside customer-support documents to illustrate the authorization problem. A relevant result can still be a result the user must not see. He says an entitlements layer is being added to respect enterprise governance and search permissions; it is not presented as a completed feature in this demonstration.

Asked about recent progress in RAG, Shah highlights improving rerankers and stronger vision-language models for images, alongside steady component-level improvements. He frames this as an immediate assessment rather than a comprehensive survey of breakthroughs.

A general parser is not automatically a replacement for every specialized invoice parser. Asked about QR codes, barcodes and their metadata, Shah says the team has not specifically focused on those cases. Image captioning might recover some information, but successful decoding is not demonstrated. His recommendation is to try the parsing API on the actual document type; the modular design makes replacement possible without promising perfect coverage.

Domain-specific terminology creates a different adaptation problem. Shah describes prompt adjustments on the retrieval side and, for some technical customers, fine-tuning the grounded language model toward their vocabulary. Self-service fine-tuning had been available but was withdrawn because tuning models for RAG is complex. In the workshop’s account, this remains work done with customer machine learning engineers.

For PHI, the audience asks whether sensitive data should be hashed before upload. Shah says the company has just become “HIPAA certified,” but that wording does not establish government certification or specify the controls needed for a particular deployment. He separately states that automatic PHI identification, filtering or masking during parsing is not in the product. A supplementary job can look for PII after parsing, but the answer does not establish a pre-upload protection workflow. Compliance claims and automatic sensitive-data handling must therefore be assessed separately.

The final substantive question reaches a limit that grounding alone cannot remove: what if the source contains two different values for the same fact? Shah says retrieval may bring both to the model, leaving it to reason about the discrepancy. His example contrasts a mosquito described as 100 feet long with one described as three millimeters long; he expects the model to reject the implausible value. That introduces model plausibility judgments into a system otherwise described as prioritizing supplied context. When both competing numbers are reasonable, he acknowledges that the conflict can cause trouble.

The questioner’s real problem is stale Excel sheets: the documents exist, but their information has not been maintained. Shah recommends richer metadata so retrieval can prefer newer material or a document written by the authoritative person. Grounding can tie an answer to a source; it cannot make an outdated source authoritative. Recency, ownership and source maintenance therefore remain part of the application’s information design, even when the RAG pipeline itself is managed.