RAG Agents in Prod: 10 Lessons We Learned — Douwe Kiela, creator of RAG

Douwe Kiela, CEO at Contextual AI, frames enterprise AI as a mismatch between enormous expectations and inconsistent realized value. Organizations invest heavily, yet leaders still struggle to show a clear return. He connects this tension to a robotics paradox: tasks that appear intellectually difficult can be easier for machines than seemingly ordinary activities that require situational understanding. 0:17

In enterprise AI, the analogous problem is context. Language models can perform impressive coding and mathematical tasks, but they still struggle to place information within the right organizational situation. Human specialists do this almost automatically by drawing on accumulated expertise and intuition; enterprise systems must reproduce something closer to that contextual judgment before they can solve consequential business problems. 2:18

This creates a tradeoff between convenience and differentiation. General-purpose assistants can make employees more efficient, but business transformation depends on handling the specific context embedded inside an organization. The more differentiated the desired outcome, the more effectively the system must incorporate enterprise knowledge rather than relying on generic model capability alone. 2:18

Kiela argues that a language model may represent only 20% of a much larger system. In enterprise deployments, that broader system often takes the form of a RAG pipeline that connects generative AI with organizational data. His practical comparison is that a merely adequate model surrounded by an excellent retrieval pipeline can outperform a stronger model surrounded by a poor one. The relevant unit of engineering is therefore the complete system that solves the business problem. 4:24

The next design choice is specialization over AGI. General-purpose assistants struggle to match the expertise already present inside a company, especially when the problem is difficult, domain-specific, and sufficiently well understood. A specialized system can be organized around that existing institutional knowledge rather than expecting generalized intelligence alone to reproduce expert performance. 5:31

Proprietary enterprise data supplies the basis for this differentiation. Kiela treats organizational data as a durable expression of what makes a company distinctive, while cautioning against assuming that extensive manual cleaning must precede useful AI deployment. The harder but more valuable capability is enabling AI to work with noisy data at scale; success there turns existing information into a competitive advantage instead of requiring the organization to sanitize everything first. 6:38

A working pilot can create a misleading impression of readiness. A small RAG demo assembled from an existing framework and a modest document collection may impress its first users, yet production introduces much larger corpora, many more users, numerous distinct use cases, and organizational expectations that the demonstration never had to satisfy. Kiela emphasizes that moving to tens of thousands, hundreds of thousands, or millions of documents is substantially harder than building the initial proof of concept. 7:45

Production also introduces security and compliance requirements alongside scaling challenges. Kiela’s recommendation is to design for production from the beginning rather than optimizing exclusively for a pilot and attempting to retrofit operational requirements afterward. This is not a call to delay release until every detail is polished; it means accounting early for the conditions under which the system must eventually operate. 7:45

Within that production-oriented architecture, speed matters more than perfection. Teams should put a barely functional system in front of actual users early, rather than relying exclusively on friendly testers, and then improve it through their feedback. This iterative approach makes deployment a process of incremental improvement toward usefulness instead of a single attempt to produce a flawless system before confronting real working conditions. 8:46

Fast iteration requires deciding what engineers should not spend their time doing. Kiela points to chunking strategies and prompt adjustments as examples of implementation details that vary across use cases and frameworks but can divert attention from the more important question of delivering differentiated business value. Where capable RAG-agent platforms can abstract those details effectively, engineering effort should shift toward the problems that actually distinguish the organization from competitors. 9:51

A deployment is not successful merely because it is technically running. Kiela describes cases where production AI systems attract almost no usage, either because organizational review processes leave them barely useful or because employees do not know how to apply them. Workflow integration addresses this adoption gap: the closer a system fits existing enterprise workflows, the more likely it is to become part of actual day-to-day work. 11:01

Onboarding should also minimize the time required for users to experience a concrete benefit. Kiela describes Contextual AI running in production globally with Qualcomm and thousands of customer engineers; in one example, an engineer discovered a seven-year-old document that had been hidden from view and finally obtained answers to longstanding questions. The lesson is not simply that retrieval can locate old files, but that a personally meaningful discovery can create the early confidence and internal advocacy needed for broader adoption. 12:08

Kiela treats accuracy as necessary but insufficient. Even if a system reaches 90% or 95% accuracy, an enterprise must still decide what happens in the remaining cases, and perfect accuracy may be unattainable. Once a minimum quality threshold is met, the practical issue becomes how the organization understands, investigates, and manages the errors that remain. 13:20

His answer is observability, including careful evaluation, audit trails, and attribution to supporting documents. In regulated settings especially, an organization needs to understand why an answer was generated and what evidence supported it. Kiela also recommends checking generated claims and applying post-processing so that attributions are substantiated rather than merely attached superficially. 13:20

Finally, ambition should be measured by the potential business value of the problem being solved. Kiela contrasts consequential enterprise applications with assistants limited to basic questions about benefits providers or vacation allowances: those narrow conveniences may be easy to deploy without producing meaningful ROI. His closing argument brings the lessons together: build complete systems instead of chasing models, specialize around enterprise expertise, make failures inspectable, and choose problems whose successful resolution would materially matter. 14:22