OpenRAG: An open-source stack for RAG — Phil Nash

If a model accepts a million tokens, why not put all your business’s information into the prompt? The first problem is capacity: a business can have more information than that. The second is recurring input cost: even when the information fits, sending it with every question may not be an acceptable way to operate. Phil Nash opens with this challenge to the claim that large context windows make retrieval-augmented generation unnecessary. The million-token example is illustrative, not a measured cost comparison.

A more useful question is whether RAG is already solved. Its familiar pipeline is easy to describe:

1. Extract text from unstructured documents.
2. Split it into chunks, embed those chunks, and store them in a vector database.
3. Embed the user’s question and retrieve the top K results.
4. Give those results to the model as context for its answer.

That description makes retrieval sound like a small, settled part of context engineering.

The difficulty lies inside those steps. PDFs can be hard to parse. Changing a chunking strategy means testing its effects. Better embedding models create pressure to replace older ones. Chunk summaries, chunk expansion, cross-encoder reranking, and query rewriting introduce further choices. A reusable pipeline is not a universal configuration: different documents, users, questions, interaction patterns, and expectations require different decisions.

OpenRAG brings together existing open source projects to provide a baseline that can be customized. Docling handles document processing; OpenSearch handles indexing and search; Langflow provides visual orchestration and agents. The integration supplies common infrastructure while leaving application-specific decisions open.

Ingestion begins with the document itself. Docling originated in IBM Research Zurich and supports formats ranging from HTML, Markdown, and Word documents to slides, spreadsheets, audio, video, and PDFs. Different inputs take different processing paths:

For scanned PDFs without embedded text, the standard pipeline can use a selectable OCR backend. Its separate processing stages also provide options for image extraction and descriptions, allowing the pipeline to match the document’s needs.

An alternative PDF path uses Granite-Docling-258M, a vision-language model trained for document conversion. Instead of coordinating several focused models, this pipeline performs the extraction through one task-specific model. Its 258M designation describes model size, not extraction accuracy or speed.

The next requirement is to retain structure after extraction. Nash describes the relationship between DoclingDocument, XML-like DocTags, and exports such as Markdown, HTML, and JSON. The precise distinction in the Docling document model is that DoclingDocument is the structured object, implemented as a Pydantic datatype; DocTags is one serialization of that object, not the object itself.

That distinction matters for chunking: Docling’s native chunkers operate directly on DoclingDocument. They can use the hierarchy produced during parsing instead of treating the document as an undifferentiated stream of text. The useful continuity is from document structure to meaningful chunks; a conversion through DocTags is not a required intermediate step.

OpenRAG does not prescribe a single embedding model. Providers include OpenAI, watsonx.ai, and Ollama for locally hosted embeddings. Nash describes running the entire stack offline with local models, including Docling in air-gapped environments. That is the local configuration; hosted model providers and cloud document connectors still require network access.

Once embedded, chunks are indexed in OpenSearch, the open source fork of Elasticsearch. It supports vector search, keyword search, filtering, and aggregation. In the configuration presented, OpenRAG combines vector and keyword retrieval into hybrid search and exposes filters for narrowing the documents searched. It also supports vector search across multiple embedding models. Nash identifies a practical tradeoff: this can help during an embedding-model migration, but searching across models slows vector search.

There is a fourth open source component beneath the three-project stack: the JVector k-NN plugin. Nash contrasts OpenSearch’s standard HNSW and IVF index options with OpenRAG’s use of JVector by default in the presented setup. JVector supports live indexing, and its DiskANN-based architecture means the entire index does not have to fit in memory. That expands the available scaling choices without implying that search needs no memory or that a particular speedup has been measured.

Langflow connects these pieces into the ingestion flow. Its drag-and-drop editor integrates Docling, embedding providers, OpenSearch, and any additional data-enrichment stages. The ingestion pipeline is therefore something a developer can inspect and change, rather than a fixed sequence hidden behind the upload interface.

On the generation side, OpenSearch already holds the indexed documents. Langflow orchestrates the agent and gives it access to model providers including OpenAI, Anthropic, Ollama, and watsonx.ai. OpenRAG uses agentic retrieval to decide how to search that knowledge.

The change is who controls retrieval:

In the traditional path, success depends on the initial nearest-neighbor search finding chunks that contain the answer. In the agentic path, the model can perform multiple searches as needed and decide what to do with each set of results. The agent controls the search process as well as the eventual response.

The laptop demonstration begins in the chat interface after onboarding. Its suggested first question is “What is OpenRAG?” The response includes tool activity, but this is not yet a document-retrieval example: the answer is already in the agent’s default prompt. The agent nevertheless calls a tool to obtain the current date. Tool use and document search are distinct events.

The interface also offers suggested follow-up questions, called nudges, generated through Langflow. Choosing a follow-up about Langflow’s role in constructing AI agents leads to a documentation search. The agent then produces an answer and another set of nudges. This second interaction shows retrieval being used to support the conversation, rather than simply answering from information already supplied in the prompt.

The knowledge section accepts an individual file or an entire folder. It also exposes a sync control and lets users inspect stored objects, documents, and their chunks. That inspection makes the result of ingestion visible: users can look at what the system will actually retrieve.

Knowledge filters use OpenSearch filtering to narrow the search scope. Users create filters from attributes available in their data, then select those filters in chat to restrict retrieval to particular documents. Uploading information and deciding which information a conversation should use are separate controls.

The settings page extends ingestion to cloud document stores. These connectors require a user identity and authentication. The demonstrated setup uses Google OAuth with a client ID and secret; users can then connect document directories from Google Drive, SharePoint, or OneDrive. OpenRAG can synchronize those sources instead of requiring repeated manual uploads. Synchronization is intended to keep the ingested documents current, but the walkthrough does not specify a refresh interval or a freshness guarantee.

Model settings support API-based providers or local models through Ollama. Nash’s demonstration uses an IBM Granite 3B language model locally. The same settings expose the agent’s instructions, allowing the system prompt to be edited without opening the underlying flow.

For embeddings, the demonstration uses Qwen3-Embedding-0.6B through Ollama. The ingestion settings also expose chunk size and chunk overlap, followed by Docling’s processing options:

Picture descriptions can make information inside images available to the pipeline. Those additional processing models also add ingestion time, which is why Nash leaves OCR and picture descriptions off for this demonstration.

At the bottom of settings, API-key configuration enables access from another application. A developer can use OpenRAG’s search capability or its agent without making the bundled chat interface the application’s user interface.

The Edit in Langflow button opens the actual agent implementation. In the generation flow, chat input first passes through a prompt template. When the user has selected knowledge filters, the template includes information about those filters before passing the input to the agent.

The agent has several tools connected to it:

• URL ingestion: An MCP server exposes a URL-ingestion capability implemented as another Langflow flow.
• Calculator: Arithmetic can be delegated to a tool rather than left to the language model.
• OpenSearch retrieval: A search component supports retrieval across embedding models, with the embedding providers visible in the flow.

The canvas exposes both the agent’s incoming context and the capabilities it can invoke while responding.

Nash then changes the generation flow to show how customization works:

1. Unlock the flow and save it so it can be edited.
2. Add a guardrails component after chat input.
3. Add a parser to the guardrail’s pass path.
4. Extract the original input text through that parser and send it to the existing prompt template.

The pass path preserves the user’s text while inserting a check before prompt construction.

For the failure path, Nash proposes sending an error message to chat output. The completed canvas shows the pass branch running through the parser toward the prompt template and the fail branch connected to chat output. This demonstrates a flow edit, not a tested level of protection.

Ollama models can also be used in this customization. The same visual editor that assembles the baseline can therefore change its behavior: the developer can add components and branches around the existing agent rather than replace the entire application.

OpenRAG also exposes an MCP server alongside its API, allowing other agents to use the system. This is separate from the URL-ingestion MCP tool inside the generation flow: here, OpenRAG itself becomes a capability available to another agent.

Whether RAG is solved still depends on the application’s data and users. OpenRAG supplies an opinionated, agentic baseline through Docling, OpenSearch, and Langflow, while leaving room to customize how documents become context. Its defaults are a starting configuration, not evidence that every dataset or interaction pattern is handled equally well.

The version presented is OpenRAG 0.4.0. Nash closes by inviting developers to try the project, star its repository, and provide feedback.

The frontend is a Next.js application; Nash describes the remaining application code as Python. Contributions can go to OpenRAG itself or to Docling, OpenSearch, and Langflow. The intended balance is practical: make useful default decisions where the stack can, and preserve choices where the application needs them—all in components developers can inspect and change.