The RAG Stack We Landed On After 37 Fails

How do you turn a convenient RAG prototype into a system whose data and processing can stay inside a financial institution? That constraint shapes Jonathan Fernandes’s stack. An independent AI engineer working with financial clients, he prototypes in Google Colab for its free hardware accelerator, then uses Docker when deployment must support on-premise processing or a cloud environment. The talk’s “37 fails” framing introduces accumulated implementation lessons, rather than a numbered account of failed experiments.

The first choices concern orchestration, embeddings, storage, and generation. APIs make experiments easy; downloadable models make it possible to keep processing within the deployment boundary.

Fernandes reports that Qdrant works for his workloads from a few documents to hundreds of thousands; he provides no accompanying latency, recall, or hardware measurements. For generation, his open-model examples include Llama 3.2 and Qwen3-4B, with Ollama or Hugging Face’s Text Generation Inference as serving options.

The remaining components make the pipeline inspectable and its answers assessable:

• Tracing: LangSmith or Arize Phoenix during prototyping, with Phoenix preferred in production because it can run in a Docker container. Component timings help identify whether generation or another stage consumes most of a request’s time.
• Reranking: A closed Cohere model is convenient for experiments; an open NVIDIA model fits the Docker deployment.
• Evaluation: Ragas supplies the framework for checking answer quality.

These are separate responsibilities: tracing locates execution problems, reranking changes the evidence selected for an answer, and evaluation checks whether the resulting behavior is useful.

The working example is a railway operator’s knowledge base: a collection of HTML help articles, queried with a deceptively simple question: “Where can I get help in London?” The answer must come from the operator’s information, so retrieval determines which facts generation can use.

The basic RAG path has three stages:

1. Retrieve: Embed the query and search the vector database for semantically related documents.
2. Augment: Combine those documents with the original question as context for the language model.
3. Generate: Ask the model to produce an answer from that context.

The model receives the selected evidence, not the whole knowledge base. Fernandes takes this minimal pipeline into Colab so each component can be inspected and replaced.

Before indexing anything, inspect what the corpus actually contains. The first article asks whether London has wheelchair-friendly taxis and explains that most black cabs have ramps. Its Paris counterpart describes an accessible taxi firm and provides a telephone number. Both are plausible matches for words associated with travel, assistance, and accessibility, but neither necessarily answers where someone should go for help inside a station. Fernandes copies the HTML files into the notebook’s data folder and views the articles there.

LlamaIndex’s SimpleDirectoryReader loads the directory into document objects. The demonstrated corpus contains 32 documents. Each object exposes a document ID, metadata such as its file path, and its text—which in the inspected examples contains HTML. Looking at these fields connects an indexed object back to the original article:

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()

The useful inspection goes beyond whether loading succeeded: check the text the index will receive and retain the file metadata needed to trace a later answer back to its source.

The initial implementation loads the documents, builds an in-memory vector index, and creates a query engine. With the notebook’s model credentials configured, the core Python workflow is compact:

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

question = "Where can I get help in London? I'm at the station."
response = query_engine.query(question)
print(response)

The answer recommends approaching London black cabs because most have ramps. It draws on relevant-looking material, but does not give the passenger a useful assistance location at the station. A functioning retrieval-and-generation chain has produced an unsatisfactory answer.

Two places in the pipeline allow more targeted intervention. Query processing happens before retrieval; for example, personally identifiable information can be removed before a question enters the RAG system. Post-retrieval processing happens after the vector search and can improve which retrieved documents reach generation. The next part of the demonstration develops that second intervention.

A cross-encoder takes the query and a document together. In the architecture Fernandes presents, BERT—a pretrained bidirectional Transformer encoder—processes the pair, and a classifier produces a relevance score. Because the model sees both texts together, it can judge their relationship directly. The slide depicts an output between zero and one; cross-encoder implementations can also emit logits, with a sigmoid optionally mapping them into that range.

Joint processing is expensive when repeated across many documents, and longer documents increase the work for each pair. Fernandes therefore positions the cross-encoder as a way to gain additional accuracy over a small candidate set, rather than as the first search over the entire corpus.

A bi-encoder separates the two encoding paths. The query passes through an encoder and pooling step to become a vector; a document passes through its own encoding path to become another vector. A similarity function such as cosine similarity then compares them. This separation lets document vectors be prepared for retrieval and compared with a query vector without jointly processing every query–document pair.

The two architectures fit consecutive stages. Vector search finds a manageable set of candidates; the cross-encoder then spends more computation deciding which candidates best answer this particular question. In that post-retrieval role, the cross-encoder is the reranker.

The first replacement is Qdrant. Although Qdrant is also the production database choice, this notebook uses its in-memory local mode. Switching the vector-store integration does not by itself turn the notebook into a persistent server deployment.

Next comes the embedder. The notebook shows text-embedding-ada-002 as its existing OpenAI embedding model and mentions text-embedding-3-large as another API option. Fernandes instead downloads BAAI BGE small from Hugging Face into Colab and makes it LlamaIndex’s default embedding model. That moves embedding computation from a closed API to a locally available model.

Generation changes from GPT-3.5-turbo at temperature 0.1 to GPT-4o at temperature 0. These and the earlier embedding default are settings shown in the recorded notebook, not a statement of current library defaults. The generation change is expressed through LlamaIndex’s settings:

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-4o", temperature=0)

Fernandes then queries the pipeline using GPT-4o, the open embedder, and Qdrant.

The answer now tells the passenger to arrive at least 75 minutes before departure and notify a team member. This is closer to station assistance than the black-cab answer, but still does not supply the requested place to get help. Changing the models has changed the response without resolving the missing location.

To investigate, inspect LlamaIndex’s source_nodes. They expose the documents used for the response, allowing the answer to be checked against its supporting HTML files:

for source in response.source_nodes:
    print(source.node.metadata.get("file_path"))
    print(source.node.get_content())

In the revised notebook response, two HTML files supplied the evidence. One is titled “Can I get assistance for a connecting UK train journey?” Following those source nodes makes the problem concrete: inspect the retrieved material before deciding that a different generator will fix the answer.

The next change adds Cohere reranking through LlamaIndex’s CohereRerank node postprocessor. Qdrant retrieves five candidates, and Fernandes describes a top-two reranking setting. Those counts serve different purposes: similarity_top_k controls the candidates retrieved, while top_n controls how many the reranker retains. Expressing the five-candidate, two-retained interpretation gives this query-engine configuration:

import os
from llama_index.postprocessor.cohere_rerank import CohereRerank

reranker = CohereRerank(
    api_key=os.environ["COHERE_API_KEY"],
    top_n=2,
)
query_engine = index.as_query_engine(
    similarity_top_k=5,
    node_postprocessors=[reranker],
)
response = query_engine.query(
    "Where can I get help in London? I'm at the station."
)
print(response)

The integration’s parameter meanings distinguish retrieval breadth from the final context size; the narration alone does not establish the exact historical configuration.

The answer now names London St. Pancras International and directs the passenger to booth number five beside the Eurostar ticket gates. It is the best response in this demonstration because it supplies an actionable location, rather than taxi information or advance-arrival instructions. That location also appears in Eurostar’s current assistance guidance; for travel planning, the current guidance recommends 60 minutes before departure, rather than the earlier generated answer’s 75 minutes.

The demonstration has tested one recurring question, not measured aggregate answer quality. Fernandes returns to tracing and evaluation at this point because a good answer does not explain how the pipeline will behave across the rest of the knowledge base. Phoenix or LangSmith can show the time spent in each component; Ragas supports broader evaluation across documents and different dimensions of quality.

The production choices follow the earlier deployment constraints: open BAAI or NVIDIA embeddings, Qdrant for storage and retrieval, a deployable Phoenix Docker image for tracing, and probably an NVIDIA reranker instead of the prototype’s Cohere API. Ragas uses a language model to help assess quality across multiple dimensions. Evaluation belongs alongside the pipeline as its corpus and component choices change, rather than being replaced by confidence in one successful response.

The deployment walkthrough ends with a compose.yml file coordinating Docker images as running containers. An ingestion service connects to the knowledge base and pulls the HTML files. A Qdrant image, available from Docker Hub, provides the vector database, while a frontend application exposes the solution to users.

Ollama usually serves Fernandes’s models, with Text Generation Inference presented as an alternative. Phoenix supplies tracing, and Ragas supplies evaluation. The completed diagram puts ingestion, Qdrant, the application, model serving, tracing, and evaluation under Docker Compose, with separate selections for embedding, reranking, and generation models. The notebook’s retrieval experiment has become a deployment of distinct services: the knowledge base can be ingested, answers can be served, and both execution and answer quality can be inspected within the chosen environment.