When Vectors Break Down: Graph-Based RAG for Dense Enterprise Knowledge

What does a retrieval system need beyond vector similarity to answer sophisticated questions reliably? The expansion of RAG made semantic search widely accessible, but retrieving similar text is only one part of assembling the right evidence. Sam Julien opens with Jo Kristian Bergum’s article about the rise and fall of the vector database infrastructure category and his subsequent Latent Space interview: the infrastructure boom following ChatGPT was giving way to recognition that sophisticated retrieval needs multiple strategies.

Writer had already described its approach in a November 2023 article on enterprise RAG. The company builds its own models, graph-based retrieval system, and tools for enterprise agents and applications. Working with large organizations, including regulated healthcare and finance businesses, made accuracy and hallucination reduction central engineering constraints. The resulting architecture grew through successive attempts to solve those constraints; the lessons from that process matter alongside the final combination of techniques.

Julien, introducing himself as Writer’s director of developer relations, describes the architecture as a historical sketch rather than a blueprint of current production. That scope matters: the presentation explains why components were added or replaced, without specifying every layer of the deployed system.

Writer’s research program connects four areas:

These areas share a practical feedback loop: customer problems inform research, which informs the product. The problem determines the technique, rather than a commitment to a particular database or model architecture.

Enterprise knowledge can comprise terabytes of documents written in specialized, repetitive language. Writer initially searched a knowledge base and passed the results to an LLM. That supported basic keyword queries, but the need for similarity matching led to the familiar vector pipeline: split documents into chunks, embed them, store the embeddings, retrieve similar chunks, and pass the results to the model. Two problems emerged.

The first was losing the association between a fact and its subject. In Julien’s Apple history example, a reader can distinguish the Macintosh’s 1984 introduction from the Lisa’s 1983 introduction. Naive chunking places nearby information together, and a nearest-neighbor retrieval pipeline can supply context that leads to the wrong answer: 1983 for the Macintosh. The passage is relevant to Apple history, but relevance alone does not preserve which date belongs to which machine.

The second was concentrated vocabulary. A mobile phone company may have thousands of documents discussing cameras, megapixels, and battery life. Asking for a comparison between two phone models can retrieve many plausible passages without clearly separating the facts belonging to each model. Semantic proximity helps find the topic; the comparison still requires the correct entity-to-attribute relationships.

Graph-based retrieval introduced an explicit way to retain those relationships. Writer queried a graph database, used keys to retrieve relevant documents, and supplied the results for answer generation. Combined with full-text and similarity search, the graph added context about how the retrieved information fit together. Julien reports that this improved accuracy by preserving relationships in the text.

The approach also introduced four operating challenges:

• Graph construction: Converting source data into a structured graph became difficult and costly at scale.
• Database operation: Growth exposed both cost pressures and limits in the team’s graph database expertise.
• Similarity matching: Cypher was difficult to use for the advanced matching the application required.
• Query representation: The team observed better LLM results with text-based queries than with complex graph structures.

These were Writer’s historical constraints, not fundamental limitations of graph databases. The next design decision was therefore to preserve the useful relationships while changing the implementation to fit the team’s expertise.

Writer’s team knew how to build models, so it addressed graph construction with a specialized model designed to run on CPUs or smaller GPUs. Julien notes that, by the time of the talk, fine-tuning an available small model could offer an alternative to building one from scratch. Writer’s model mapped text into nodes and edges, while context-aware splitting and chunking helped preserve semantic relationships across the extraction process.

The storage decision separated graph representation from graph database infrastructure. Writer serialized graph data as JSON and stored it in a Lucene-based search engine. To make that representation concrete, the earlier Apple example could retain the product-to-year associations as explicit records:

{
  "nodes": [
    { "id": "lisa", "name": "Lisa" },
    { "id": "macintosh", "name": "Macintosh" }
  ],
  "edges": [
    { "source": "lisa", "relation": "introduced_in", "year": 1983 },
    { "source": "macintosh", "relation": "introduced_in", "year": 1984 }
  ]
}

This illustrative JSON makes the associations explicit; it is not Writer’s published storage schema. The underlying design choice is to keep relationship information while using search infrastructure the team can operate well. Julien reports that this supported large data volumes without speed degradation, but supplies no workload measurements for that claim.

The historical ingestion pipeline now had three distinct responsibilities:

1. Split the source with enough context to preserve meaning.
2. Convert text into graph relationships with the specialized model.
3. Pass the resulting representation to the search engine.

This improved the retrieval foundation, but did not yet resolve how to combine retrieved evidence effectively for generation.

Similarity matching and query representation remained open problems. The team returned to retrieval research to reconsider its assumptions. The original RAG paper combines a retriever with a pretrained sequence-to-sequence generator. Julien contrasts that trained architecture with the now-common practice of placing retrieved context and a question into an instruction prompt. The distinction is architectural: the original paper does use retrieved context and concatenate it with the input, so it should not be read as omitting question-and-context conditioning.

Fusion-in-Decoder offers another way to combine retrieval and generation. Instead of jointly encoding all retrieved passages as one long sequence, it encodes passages independently. The decoder then attends to their representations together, allowing evidence from different passages to contribute to one answer.

For a fixed passage length, separate encoding makes encoder work grow linearly with the number of passages, rather than incurring the quadratic self-attention growth of encoding their concatenation. That comparison concerns passage encoding, not a guarantee about total system latency or every part of decoding. The useful separation is that passages remain independent during encoding while their evidence is combined during generation.

Julien points to the Facebook Research FiD repository as a practical way to explore the technique. It provides a concrete research implementation of the encoder–decoder approach behind this alternative to a single assembled prompt.

The connection back to graphs is KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder. Retrieved passages are not necessarily independent pieces of evidence: graph relationships can connect them and inform which passages deserve further processing. The research architecture uses graph-based passage reranking in two stages, at retrieval and at an intermediate encoder stage. This addresses an efficiency bottleneck by improving the selection of evidence passed through the system.

Writer drew on this research to build its own Fusion-in-Decoder implementation, using its existing model-building capabilities. Julien reports that the addition reduced hallucinations and that Writer published a white paper describing its findings. He does not give a numerical reduction, and the research paper’s results are not measurements of Writer’s implementation. Fusion-in-Decoder became another component in a system assembled from multiple techniques; other techniques remained outside the presentation’s scope.

Writer evaluated its combined system using Amazon’s RobustQA benchmark, comparing it with seven vector-based RAG configurations. Those alternatives included hybrid search, so the comparison was broader than seven pure nearest-neighbor systems. In Writer’s 2024 report, its system has the highest reported RobustQA average score among the eight tested pipelines. Julien also describes it as the fastest, but the published table gives both Writer and the LangChain/Pinecone/Cohere configuration the same response-time bound of less than 0.6 seconds. That table does not establish a unique fastest system. The result is a vendor evaluation whose scoring and timing details are insufficient for a fully reproducible performance comparison.

The architecture also supports a visible evidence trail. Writer can expose snippets, subqueries, and sources through its product and API, giving users and developers material to inspect when assessing an answer. Although Julien describes this as exposing the thought process, these are observable retrieval artifacts, not access to a model’s private internal reasoning.

Writer also presents the system as supporting multi-hop questions that combine evidence across documents and topics. Related cases include answers spread across multiple pages and passages containing similar terms that do not match the user’s intent. Graph relationships help maintain the connections among those pieces, while Fusion-in-Decoder provides a way to aggregate their evidence. These are the capabilities Writer attributes to the combined architecture, rather than guarantees that graph retrieval will answer every such question correctly.

The benefits of graph relationships do not require one storage technology. A graph database, a design built with Postgres, or a search engine can each support an approach that preserves and uses relationships in retrieval. Writer’s journey illustrates why that distinction matters: changing storage did not require abandoning the information structure that improved its answers.

The engineering process followed the same logic. Customer needs set the objective; team expertise shaped which implementations were workable; research challenged assumptions about how retrieval and generation should fit together. Those principles allowed the team to change components while continuing to pursue the same goal: reliable answers from dense enterprise knowledge.