Scaling Enterprise-Grade RAG Systems: Lessons from the Legal Frontier

How do you build retrieval for lawyers whose work ranges from analyzing uploaded documents to researching entire legal corpora? Calvin Qi approaches that problem through his work leading a retrieval team at Harvey. Chang She brings the infrastructure perspective as LanceDB’s co-founder and CEO, a pandas co-author with about two decades of experience building data tools. Their shared problem connects retrieval quality to the systems that store, process, and serve the underlying data.

Harvey’s legal assistant supports drafting, document analysis, and legal workflows. Those tasks operate across three progressively larger contexts:

A vault supplies the context for a particular matter; a corpus supplies the broader legal knowledge against which that matter may need to be understood.

The difficulty grows with both volume and density. Long legal documents complicate choices between sparse retrieval, which preserves lexical signals, and dense retrieval, which captures semantic relationships. Expert questions add another layer: lawyers must help engineers understand which details belong in representations, preprocessing, indexes, and queries. Confidential deals, IPOs, and financial filings also impose privacy requirements. Finally, the system needs evaluation that can detect legal mistakes its builders may not recognize themselves.

Qi illustrates query complexity with a question about the applicable regime for covered bonds issued before 9 July 2022, under directive EU 2019/2062 and Article 129 of the CRR. These are the identifiers and date used in the presentation’s example, rather than a verified legal citation. He places the question near the middle of Harvey’s query-complexity range. Even this ordinary case contains several distinct retrieval requirements.

The query must be decomposed before its retrieval needs can be matched to appropriate technologies. Semantic similarity alone does not express the date constraint or guarantee an exact identifier match. Conversely, matching identifiers does not establish how the provisions apply together. The example demonstrates a retrieval-planning problem; it does not provide the legal answer.

Once a question requires several retrieval mechanisms, validating the combined system becomes substantial engineering work. Qi describes spending considerable effort on evaluation, beyond developing algorithms or agentic techniques. That investment matters especially when engineers and researchers lack the domain knowledge needed to judge an answer directly. There is no single evaluation that covers every task depth and complexity.

The fastest layer includes retrieval precision and recall, along with checks for the expected folder, section, or keywords. These tests provide quick signals while expert review addresses questions that a simple retrieval score cannot resolve.

A small Python evaluation can make that fastest layer explicit. Here, document relevance labels support precision and recall, while metadata checks test whether retrieval stayed inside the intended folder and found the expected section. The records are a teaching example of the evaluation pattern:

relevant_ids = {"doc-a", "doc-b"}
retrieved = [
    {
        "id": "doc-a",
        "folder": "eu-law",
        "section": "Article 129",
        "text": "covered bonds",
    },
    {
        "id": "doc-c",
        "folder": "eu-law",
        "section": "Overview",
        "text": "capital requirements",
    },
]

retrieved_ids = {row["id"] for row in retrieved}
hits = retrieved_ids & relevant_ids

metrics = {
    "precision": len(hits) / len(retrieved_ids) if retrieved_ids else 0.0,
    "recall": len(hits) / len(relevant_ids) if relevant_ids else None,
    "correct_folder": all(row["folder"] == "eu-law" for row in retrieved),
    "expected_section_found": any(
        row["section"] == "Article 129" for row in retrieved
    ),
    "keyword_found": any(
        "covered bonds" in row["text"].lower() for row in retrieved
    ),
}
print(metrics)

Finding the right section or keyword is useful evidence about retrieval behavior. It still does not establish that the final answer interprets the law correctly.

The data side introduces its own domain structure. Legal sources from different countries require different filtering, organization, and categorization. Harvey works with experts to understand those distinctions, then turns their guidance into heuristics or LLM processing where automation is practical. The expert contribution therefore affects the organization of the corpus, not just the evaluation of generated answers.

Performance has two separate meanings here. Online performance means acceptable latency when a user queries the corpus. Offline performance means ingesting and re-ingesting data, running ML experiments, and comparing variations without making each experiment prohibitively slow. Qi describes individual corpora reaching tens of millions of documents, many of them large. At that scale, repeated processing is a core workload rather than an occasional maintenance task.

Reliable service and smooth onboarding must accompany scale. ML and data teams need to spend their time improving quality and building customer applications, rather than repeatedly tuning databases or scaling them manually. Qi also qualifies that goal: automation does not remove the need for operational awareness.

The remaining requirements span governance, observability, and retrieval:

• Storage segregation: isolate data according to the customer and use case.
• Retention: respect legal limits on how long particular documents may remain stored.
• Telemetry: expose database usage and behavior so teams can understand the service they operate.
• Query flexibility: support exact matches, semantic matches, filters, and dynamic or agentic navigation over the data.

These requirements must hold together at scale. They establish the infrastructure problem that leads into LanceDB’s part of the architecture.

LanceDB’s proposed foundation is a multimodal lakehouse: AI data stored together on object storage, with support for the work surrounding retrieval. Feature extraction, summaries, and text descriptions of images all create useful derived data. The same foundation should support search, analytical queries, training, and preprocessing, so teams can experiment with new features without rebuilding the data pipeline for each workload.

A lakehouse that handles large offline batches does not automatically provide good online serving. LanceDB addresses that distinction with a distributed architecture that separates compute, memory, and storage while serving data from cloud object storage. Its retrieval interface can combine multiple vector columns, vector search with full-text search, and reranking. Python and TypeScript APIs expose those operations in a dataframe-style interface familiar to pandas or Polars users.

She reports a GPU-indexing record of roughly 3–4 billion vectors in one table in under approximately 2–3 hours; hardware, vector dimensions, and index settings are not specified. That is an approximate reported record, not a general indexing-time guarantee. He also attributes lower costs to compute/storage separation and object storage, without supplying a quantified cost comparison.

The shared-data design extends beyond storing vectors beside text. Images, videos, audio, embeddings, tabular values, and time series can share a table. That table becomes a common source for search, analytics, training, preprocessing, and feature engineering.

The open-source Lance format supplies the underlying storage capabilities. She emphasizes random access, large-blob support, and efficient schema evolution, contrasting these with limitations he encounters in workflows using WebDataset, Iceberg, and Parquet. This is his architectural comparison, rather than evidence of universal superiority over those systems. The practical problem is duplicated fragments of data across different stores: teams spend time keeping the copies connected and synchronized.

She’s shorthand for Lance is Parquet plus Iceberg plus secondary indexes, adapted for AI data. The important distinction is the access pattern each capability serves:

A useful shared format must support selective reads without giving up efficient bulk processing. It must also handle records whose payloads differ substantially in size.

Apache Arrow provides the main interface to the surrounding data ecosystem. Spark and Ray can write Lance data in distributed jobs; PyTorch can load it for training or fine-tuning; pandas and Polars can query it. This allows the storage layer to participate in existing data workflows rather than requiring every workload to move into a separate toolchain.

The engineering choices begin with understanding the domain: the structure of the data, the user’s task, and the explicit and implicit patterns inside queries. Working closely with experts makes those requirements concrete enough to guide modeling and infrastructure choices. Flexibility then becomes essential because tools, retrieval paradigms, and model context windows keep changing.

Evaluation makes that flexibility usable. Good evaluation sets, expert procedures, and automated checks provide a faster signal about whether a change improves accuracy. Infrastructure must support the same pace of experimentation while accommodating multimodal data, vectors and embeddings, increasingly diverse workloads, and growing collections of public and private information. The capacity to revise a retrieval system depends on both trustworthy quality signals and a data foundation that can keep up with the revisions.