Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora

A lawyer reviewing an acquisition needs to search the documents for that particular deal. A lawyer researching a dispute may need to search laws, cases and regulations, then follow relationships between them. Legora serves both workloads, alongside contract review, drafting and collaboration. Jacob Lauritzen, a Legora engineer, introduces the application; Simon Eskildsen, turbopuffer’s CEO and co-founder, explains the search engine underneath it.

• Project search: Lauritzen imagines SpaceX acquiring Cursor. The law firm handling this hypothetical deal uploads employment agreements and supplier contracts into one project, then searches within that collection. Projects range from tens of documents to millions, but the scope stays clear: this matter’s documents.

• Legal research: The collection spans laws, previous cases and regulations. Lawyers use it to answer legal questions or support litigation. Finding relevant material also means understanding which sources apply and how they relate.

The project is already the application’s unit of work. The first part of the migration is about making it a useful unit of storage, too. That matters because a project can be intensely active for a while, then close and almost never receive another query.

Legora’s project-search journey runs from hundreds of thousands of documents to two billion. Initially, shared blob storage held the raw documents and one Elasticsearch cluster indexed and searched them for every tenant. Lauritzen says the arrangement worked reasonably well: it was simple, and the team had one search deployment to operate.

Customers in different regions wanted their processing to stay local. Legora reproduced its setup across the EU, US and Asia Pacific to meet requirements from European, American and Australian customers. The design still worked, but each regional copy brought more infrastructure to run. Residency changed the deployment topology before document volume forced a different search architecture.

Large banks and law firms added two further requirements:

• Physical isolation: Customers wanted their data separated, which Lauritzen describes here as effectively asking for their own database. He acknowledges that customers can mean different things by physical isolation.

• Customer-managed encryption keys: A customer keeps an encryption key in its key vault and grants Legora access. Legora uses the key to encrypt and decrypt data at rest. Revoking access prevents Legora from using that key for further decryption, giving the customer control over access to the encrypted stored data.

These demands made a shared search cluster harder to fit into the rest of the system. Legora already had to separate its transactional databases and blob storage, so the next move tried to consolidate search into those existing databases.

Moving search into Postgres had a practical appeal. Legora already ran it for transactional workloads and already needed multiple database and blob-storage arrangements. Adding vector and text search meant fewer kinds of systems to operate. The setup used pgvector for vector search and tsvector for text search. Text ranking no longer used BM25, and Lauritzen reports a loss in retrieval quality.

Hashing projects separated keys, but mixed their lifecycles

The team split its document-chunk table into roughly 4,000 partitions, hashed each project key and packed projects into those partitions. This spread projects across the database without keeping each project independently stored. Active projects and abandoned projects could land together. As the corpus grew, the partitions accumulated large amounts of data that almost never needed searching.

Lauritzen describes a query bringing partition data into memory, followed by another query loading a different partition and displacing the previous data. The cache kept making room for large collections containing both useful active data and inactive data. Repeated queries therefore failed to benefit from a stable working set. Search and ingestion P99 latency rose from about 100 milliseconds to 20 seconds.

The failure was specific to the packing arrangement and access pattern. Increasing the partition count had distributed the data, but a lawyer’s project remained only part of a larger storage collection. A cold project could still compete for cache space whenever an active neighbor was searched.

One namespace per project

Legora moved to turbopuffer at approximately 400 million documents, a size Lauritzen gives as an estimate. Each project received its own namespace. An unused project could remain in blob storage while an active project was cached and searched independently. The storage unit now followed the application’s unit of work.

The migration also restored BM25, improved relevance and latency, lowered cost and let Legora operate a single turbopuffer cluster instead of its collection of separate search databases. These benefits arrived together with the new storage layout; they do not isolate the effect of any one indexing algorithm. The clear structural improvement was that inactive projects no longer had to share large partitions with active ones.

Eskildsen begins with turbopuffer’s fundamental tradeoff: writes go directly to object storage, using S3 in his example. Turbopuffer does not first run its own disk-replication or Paxos path. Writes therefore take hundreds of milliseconds. He uses inventory reservations for a Kylie Jenner flash sale on Shopify as an example of a workload that would be a poor fit. Search ingestion can often accept that write delay if reads remain responsive.

A write enters a write-ahead log in object storage. Background workers build vector, text and columnar indexes to answer subsequent queries. Eskildsen pictures the log as successive files—one.json, two.json, three.json—while explicitly saying that JSON is an illustration. The useful model is a stored sequence of writes from which searchable structures are built.

Route toward cached data, then descend the hierarchy

A namespace acts much like a table; Eskildsen compares it to an isolated directory on S3. Any read-replica node can serve its queries, but routing favors the node most likely to have its data cached. The node checks memory, then an NVMe SSD cache, then object storage. This affinity encourages reuse without making one node the sole owner capable of answering.

For cold data, the costly part is waiting through dependent storage requests. Eskildsen cites an S3 P99 of around 200 milliseconds for a one-megabyte blob and describes roughly three round trips as an ideal target. If each fetch must finish before the next can begin, those waits accumulate. Turbopuffer instead tries to do as much work as possible concurrently within a few rounds. The same design helps use modern disks efficiently.

That requirement reaches into the indexes and query planner: they must make useful progress with the data fetched in each round. Eskildsen’s architecture article develops the same tradeoff between downloading more data per round and making additional requests. Object storage supplies cheap capacity; the search engine has to organize its work around the cost of reaching it.

The namespace also gives turbopuffer a natural place to apply customer-specific storage and encryption settings:

• Separate keys: Each namespace can use a different encryption key, and data can be re-encrypted with another key.

• Separate buckets: Namespaces can live in different buckets, including buckets in customers’ own cloud accounts. Eskildsen describes customers using thousands of buckets.

• Shared storage where permitted: Namespaces can also share a bucket. The engine can change storage arrangements at namespace granularity rather than requiring a whole search deployment for each collection.

Legora needed its namespaces separately encrypted at rest. S3, GCS and Azure Blob Storage could meet the storage requirement, but the NVMe SSD cache created a disagreement. Turbopuffer treated that cache as volatile, like memory. Legora’s customers treated the SSD contents as data stored at rest. The word cache did not exempt those bytes from their encryption requirements.

Disable first, then measure

The team expected to implement encryption in the disk cache. Before doing that work, it disabled the cache and measured performance. Memory caching plus object storage performed well enough that some Legora workloads kept the SSD cache disabled. This was a useful simplification for those workloads; it does not establish that SSD caching is unnecessary in general. Encrypted disk-cache support remained future work in Eskildsen’s account.

Returning to project-search results, Lauritzen reports approximately an order-of-magnitude improvement in median latency and a larger improvement at P99. He gives no resulting absolute latency values in this explanation. The gains matter across an agent’s work: an agent making 20 or 100 searches repeatedly encounters retrieval delay, so search performance affects how much investigation it can complete within a useful response time.

The newer legal-research corpus was growing toward ten billion vectors. Lauritzen presents that as a direction of growth, rather than an already reached total. The workload also creates spikes in queries per second: one user question fans out into several searches, and the investigation keeps generating more. A large corpus and a high query rate place different demands on the engine.

The searches expand because retrieving a similar passage does not settle whether it applies. Lauritzen identifies three relationships that require filtering and further retrieval:

• Jurisdictional hierarchy: Sources can sit at city, county, state or federal levels. Retrieval must respect the relevant hierarchy of authority.

• Temporal validity: A decision may have been overruled. Finding the decision creates a need to determine whether it remains valid.

• Exceptions and special cases: A newer regulation may create an exemption to an older one. Finding either source can require finding the other to understand the rule that applies.

These relationships make the research problem graph-like: a useful result points toward other material that must be checked. Lauritzen explains why the search expands, but does not detail an algorithm for determining legal authority or resolving every temporal relationship. The storage engine supports the retrieval work; the application still has to decide which searches and filters the legal question requires.

Jurisdictions become namespaces

Legal research also started on Elasticsearch, and Legora was moving it to turbopuffer because keeping the whole corpus in Elasticsearch had become expensive. Here the namespace follows a jurisdiction. Lauritzen uses frequently queried EU law and less frequently queried Danish law to illustrate the hot and cold ends of the workload. The distinction describes access frequency in his example.

An infrequently queried jurisdiction can remain in blob storage until needed. Lauritzen says a 500-millisecond cold fetch is acceptable for this deep-research workload. That tolerance makes the long tail economical: keep busy collections closer to compute, and pay an occasional fetch delay for the many collections that rarely receive attention. The cost advantage depends on being able to accept those cold reads.

Eskildsen describes turbopuffer as puffing data into and out of the memory hierarchy. Frequently used data can justify DRAM, SSDs offer another balance of capacity and latency, and cold data stays in object storage. The goal is to keep as much data as possible in cheaper tiers while still meeting performance needs. Each tier changes the economics of random access, sequential access and waiting for a round trip.

Graph navigation creates dependent waits

For vector search, he contrasts navigating a graph with organizing vectors into clusters. In the graph example, the engine visits a node to discover where to go next. If each step requires an uncached object-storage request, the query can repeatedly encounter the roughly 200-millisecond S3 P99 he cited earlier. Shrinking the graph’s diameter helps reduce steps, but the access pattern still creates dependencies between fetches. A layout effective in memory can become expensive farther down the storage hierarchy.

Turbopuffer instead groups nearby vectors into clusters, groups those clusters into larger clusters, and repeats this to form a tree. Eskildsen likens it to a complicated B-tree over the geometry of the vector space. The comparison describes the hierarchy: approximate geometric clustering lets a query narrow down which parts of the collection to inspect.

The tree separates two kinds of data with different access frequencies:

• Upper-level centroids: Searches repeatedly consult these cluster representatives to choose a path through the index. Their frequent use makes them good candidates for DRAM.

• Leaf data: The much larger collection of underlying records need not all occupy DRAM. Eskildsen describes leaves on SSDs with an illustrative one-millisecond final round trip. Colder objects can remain in object storage.

The index’s shape therefore helps determine what stays cached. Compact routing information is useful across many searches, while any particular leaf is needed less often. Keeping them in different tiers saves fast-memory capacity without forcing every query to discover its route through a succession of cold reads.

Eskildsen calls this the cheapest way to run a database and extends the argument to customers indexing large portions of the web. That is his architectural judgment, rather than a universal result established by a comparative benchmark here. The mechanism is more specific and useful: even within one searchable collection, routing structures and underlying records have different access patterns and can justify different storage tiers.

Eskildsen closes the storage explanation with a simplified model of full-text search: a map from each token to the document IDs containing it. For New York population, the engine looks up three entries and combines their document-ID sets. It also ranks candidates. A rarer term such as York can carry more ranking information than a common term such as New; term rarity is the aspect of BM25 that he uses to explain the scoring.

Two costs: storage waits and local list processing

The engine first fetches the relevant portions of the term dictionary, possibly after an additional lookup to locate them. It then fetches the document-ID lists, also called posting lists. The stages limit how much work can happen before the previous information arrives. Once those lists reach the node, combining and scoring them creates another cost: reading and processing large amounts of data through memory.

• Compress the lists: Smaller representations reduce the amount of posting-list data fetched from storage.

• Reduce dependent requests: Organizing dictionary and list access into a few rounds limits remote-storage waits.

• Avoid weak candidates: Once the engine has enough high-scoring documents matching population and York, documents matching only New may no longer affect the result. Scoring can therefore help avoid unnecessary list processing.

The last example gives the intuition for pruning work, without specifying the algorithm used to prove that a candidate cannot enter the results. It also makes Eskildsen’s closing comparison less surprising: he says full-text search at web scale is more difficult and computationally expensive than vector search. A familiar keyword query can involve enormous document-ID lists, storage transfers and memory-bandwidth demands.

The product payoff depends on the workload fit

Lauritzen returns to why this matters for Legora. Retrieval is central to legal reasoning, but running a separate Elasticsearch database for every enterprise tenant would create substantial operational work. Namespace-level storage and encryption let the team accommodate residency and customer-key requirements without multiplying independent search deployments in the same way.

The cost advantage follows a particular shape: many independent collections, a few active ones, and a long tail of cold indexes whose occasional fetch delay is acceptable. Slower ingestion is also a reasonable tradeoff for these search workloads. Projects and jurisdictions give the engine useful units to cache, store and encrypt independently.

Lauritzen ends with the engineering consequence: the team can spend more time improving Legora and less time working on scalability and infrastructure. Its CFO is pleased with the cost, too. Making storage follow how legal work actually arrives and goes idle gave the team room to focus on the application.