Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora
Read the talk
Making the project the unit of legal search
Legora’s search migration shows how storage layout, encryption requirements and cache behavior meet—and why an object-storage search engine needs indexes designed around round trips.
From a talk by Jacob Lauritzen and Simon Eskildsen
At a glance
Ideas worth remembering
Partition count alone does not preserve a useful working set. Legora’s roughly 4,000 partitions mixed active and inactive projects; one namespace per project let cold collections remain outside the cache.
Object-storage search exchanges slower writes and occasional cold reads for cheaper storage. Its indexes must reduce dependent round trips and use concurrency within each round.
Encryption requirements extend to caches. Disabling the SSD cache performed well enough for some Legora workloads, allowing the team to meet customer requirements without first implementing encrypted disk caching.
Legal research generates repeated searches because retrieval must follow jurisdictional hierarchy, temporal changes and regulatory exceptions. Lower latency matters across the agent’s whole sequence of queries.
The memory hierarchy shapes both vector and text indexes: frequently used vector-routing information can stay in DRAM, while text search must compress and efficiently process large document-ID lists.
Two kinds of legal search
Legora serves law firms and in-house legal teams reviewing contracts, drafting new ones and researching law. Jacob Lauritzen, an engineer at Legora, begins by separating the platform’s retrieval work into two workloads: searching documents inside a project, and researching laws, cases and regulations across a much larger corpus. Simon Eskildsen, turbopuffer’s CEO and co-founder, joins him to explain the storage architecture behind that search.
A project is a unit of legal work. Lauritzen uses a hypothetical acquisition of Cursor by SpaceX: a law firm would upload employment agreements and supplier contracts, then search within that collection. A project might contain tens of documents or millions. The important property is its scope: the lawyer’s query concerns the documents belonging to this particular matter.
Legal research has a different shape. A question can require searching laws, previous cases and regulations, then following related material to support an answer or a litigation argument. Project search starts with a known collection. Legal research must also work out which sources matter and how they relate. That difference eventually changes both the unit of storage and the number of searches generated by one user request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A simple cluster becomes several systems
Legora’s project-search journey runs from hundreds of thousands of documents to two billion. The initial design was straightforward: shared blob storage held raw documents, and one Elasticsearch cluster handled indexing and search for all tenants. Lauritzen says this worked reasonably well at first. There was one search system to operate, with no need to reproduce the deployment for each customer.
Data-residency requirements forced the first split. Legora’s customers wanted processing to remain in their respective regions: the US, the EU or Australia. The team reproduced the setup across the EU, US and Asia Pacific. This satisfied the regional requirement, but multiplied the infrastructure and operational overhead.
Large banks and law firms then asked for physical isolation and customer-managed encryption keys. Lauritzen acknowledges that physical isolation can mean different things to different customers; in this part of the migration, it effectively meant having their own database. Customer-managed keys added another form of control. The customer grants Legora access to a key used to encrypt and decrypt data at rest, and can revoke that access to prevent further decryption through the key. The mechanism concerns encrypted stored data; it is not a claim that revocation erases every copy or every previously decrypted value.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why 4,000 partitions still thrashed the cache
Legora moved search into Postgres because it already used Postgres for its transactional workloads. The team already needed separate databases and blob storage to meet customer requirements, so adding search to those databases offered a practical simplification: fewer kinds of systems to operate. Vector search used pgvector, while text search used tsvector rather than BM25. Lauritzen reports a loss in retrieval quality from that text-search change, alongside costs and search performance that were already less satisfactory than the team wanted.
The document-chunk table was split into roughly 4,000 partitions. Legora hashed project keys and packed projects into those partitions. This distributed projects across storage, but it did not preserve their access patterns. A legal project often receives intense attention for a while, closes, and is rarely opened again. Other projects remain active and receive frequent queries. Hashing allowed these cold and hot projects to occupy the same partition.
As the corpus grew, the mixed partitions became large. Lauritzen describes queries bringing partition data into memory, followed by queries against other partitions that displaced it. The useful working set kept competing with data from inactive projects, and the cache repeatedly lost what the next query needed. Search and ingestion P99 latency rose from about 100 milliseconds to 20 seconds. The failure was tied to this storage layout and workload: adding many partitions had not made the project itself an independently cacheable collection.
Legora switched to turbopuffer at approximately 400 million documents, a figure Lauritzen gives as an estimate. The decisive layout change was one namespace per project. An inactive project could remain in blob storage without sharing a large partition with an active one. Lauritzen also reports better relevance after restoring BM25, better latency, lower cost and simpler operation through a single turbopuffer cluster. Several changes arrived together, so the migration’s benefits cannot all be attributed to one indexing algorithm.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Write to object storage, then make reads economical
Eskildsen explains turbopuffer’s central tradeoff: writes go directly to object storage, specifically S3 in his example, rather than through turbopuffer’s own disk-replication or Paxos path. That means accepting write latency measured in hundreds of milliseconds. He contrasts search ingestion with inventory reservations during a flash sale. The latter needs a fast transactional response; a search workload can often tolerate slower writes if reads remain useful and responsive.
A write enters a write-ahead log in object storage. Background work builds vector, text and columnar indexes that support later queries. Eskildsen illustrates the log as successive files named one.json, two.json and three.json, while explicitly noting that JSON is only an analogy. The mechanism is a stored sequence of writes followed by index construction; the example does not specify the database’s actual file format.
A namespace plays a role similar to a table, and Eskildsen suggests thinking of it as an isolated directory on S3. A query can run on any read-replica node, but routing favors the node most likely to have the namespace cached. That node looks first in memory, then in an NVMe SSD cache, and finally in object storage. Routing affinity helps reuse cached data without making one particular node the only place capable of answering.
The expensive part of a cold read is the sequence of storage requests. Eskildsen cites an S3 P99 of around 200 milliseconds for a one-megabyte blob and describes roughly three round trips as an ideal target. The database therefore tries to perform a lot of work concurrently within a few rounds, rather than making many dependent fetches. He says the same approach also suits modern disks: concurrency can use their capacity well while reducing the number of times a query must wait before it can continue.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The namespace also becomes the unit of encryption
Making the namespace the unit of separation also makes customer-specific storage arrangements possible. Each namespace can use a different encryption key or live in a different bucket. Buckets can be shared where appropriate, or placed in a customer’s cloud account. Eskildsen describes customers using thousands of buckets, and namespaces that can be moved or re-encrypted. These controls let storage and encryption follow the customer’s requirements without reproducing an entire search database for every collection.
Legora needed namespaces to remain separately encrypted at rest. Object storage could meet that requirement, but the NVMe SSD cache exposed a disagreement about what counted as stored data. Turbopuffer treated the cache as volatile, like memory. Legora’s customers did not. Calling an SSD a cache did not remove the customers’ requirement to protect the data written there.
The team initially expected to implement encryption in the disk cache. Instead, it first disabled that cache and measured the result. Performance with memory caching and object storage remained good enough that some Legora workloads continued with the SSD cache disabled. The result was specific to those workloads, rather than evidence that disk caching is generally unnecessary. Encrypted disk-cache support was future work in Eskildsen’s account.
Lauritzen reports an approximately order-of-magnitude improvement in median latency after the migration, and says P99 improved even more. He does not state the resulting absolute latency values. The effect matters beyond one retrieval call: an agent may perform 20 or 100 searches, so repeated search delays can accumulate across its work. Faster retrieval changes how much searching the application can afford to do within a useful response time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Legal research makes one question become many searches
The newer legal-research workload was growing toward ten billion vectors; Lauritzen presents that as a direction of growth, rather than an already reached corpus size. Query load also spikes because one research request fans out into multiple searches and continues generating more. Corpus size and queries per second are separate pressures: a large collection must remain searchable, while each user request can create substantial retrieval work.
The fan-out serves legal relationships that similarity alone does not settle. Lauritzen names jurisdictional hierarchy, temporal validity and exceptions between regulations. A result may belong to a city, county, state or federal level; a decision may have been overruled; a newer regulation may create an exemption to an older one. Finding one relevant source can therefore create a need to find additional sources and apply filters. This is a description of the retrieval problem, not a complete account of how Legora determines legal authority or validity.
Legal research also began on Elasticsearch, and Legora was moving it to turbopuffer because maintaining the whole corpus in Elasticsearch had become expensive. Here, jurisdictions become namespaces. Lauritzen uses EU law as an example of a frequently queried collection and Danish law as an example of a less frequently queried one. That contrast concerns access frequency in the workload, not the legal importance of either jurisdiction.
A rarely used jurisdiction can remain in blob storage until requested. Lauritzen says a 500-millisecond fetch for that cold data is acceptable in a deep-research workload. The economics depend on that tolerance: Legora can keep a long tail of infrequently queried collections in cheaper storage while caching the busy ones. The same delay could matter much more in an application that requires every individual search to return immediately.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why clustered vectors suit the storage hierarchy
Eskildsen describes the engine as moving data through a memory hierarchy. Frequently accessed data can justify DRAM; SSDs provide another latency and cost point; cold data can stay in object storage. Turbopuffer tries to keep as much data as possible in the cheaper levels while meeting performance needs. Doing that requires more than a cache policy. The indexes themselves must work with the round-trip costs and access patterns of those storage levels.
For vector search, he contrasts graph navigation with hierarchical clustering. A graph search proceeds through connected nodes, and later steps can depend on what an earlier step discovers. If those steps require uncached object-storage reads, repeated waits become expensive. Reducing graph diameter can help, but the underlying problem remains: an access pattern that works well in memory can become costly when navigation repeatedly waits on remote storage.
Turbopuffer groups nearby vectors into clusters, then groups those clusters into larger clusters, forming a tree. Eskildsen compares it to a complicated B-tree over the geometry of the vector space. The comparison explains the hierarchical shape; the clustering is approximate, and he does not present it as ordinary ordered-key B-tree search. The tree helps a query narrow down which parts of the vector collection to inspect.
Upper-level cluster centroids participate in searches repeatedly, so they tend to remain in DRAM. The much larger collection at the leaves can sit on SSDs, where Eskildsen gives a one-millisecond final round trip as an illustrative access cost. This places the compact, frequently consulted routing information in fast memory without requiring the entire searchable collection to live there. Cold data can remain further down the hierarchy.
Eskildsen argues strongly for the cost efficiency of this arrangement, including for customers indexing large portions of the web. The mechanism supports the narrower lesson: access frequency can differ even within one index, so the routing structure and the underlying records need not occupy the same storage tier. His claim that this is the cheapest way to run a database is an architectural judgment; the talk does not provide a comparative cost benchmark that establishes it for every workload.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full-text search still has to move enormous lists
Eskildsen closes the storage explanation with full-text search. His simplified model is a map from a token to the document IDs containing it. For a query such as New York population, the engine looks up the three terms and combines their document-ID sets. It also scores results. A rarer term such as York can contribute more useful ranking information than a common term such as New; this is the aspect of BM25 he uses to explain relevance scoring.
The storage work comes in stages. The engine fetches the relevant portions of the term dictionary, possibly after an additional lookup to locate them, then fetches the document-ID lists. Compressing those lists reduces the data transferred. Combining and scoring them also consumes memory bandwidth, so a good implementation must limit both remote requests and the amount of list data processed locally.
Scoring can help avoid work on weak candidates. In Eskildsen’s example, once the engine has found enough high-scoring documents containing population and York, documents that only match New may no longer matter to the results. He presents this as the intuition behind reducing unnecessary processing, without specifying the pruning algorithm. It also explains his closing observation that full-text search at web scale can be more computationally expensive than vector search: familiar keyword queries can require fetching and processing enormous lists of document IDs.
Lauritzen’s final point returns to operating the product. Retrieval is central to Legora’s legal reasoning, but separately operating an Elasticsearch database for every enterprise tenant would create substantial overhead. Namespace-level storage and encryption let Legora meet residency and customer-key requirements without managing that many independent search deployments. The long tail of cold indexes remains economical because the workload accepts some additional latency when those indexes are needed.
The payoff is engineering time as well as search performance. Lauritzen says the team can spend more attention improving Legora and less on scaling infrastructure, while its CFO is pleased with the cost. That outcome rests on a particular fit: projects and jurisdictions form useful independent collections, many collections are cold, and slower ingestion or occasional cold reads are acceptable. Aligning storage with those properties made the system easier to run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:12
Go.
- 0:13
Okay. Hi everyone. Welcome, um, this 20-minute talk about connecting AI to loads of legal documents. My name's Jacob. I'm an engineer at Legora.
- 0:26
Yeah, and I gotta step into frame here. Uh, I'm Simon. I'm the CEO and, uh, CEO and co-founder of TurboPuffer, a search engine that we work with Legora and others on.
- 0:35
It, um, super quickly, um, introduction to Legora. We're a collaborative AI platform for legal work, and so that means we have law firms that are clients, and we have, uh, in-house legal teams that are clients, and they use Legora to do reviews of contracts. They use it to go through an absurd amount of contracts and make sure that they all look good. They use them to, uh, create new contracts. They do legal research, which means, um, looking over all, uh, potential law,
- 1:06
and they collaborate inside Legora. So you can think of Legora sort of as a Linear/Figma/Notion/GitHub for legal work. It's a lot. We, um, are one of the fastest growing companies, uh, right now. We grow extremely fast. Yeah, tons of numbers on the screen. I'll just skip through that. What we really wanna talk about is search today. So at Legora, there's two types of search that we do. There is project search and legal research. Project
- 1:36
search is, um, basically projects in Legora is, like, the, the unit of work that you have. So if, um, let's say you are SpaceX and you want to acquire Cursor, then that would be one project with your law firm. Uh, and so they would go into Legora, and they would upload all these documents, and the law firm that helped you would go through all of the employment agreements, all of the contracts with suppliers. Um, I know Cursor is using, uh, TurboPuffer, so maybe there's a contract there they'd look at. Um, but
- 2:05
basically you do the search confined to a project, and projects can be tens of documents to millions of documents. The other use case is legal research, and legal research is sort of a deep research-style workload where we'll search across tons of laws, previous cases, regulations, et cetera, et cetera, and people use this to answer questions such as, like, "How do I handle this specific thing?" And they'll also use it for litigation. Maybe they want to sue someone or maybe they are getting sued, and they'll use legal research to support and help their case.
- 2:36
So if we start at number one, project search. We've been through a little bit of a ride here on how we do search, um, starting at, you know, hundreds of thousands of documents all the way into two billions of documents, and we've tried a lot of different things. So first we start, started with a very, very simple one, which is just a single Elasticsearch cluster for all of our search workloads. Um, that worked relatively well. It was sort of a simple setup. All of the tenants, you know, our clients, our users,
- 3:07
would be on one big blob storage, where we store the raw documents, and on one big Elasticsearch, where we would do all of their searching, the indexing and the searching. Super simple. Worked relatively well initially. Then we wanted to enter the land of the free, and, uh, we got some new requirements. They... You know, Americans only want processing to happen within the US, and Europeans only want it to happen within the EU, and Australians only want it to happen within Australia. And so we had to s- basically, um... Well, here's a scaling
- 3:36
graph. We had to move to multiple Elasticsearches, and what we actually did was we took the entire setup, and we just basically iterated over the set that is EU, US, and Asia Pacific, and so we just had this, like, multiplied by three or four. Kind of annoying, lots of overhead, but it, uh, got us to where we needed to be.
- 3:57
Then the next iteration of the story is enterprise. So really big banks, the biggest law firms in the world, they have really annoying requirements. And number one they have is they'll ask for full physical isolation of all of their data. There's probably a little bit of a war on what physical you know, isolation actually means, but essentially it means they want their own database. They also want customer-managed encryption keys, and what that means is they basically have a key vault thing where they have an encryption key, and they give us
- 4:27
access to read the key, and we then use that key to, uh, encrypt and decrypt all of their data at rest. And what that gives them is they can just revoke our access to their key, and then we can't decrypt their data anymore, and so it's safe. And so in a way, that gives enterprises a lot of control over all of their data 'cause they control, you know, the key to, to reading it.
- 4:48
So we moved from Elasticsearch to Postgres, and I imagine a bunch of you guys are like, "Why would you ever put your vectors into Postgres?" Um, it actually works surprisingly well, and the reason that we did this was we were already using Pro- Postgres for OLTP workloads. And, uh, so we, we sort of already had to do this split of multiple Postgreses and multiple blobs. And so it was really easy for us to try to shift all of our search into Postgres as well 'cause then we only have one system. So the setup here was pgvector, uh,
- 5:18
specifically diskANN, tsvector for the search, so not BM25, which was, you know, we lost a little bit of, uh, retrieval performance there. And what we'd do is we would partition the table where we would store all the document chunks. We'd partition it super aggressively, like 4,000 partitions, and then each project, we'd basically hash the project key, and we'd bin pack them into the partitions. That actually worked relatively well, but it was expensive, and search performance wasn't super, super good. And what happened was when we scaled
- 5:48
a lot, uh, everything just broke and exploded. And so what happened was the... Basically, you can imagine, like, you have a bunch of projects, and some of them you spin up a project, you work on it, and then you close it, and you basically never go back to it again. And we have a bunch of those where, like, they never get queried, and we have a bunch that get queried all the time 'cause they're super active projects. And when we pack them into partitions, the cold ones and the hot ones would land on the same ones, and the partitions would get really, really big. And so when we queried them, they would... Postgres would pull the partition, put it into memory, we'd do the
- 6:17
stuff, and then we'd query another partition, and another partition, and it would essentially thrash the cache all the time. And what that meant was our latencies would spike. So we went from, like, search and ingestion P99 of a hundred milliseconds into twenty seconds, which you can imagine is a really bad user experience.
- 6:37
So then we went to Turbopuffer in about, you know, when we were about, I think, four hundred million documents, something like that.
- 6:45
And what we did with Turbopuffer was we did one namespace per project. And the advantages of moving to Turbopuffer is we got BM25, real BM25, much better, uh, relevancy, much better latencies, and it was extremely simple to operate 'cause we could just have a single Turbopuffer cluster. You know, we didn't have to have a bunch of different ones like with Postgres, and it would, since it's blob-based, it could just query the blobs that we had anyway. And so much lower cost, and it was extremely simple to operate,
- 7:16
and we didn't have this problem with the partitions 'cause if a project's not used, it's just in blob. And so it's really easy. And Simon can talk a bit more about why that works so well.
- 7:28
Yeah. So Legora has some of... And, and legal in general has... By the way, if, if, uh, Jake and I s- have similar accents and maybe even look a bit similar, it's because we're both Danish. Um, Turbopuffer has a very, has a particular architecture that supports these kinds of very regulated environments really, really well. But in order to understand that, we have to understand what kind of search engine is Turbopuffer. Why is it different than the ones that they used in the past? Since the very beginning of Turbopuffer, um, the design has more or less been the same. Uh,
- 7:58
there may be changes in the future, but the design has stood the test of time. When you do a write to Turbopuffer, we write directly to object storage. There is no, like, disk replication, there's no Paxos, there's, there's none of that. Direct to S3. That's the fundamental trade-off in Turbopuffer, right? Hundreds of milliseconds. If you're like Shopify and doing inventory, uh, reservations for a Kylie Jenner flash sale, not gonna work. Very, very good for search because generally when you're doing search, doing a slow
- 8:28
write is fine, um, as long as the read performance is, is adaptable and good. So that's what happens on write. It just goes in the write-ahead log. You can imagine you write one.json, two.json, three.json. Obviously, it's a da- it's, it's a database, so it's not JSON, but for illustrative purposes, that's what happens. And in the background, we build the vector indexes, the text indexes, the columnar indexes, and so on to satisfy the queries that, that Jake and other customers have. Um, so then at query time, we can go in and then, um, the query
- 8:58
reaches some namespace, and namespace is kind of our concept of a, of a table. Um, you can think of it as a directory on S3 that's isolated from everything else. We go to the node that is most likely to have it. It could go to any node, right? It could go to every single node, and they're all read replicas, but it would... we go with some affinity to the node that has the highest probability of having it in cache. We check the memory cache for any objects, NVMe SSD cache, and then finally to object storage. Everything in Turbopuffer is optimized around doing as much work in as few round trips as possible, right? S3 has a P99,
- 9:28
um, on a, like, one megabyte blob size of around, uh, two hundred milliseconds, so you wanna do as few round trips as possible, right? Ideally, you do around three. And everything in Turbopuffer, the database, is... Oh, I'm gonna need your fingerprint.
- 9:41
You got it.
- 9:42
Um, everything in, in Turbopuffer is designed around minimizing the number of round trips. This is also amazing for modern dri- disks. If you do a lot of concurrency in few round trips, you utilize them optimally, and everything in Turbopuffer is designed around this. So why is this so good for a company like Legora? Well, object storage native, if you design it around the atomic unit of separation being the namespace or the table, every single table could be encrypted with a different key. Every single namespace could be in a different bucket. We have
- 10:12
customers that have, um, thousands of buckets that they have namespaces in so that their customers get the warm IT fuzzies of having the bucket in their own cloud account. They can also be encrypted with their own keys. You can share buckets. You can, you can do whatever configuration that you need at the namespace level. You can re-encrypt with different keys, you can move them around, um, and you can re-encrypt with other keys. Um, for Legora in particular, this was really important for this full physical separation, right? An encryption
- 10:41
separation. All of the namespaces needed to be physically at rest with different keys and as separate as possible. S3, GCS, Azure Blob Storage, they pass that. Um, and the other parts of the hierarchy also. Except the NVMe SSD cache, because in the SSD cache, we consider that to be volatile like memory, um, but your customers did not. So in the, uh... So what we did was that we thought we were gonna implement encryption into the disk cache, but instead we just disabled the disk cache
- 11:11
and saw how it fared. And the performance of Turbopuffer, even without the disk cache with just the memory cache, was so good that we just kept it that way for some of the Legora workloads where we couldn't have the disk cache for multi-tenancy. Turbopuffer will support that in the future. But it just goes to show the, um, natural point where Turbopuffer allows these encryption and storage and separation to become fully multi-tenancy, uh, native. I'll hand it back to you on what happened then.
- 11:39
And then, drum roll please, latencies looked like this. Um, is my mic working?
- 11:47
No.
- 11:48
No? Could I... Or I'll start screaming really loudly. Um, it speaks for itself if you can't hear me. Okay. Um, latencies improved in order of magnitude basically, and these are median latencies, so P99 were even better. Um, so obviously this is a huge thing when you're doing-- I mean, one thing is if you're doing a single sort of RAG-style thing, but if you have an agent that does twenty queries, a hundred queries, these really, really add up. So that was on the project side, and then a more recent thing is legal research.
- 12:18
So legal research, um, is a kind of a difficult problem, and the reason it's difficult is that, um, the corpus is extremely big, so we're racing towards ten billion vectors, and we're growing extremely fast. We also have quite high read, um, so QPS can spike a lot 'cause we do a lot of fan-out. Like, if you do a, a, a sort of legal research query, we will fan it out into a bunch of different queries, and we'll keep going. And the reason we do that is we need this, um, heavy filtering 'cause essentially
- 12:48
it's a-- kinda like a graph for a few different reasons. Firstly, it's, um, hierarchical. You know, you have cities, and you have counties, and you have states, and you have federal law, and it's the same all around the world. Um, and so you need to respect that authoritative sort of hierarchy. There's also some temporal validity, so one judge might overrule a decision that's been made somewhere else, and you need to also respect that and figure that out. And then sometimes there's even, like, a new regulation that has exemptions or special cases of an old regulation, and
- 13:18
so if you're finding this one, you need to find all the other ones as well. So you can imagine that it, uh, sort of explodes the search.
- 13:26
And so we started on Elasticsearch for this, but also moving to TurboPuffer. Um, Elasticsearch just got extremely expensive 'cause we have to have everything there. But, um, with TurboPuffer, we can basically, uh, take different jurisdictions, and we can make them namespaces in TurboPuffer, and that means some of them... Here's an example where, like, you have the EU. That gets queried all the time. That's super hot. And some of them, let's say Danish law 'cause we're Danish, no one cares really. It's such a small country, so, like, it doesn't really get queried, and so that can just stay on Blob, and that's fine.
- 13:56
Um, and because it's sort of a deep research-style workload, if there's five hundred milliseconds latency to fetch that cold Blob, that's okay. That's fine. It's not really a big problem. So the way that TurboPuffer is, is designed lends itself super well to this super long scale of, like, cold, weird namespaces and a few that are really, really hot. Um, yeah, and Simon wants to talk more about that.
- 14:20
Yeah. So, um, um, I was, I was talking about why the company is called TurboPuffer at another, uh, talk here earlier today. But, uh, one of the other explanations of the name of TurboPuffer is that it's about puffing into the different memory hierarchies and really mastering when data should be in particular memory hierarchies. So you can think about it here, right, of something like the EU law might be more or less part of almost every one of the legal research, uh, queries, right? So that probably sits closer
- 14:50
to NVMe SSDs than memory, right? The economics kinda change as you move up and down this hierarchy. Um, in, in memory, you want things that are queried a lot, right? Then the economics of memory are great. NVMe SSDs can-- you can do a lot of things directly on them, but the economics change as you move up and down this boundary. The latency changes, and the way that the database is architected to take advantage of it in terms of round trips versus random versus sequential all changes as you navigate this hierarchy. TurboPuffer is a database that is really designed around the memory
- 15:20
hierarchy, and all of the smarts in TurboPuffer is that all of these namespaces are puffed in and out, um, of the cache. You can think of this as we wanna spend as much time, have as much data pushed as far down in this hierarchy as possible to get the, the best, um, performance cost ratios. So how does that apply to search? Well, for something like vector search, for example, there's two fundamental ways to do vector search. One is to navigate it, basically design a graph. The problem with a graph on something like object storage or disk, again, we wanna
- 15:49
have things as far down that memory hierarchy as possible. The problem with a graph, this is not a graph, this is a tree, um, but in a graph, you have to navigate from the center of the graph. So-- And then every time you navigate through these nodes, you're doing two hundred millisecond P99 to S3, right? And so you're trying to shrink the diameter of the graph. You're trying to do all these tricks to make the graph. But fundamentally, you're at odds with the fact that a graph is about a random sequential trade-off that you have in memory and in registers but not further down the memory hierarchy.
- 16:20
The way TurboPuffer does it is organize it into clusters, right? Vectors you can think of in two dimensions just as point is a massive coordinate system, and we can organize them into clusters. TurboPuffer then creates clusters of clusters and clusters of clusters of clusters to essentially organize all of the vector data in a tree. You can basically think of TurboPuffer as a very, very complicated B-tree, right? Because it's a tree on this geometry of this entire space and the clustering of it in an approximate way. Now, the root centroids further up
- 16:50
the tree, you can imagine, are part of every single time you search, right? They're-- We're always trying to figure out which clusters that we're in, and we're always looking at the upper levels of the tree. So they're gonna be further up the memory hierarchy, right? Closer to the registers, m-almost all in DRAM. Now, the leaves that have all of the actual legal cases of whatever long document it could be, it could be images, all of that is probably gonna be on SSDs with that single one-millisecond round trip at the end. It doesn't make sense to have all that puffed into DRAM. This is fundamentally the cheapest
- 17:19
way that you can run a database, period. So for something like Legora or even web search, which is in the hundreds of billion or tens of billions, depending on how much of the web you've scraped, this is fundamentally the cheapest way to do it. And we have customers that are indexing massive parts of the entire web into TurboPuffer, which is really also a part of what legal research is. Full text is also, also really respectful of the memory hierarchies. The way that text search works is essentially you can think of it as a HashMap. You have a
- 17:50
big document, and then you take every single one of the tokens, and you put them into the key in the HashMap. The value in the HashMap is some set with all of the document IDs that has that term. So then if you search for New York population, you're finding those three places in the HashMap, and then you're taking the three sets and doing an intersects on the- intersect on the sets. While you're intersecting, you're also trying to do some kind of scoring, right? A document that has York in it is probably more valuable than a document that has New in it because York is a more
- 18:19
rare word. When people say BM25, this is the scoring that they're referring to. The art of full-text search is, one, we wanna minimize the number of round trips. So first you download the parts of the dictionary that are relevant, round trip one, maybe a round trip one before that to index into where the parts of the term- terms are. And then the second round trip is to get these massive lists. Try to make the lists as small as possible by compressing them. But also while you're doing the text search, you're trying to minimize the amount, again, of memory bandwidth that you
- 18:49
want to intersect these lists. You can probably imagine that at some point, there's a point where you've seen so many documents with population and York that have much higher scores that documents that just have New in it are irrelevant anymore. This is like a mega crash course in how text search works. And counterintuitively to most people, text search at web scale is more difficult and more computationally expensive than doing vector search. I'll hand it over to you.
- 19:16
Cool. So key learnings from, um, what you heard today. Retrieval is extremely important to Legora. Um, it's key to legal reasoning. TurboPuffer really excels when, uh... for us 'cause, uh, it makes it extremely easy to operate. We have seventy-plus tenants. We have a hundred. We have two hundred tenants. You know, if we had to have separate Elasticsearch databases for each of these, it would be just hell. Um, but we can do this natively with TurboPuffer with, uh, data residency and CMAC, et cetera, et cetera.
- 19:47
And then it's extremely cost efficient generally when you have these types of workflows, um, or workloads that we do where there's a long tail of cold indices basically that you don't need to query so much, and you don't-- you, you're sort of... You're okay paying the small, uh, latency cost for it. So now with, um, TurboPuffer and four seconds to go, now we can focus on making Legora. We can focus on the product, making it really, really great, not on scalability and infra. And also David, our CFO, is really happy about the cost, so it's great. Thanks, everyone.