[
  {
    "startMs": 12760,
    "text": "All right. Hello, everyone. Thank you all for coming. I'm Nikhil, CTO of TurboPuffer."
  },
  {
    "startMs": 18360,
    "text": "And I'm Aaron Linsky, the head of product engineering at Bridgewater Associates."
  },
  {
    "startMs": 23280,
    "text": "And we're here to talk to you today about how you build an AI analyst that can keep a secret. And I'll explain a bit more about what that means here. Now, this is a very oversimplified view of how you would go about building an AI analyst, but it is meant to capture an important dimension around permission. So in this diagram, you're a user interacting with this AI agent. You start with a question, maybe something like, \"Why have token prices come down so much in the last month?\" And you feed that question"
  },
  {
    "startMs": 53160,
    "text": "into an LLM, which will begin thinking about that question, investigating, researching. And as it does that investigation, it's going to search for facts to ground its information, to lead its investigation. Some of those facts might come from the public web, but other facts might come from private documents in your organization. Now, the public web is, on one dimension, much easier to search than the information that's private to your organization. By definition, the public web is public. Everyone in your organization can see"
  },
  {
    "startMs": 82960,
    "text": "exactly the same documents. So there's no permission issue there. Every public web document can be immediately fed into the AI analyst, no matter who's asking the question. But the information internal to your organization looks very different. Every email, every meeting note, every report might have a completely different set of permissions attached to it, belonging to different groups, maybe individual permissions assignments on those docs. And you can see here, we've got different colors here reflecting the fact that every document in that private dataset has a different set of permissions."
  },
  {
    "startMs": 113080,
    "text": "And this is a classically hard problem in information retrieval, and it's something we explicitly designed TurboPuffer to solve. But before I get into that, I'm gonna hand it over to Aaron to talk about why this private information is so important to Bridgewater's AI analyst."
  },
  {
    "startMs": 127760,
    "text": "Great. Thank you. Oh, there we go. So first, I'm gonna go talk a little bit about wh-what-- who Bridgewater is, what we do, and then we get into how, uh, how we're using TurboPuffer and, and the needs there. So for the past fifty years, we've, at Bridgewater, have pursued the best understanding of how markets and economies work. We then encode that systematically with logic into a system that we can then trust test over"
  },
  {
    "startMs": 157620,
    "text": "time. What you see here are on the, on the left with the yellow legal pad, that is our bond system from nineteen eighty. It's a set of if-then statements that we used to trade bonds back then. Over the next few decades, we've evolved that process from a yellow legal pad to Excel to eventually Scala and now Python, um, to, uh, to a different learning process with humans at the center."
  },
  {
    "startMs": 188180,
    "text": "This five-- f- this research circle is what we call it, has five steps. Perception, that's seeing what's going on in the outside world. Hypothesis, which is coming up with ideas of why that might be true. Investigation, which is the research of seeing is that thing true. If it is true, we then systemize and integrate it with our system and industrialize and run that every day and trade on that. That's been our learning process. I've been at Bridgewater for about twenty years. That's been what we've been do-doing for twenty years with humans at the center."
  },
  {
    "startMs": 218460,
    "text": "We're now moving towards putting machines at the center and running that process. In order for even, you know, for both humans and machines, in order for that to work well, we need access to high-quality data. We need both time series data, which we've spent decades, uh, decades creating a great bitemporal macro time series database, which we call DeLorean. And more recently, we've been creating a"
  },
  {
    "startMs": 248140,
    "text": "unstructured database, which we call the Vault. And so what is the Vault? Um, today it's about a hundred million documents, which we index, organize, and secure that we think represent some of the best third-party and external-- third-party external research and also our internal research. So we've sourced with a hundred and thirty-five different external sources, which is a huge deal for both legal and our data team to actually nego-negotiate and get them in."
  },
  {
    "startMs": 278600,
    "text": "And all of, all of those sources have, um, some sorts of permissions around it where it might not be open to everyone at Bridgewater. And then, of course, we also have our interm- internal sources of unstructured data. That's our own writing, our meeting recordings, and, and our internal reports as well. And both-- That those all go into the Vault, and then both humans, investors, and our AI agents read out of the Vault. What I haven't shown here is actually it circles back. So the investors and, uh, a- and the agents will"
  },
  {
    "startMs": 308360,
    "text": "create new, new content that goes back into the Vault. So it's constantly improving. And so here's a very high-level view of what permissions in the Vault look like. So I have a user, w- we'll just say their ID is user one. They're in groups B and C. This is incr- incredibly simplified, unfortunately. Um, and there's a set of all of our documents where it could be either, um, uh, permission to them"
  },
  {
    "startMs": 338260,
    "text": "via groups or directly. And then here we have the actual code of us going ahead and creating a query to try to look for some data with whatever that, uh, query vector was that matches the viewers and groups. And so this is simple in one way, and actually when we first implemented this without TurboPuffer- We thought it worked fine. Um, but then"
  },
  {
    "startMs": 368232,
    "text": "pretty quickly as we started to add more and more documents to the vault, it fell over, um, tremendously. And so what I did then was I looked for other possible providers. We found TurboPuffer, tested them out, and they had a solution which Nikhil is gonna go over right now."
  },
  {
    "startMs": 385712,
    "text": "Awesome. Thank you, Aaron."
  },
  {
    "startMs": 389692,
    "text": "Let me very quickly talk about what TurboPuffer is. We are an extremely fast, extremely scalable, extremely cheap search engine built entirely on top of object storage. And what that means is we write directly to S3 or GCS or Azure Blob Storage, whatever your hyperscaler provides, and that gives us S3 economics. So by default, your data is inert, cold on S3. Only when you start querying it do we pull it into more expensive storage mediums like SSD or"
  },
  {
    "startMs": 419452,
    "text": "memory. And this is really powerful because once a query is hot, once the data a query is accessing is in the cache, it's fast. But if you're not accessing a particular namespace or a particular region of a namespace, that data can just hang out on object storage where you pay rock bottom prices for it. So this architecture allows TurboPuffer to scale cheaply to terabytes of data, which would cost tens of millions of dollars in, in other systems, and we can often do it at a tenth of the cost at performance"
  },
  {
    "startMs": 449432,
    "text": "that's comparable once you get the data hot, uh, to what you get with traditional disk-based search systems. And we support all types of search. We support keyword search, we support regex search, and what Aaron was showing earlier, vector search. Um, so we're really not, uh, opinionated about how you search your data in TurboPuffer. We wanna do search generically. But for the rest of this talk, I'm going to focus on vector search because, uh, it's, it's good to zoom in and go deep on"
  },
  {
    "startMs": 479472,
    "text": "one particular type of search. Let's do a little bit of napkin math. That's something we love to do at TurboPuffer. It helps us anchor in the simplest possible solution. We never want to introduce complexity to TurboPuffer that's not justified. One of the justifications for taking on complexity is if you need it for performance. The absolute simplest way to do search is to put everything in a list, and every time somebody shows up saying, \"I want queries that match this. I want documents that match this query,\" you look at every single piece"
  },
  {
    "startMs": 509452,
    "text": "of data. Let's conservatively say that every document that you store in TurboPuffer is one kilobyte. This is incredibly conservative. A vector embedding is often 1,024 dimensions, so if you're using just a byte per dimension, that's a kilobyte right there. Uh, in practice, documents are often larger than this. But for the sake of example, let's say it's a kilobyte per doc. Even if you had all of that data resident in memory and you were ripping through it as fast as you possibly could, which on modern hardware is about twenty gigabytes a second,"
  },
  {
    "startMs": 540192,
    "text": "once you get up to a hundred million docs, that's gonna take five seconds. And for most search workloads, that's way too slow. What we see in practice is most people want their search results to be about a hundred milliseconds. So the naive solution here does not work. If you've only got ten K docs, great. That's gonna take half a millisecond. TurboPuffer can do literally anything and it will be fast enough. But as you get to scale, once you're at ten million or a hundred million, you really need to start thinking about how you represent your data so that you're not crunching through all of that. And of course, I gave you the n-"
  },
  {
    "startMs": 570172,
    "text": "the numbers from memory here. That's only if the cache is hot. If that data's on S3, you're looking at a minute or more to pull that data off S3. So this just isn't viable at the scale that Bridgewater operates."
  },
  {
    "startMs": 583312,
    "text": "So step one in TurboPuffer is to build a vector index, and this is something that we handle under the hood for you. When you store vectors in TurboPuffer, we automatically build this index behind the scenes once you get enough data to, to justify it. Vector embeddings, if you're not familiar, are the internal numeric representation of concepts in LLM space. And we like to use this example of red shoes and maroon sandals. You as a human know that these are very similar concepts and much more similar"
  },
  {
    "startMs": 613272,
    "text": "to one another than concrete block is. Maybe you're thinking about shape and size and texture and, and weight. The LLM can translate that into this numeric vector. Here I've only got two dimensions shown so that we can think two-dimensionally, but in practice there's often thousands of dimension. 1,024 dimensions is a very common size. Thinking multidimensionally like that is very hard, so for the sake of example, we're just going to imagine that there are two dimensions here. Now we don't actually know what those dimensions represent, but"
  },
  {
    "startMs": 643212,
    "text": "again, you can think of it as like size and color. And then you can plot that on a coordinate system. So red shoes might be at the top left along with maroon sandals, and then concrete block is gonna be down at the bottom right. And I'm gonna simplify wildly here, but a very, uh, workable vector index in two dimensions is just to plot things on a graph, and then you can zoom in. If you're looking for red shoes, that's the Q on this diagram, you plot that on this two-dimensional graph and"
  },
  {
    "startMs": 673192,
    "text": "you zoom in and you look for the vectors around it. And because of the way these vectors are created by the LLM, vectors that are near each other represent similar concepts, represent good search results to return. So if we were just doing similarity search, great. We look in our two-dimensional coordinate grid, we find some nearby vectors, we return them, and life is good. And this saves us a lot of time. We don't have to search the whole vector space. We can zoom right in to the documents that are gonna be relevant to the query."
  },
  {
    "startMs": 703072,
    "text": "So that napkin math from earlier we can, we can ignore. We're maybe only looking at ten thousand vectors here, and we know we can do that in a millisecond or two. The problem is when you add in these permissions. Not all of the documents that are most relevant for a given query are going to match those permissions. And you can see in this diagram here the minuses don't match the permissions while the pluses do match the permissions. So our vector index is not immediately helpful to us, and we've invested a lot of"
  },
  {
    "startMs": 733052,
    "text": "time and energy at TurboPuffer into solving this problem."
  },
  {
    "startMs": 739472,
    "text": "Naively, you might try a post-filtering strategy, and the way that works is the query comes in, you get maybe the 10,000 closest vectors, and then you throw away all the ones that don't match. This does not work very well because it doesn't provide the results that you're looking for. If I'm the user that only has access to that data on the bottom right and I search for Q, I still wanna see the documents that"
  },
  {
    "startMs": 769332,
    "text": "were closest to Q from my perspective, that include the results that I can see. But when you do post-filtering, you kinda spiral outwards, and then at some point you have to give up because it's been a couple seconds and you haven't found any docs that match, or you haven't found the top 100 docs that match. Maybe you were only able to find 40. And this makes users unhappy because they might know that a doc exists with maroon sandals, and they were searching for red shoes, and you don't turn it up. So you have very poor recall."
  },
  {
    "startMs": 800212,
    "text": "So instead of starting with the vector index, what if we start with the filters? TurboPuffer builds an index of all of the documents that match the filters. So Aaron can show up and say, \"Hey, show me all of the documents that I have access to.\" And then TurboPuffer can scan over all of those documents and see which of them are closest to the query vector. But what if Aaron has access to 50 million out of the 100 million documents in the vault? Now we're back to square one. We're"
  },
  {
    "startMs": 830232,
    "text": "back in that napkin math problem where it's going to take multiple seconds, if not minutes, to get through all of that data. We're not actually using the vector index. If you look here, you can see we're just finding all the docs that match the filters and then scanning through them sequentially. So this also doesn't work very well."
  },
  {
    "startMs": 849072,
    "text": "And the trick to doing well on this problem is to do both. So TurboPuffer's vector index automatically clusters similar vectors. So you can see there's a group of documents here that are similar, labeled five, and another group labeled zero, and another group labeled one. And when we build these permissions indexes, we say, \"Well, Aaron has access to documents in cluster five and maybe cluster two, but he doesn't have access to any documents in cluster three or any"
  },
  {
    "startMs": 879052,
    "text": "documents in cluster zero. So we can just throw those clusters out. We don't have to consider them at all.\" So first, we find all of the clusters that Aaron has access to, uh, i.e. he has access to at least one doc in those clusters, and then we search the closest clusters, and then we just scan sequentially inside those clusters. And in TurboPuffer right now, the clusters have about 200 documents. This is something we're always tuning. Over time, we'll probably increase that number to, to maximize throughput. But this"
  },
  {
    "startMs": 908972,
    "text": "basic strategy fundamentally makes search at scale with permissions possible, and the trick is to combine pre- and post-filtering into what we call native filtering."
  },
  {
    "startMs": 923752,
    "text": "And this is the result. Across 450 million vectors, those 100 million documents turn into 400 million vectors, TurboPuffer's able to get 120-millisecond searches. That's the P90 latency across all of those namespaces, by employing this sophisticated native filtering strategy."
  },
  {
    "startMs": 947732,
    "text": "So with TurboPuffer, you show up with your permissions logic and your conceptual model, and you do what I think is a very simple translation of those permissions into TurboPuffer filter language. And hopefully you can directly map this code sample here, viewers Contains one, uh, viewers Contains one, groups ContainsAny B C. Hopefully you can see how you would project that onto your organization's own permission structure, and that turns into this very complicated systems engineering"
  },
  {
    "startMs": 977692,
    "text": "challenge under the hood. But TurboPuffer hides that all away from you so that you don't have to think about it. You can just express the logical filters that represent your permissions in a fairly direct way, and we handle the complicated query planning under the hood to make that search come back in one to 200 milliseconds, even at 100 million document scale. Now, we're engineers at TurboPuffer, so there are some limitations here. In particular, when you have nested groups, things get tricky. I'm not gonna have time to go into that in too much detail. But"
  },
  {
    "startMs": 1007972,
    "text": "if you have problems that look like that, please come talk to us. Uh, we have a lot of experience working through these, and we're always looking for ways to improve the product so that stuff is handled more automatically."
  },
  {
    "startMs": 1020592,
    "text": "So permission filters are really just the tip of the iceberg in TurboPuffer. We're trying to build an isolation-native search engine. That means whether you have fine-grained permissions within namespaces, whether you have security, regulatory compliance obligations that force you to regionally lock data, keep that data in your own VPC, encrypt that data, potentially not just with an encryption key you control, but that your own customers control, TurboPuffer has solutions for all of those levels of"
  },
  {
    "startMs": 1050272,
    "text": "isolation. So permission filtering is just one piece of the puzzle in building isolation-aware search in TurboPuffer."
  },
  {
    "startMs": 1060572,
    "text": "So that's all we got. That's how TurboPuffer and Bridgewater work together to do fast isolation-native search at 100 million scale. Thank you, everyone."
  }
]