RAG at scale: production-ready GenAI apps with Azure AI Search

[upbeat music] All right, let's get started. Hi, everyone. Thanks for coming.

Uh, I'm Pablo. I work in the Azure AI team at Microsoft. Um, and in this session, we'll talk about, uh, RAG at scale. Uh, and in particular, I'll focus on the retrieval portion of, of the RAG pattern.

The plan for, for this session is, uh, we'll do a quick recap, just make sure that we all have the s- we're, uh, use the same terms for the same things.

Uh, and then I'll go through kind of different dimensions of scale, and I'll comment a little bit on how do we tackle this in the context of AI search, what we've learned from doing this, and what are we doing to make it easier, uh, to scale these applications.

So kind of as a... By u-by means of a quick recap, when it comes to bringing your own domain knowledge, uh, to work together with, with language models, you effectively you have three options.

Either you, you can do, um, uh, prompt engineering. And, you know, while it's easy to dismiss, you can go a long way with, uh, prompt engineering along, especially these days, you know, models have longer co-longer context and whatnot.

Um, if, if that's not enough, sometimes the challenge is more along the lines of I wanna teach, uh, particular patterns o-of, or, or I want the model to learn jargon of some, in some particular vertical domain and things like that.

And for that, fine-tuning is often a good option.

However, in many, many of these cases, what I want is to have the model work over a set of data that the model didn't see during training. This could be my application data, my company data, company about the information about my users or anything like that.

And for that, the, the, uh, prevailing pattern right now is to use retrieval-augmented generation. Effectively, uh, what that means is if what-- if you want the model to know facts, then what you do is you kind of separate the reasoning piece of the picture from the knowledge piece of the picture.

You lean on the language model for the reasoning capabilities, and you lean on the use an external knowledge base to model what you know about a particular domain, the data that you have.

And you-- And the way you glue them together is, like, in principle, mechanically very simple. Of course, then it gets complicated because, you know, life is never that easy.

But in principle, uh, you have some orchestration component that, uh, you know, takes the task, uh, at hand. Let's say you have a chat application, and then the user takes a turn and asks, asks the next, uh, question.

Um, uh, the orchestration component hits some knowledge base, uh, seeking for pieces of information that could be used to produce an answer, uh, to this question and then grabs a bunch of candidates, and then you go to the language model and then give it a bunch of instructions, your candidates, uh, and ask the model to produce, uh,

like an answer to the, to the user question. That's kind of the essence o-of the pattern. We all know that in practice, it usually takes multiple goes, and there is a lot of tuning in the middle and, and whatnot, but the fundamentals boil down to that.

Um, so, uh, just by, by way of context, like, how many of you are doing, are creating and working on RAG applications today?

Everyone. Okay. Excellent. Um, so with that backdrop, what I wanted to do is talk about what are the pressure points when it-when you scale these applications. One, one thing that has been fascinating to see is, you know, we had, we had the opportunity to be involved in this space, uh, from very early, like Azure OpenAI has been

there from, you know, the early days of scaled language models. And one thing we saw was all of, I'd say last year, twenty twenty-three, everybody built a prototype of something to kind of learn and figure out what could be done with this technology.

Um, and, uh, the interesting shift to this year, to twenty twenty-four, it's been these applications are going to production. And when you go to production, you go from, "Oh, this demo is really cool," to all these users are, are, you know, are using it at the same time, and they want more.

They want more data. They want, uh, more, uh, faster answers and, and whatnot. Um, so the result of that is that, you know, when before we could focus only on figuring out the interaction model and the applicability of this technology, now, you know, elements of scale also, also play a role.

Uh, and scale can-- takes multiple kind of, uh, flavors. Like you-- These thing-things tend to scale in volume because one of the things that happens is when your application works well, users come back or the leadership of the organization come back, comes back and says, "Let's put all the data there."

Uh, and then you have to deal with it. Um, uh, the, also the rate of change of the data kind of increases, uh, and the query load increases as well because, you know, more people are using the stuff.

Um, also workflows tend to get more complicated. At first you go like, "Oh, like how complicated it can be? I'll take the question, search the thing, and then send it to the model."

Well, it turns out that, that sometimes it works, but often it doesn't. So you end up doing these multi-step workflows that hit the retrieval system and the language model multiple times, um, and that taxes all the systems and they all have to scale.

Also in, in the kind of, in the spirit of now let's put all the data there, now you have to deal with more, uh, more data types, different kinds, different data sources and whatnot.

Um, so, um, let's cover each, each of these dimensions of scale, um, in detail. And what, what I'll do is I'll cover it in the context of Azure AI Search.

That-that's what I, what I work mostly, uh, and, uh, because it's, you know, the way we think about this in Azure is we want to produce a system that is, that encompasses entire retrieval problem.

Uh, it's of course it has this kind of vector database capabilities. The, uh, vector-based retrieval emerged as a very, uh, useful solution in many contexts. So, uh, we have full support for that.

But we also brought into it like, uh, years and years of Microsoft experience in, in retrieval systems in the more general sense. Um, and, uh, we integrate them together so you don't have to connect a bunch of parts, uh, to have a proper, uh, high-quality retrieval system.

It all comes integrated. And of course, you know, we integrate it into the rest of Azure, so it's easy to pull data in, to connect to the other data sources and whatnot.

Um, and, you know, Azure is, is a place that is used to build some of the largest applications out there. So all the kind of enterprise readiness comes pre-built in, you know, from security to compliance to all these things that you don't wanna deal with directly.

You wanna kinda build on a platform that is already taken care of.

So while there are multiple moving parts to retrieval systems, what we've seen the last, uh, I don't know, eighteen months or so since the emergence of, uh, kind of RAG, uh, patterns at scale is, uh, vector retrieval is an important part of the solution.

And, you know, you, you can see why. Like, the interesting part about vector retrieval is, like, it's incredibly effective at getting this kind of soft conceptual similarity, uh, and, uh, put it to work right away.

So in Azure AI Search, we built a system that has kind of complete, uh, a complete feature set when it comes to, uh, vector search, including, uh, a pr-fast approximate nearest neighbor search, uh, and a-as well as also exhaustive search.

Sometimes you want precise search either to create baselines, uh, to know how kind of your recall performance is looking and things like that. Um, also, often applications need to combine, uh, vector search with the rest of the queries.

You need to filter, uh, filter things. You need to, uh, slice and dice, like, you know, filter, select the columns you want to project and things like that. Like effectively treat it like a database that also does retrieval.

Um, and we also see multiple scenarios where the documents have multiple queries, maybe for different parts of the content or for different data types that use different embeddings. Um, and, uh, and sometimes the queries need multiple vectors.

So we try to make it, uh, as these, all these specific needs come up as you're building an application, you'll find answers to all of them directly in, in Azure Search.

Let me show you this in action. Um, so let me start with a very simple example. So what I'll do is, uh, I'll-- I'm just here, I have a sm-small, uh, Jupyter notebook, uh, and what I'll do is I'll connect to, um,

uh, I can set up a default credential point at my Azure Search service, and I'll create an index from scratch. So this is all, like, all it takes to, to create an index once you have a service provisioned.

So in this case, I create a few fields. I create a categorical field, like this serves as metadata that you want sometimes to attach, to be attached to, to this.

I create a text field. Sometimes you want to mix text and vectors. Um, and I'll talk a little bit more about this later. And I create a vector field.

In, in this case, this is a toy example, so dimension is three. Of course, that's not very useful. You know, in practic- in practice, this is gonna be in the hundreds or maybe thousands of dimensions.

I'll also say what strategy you want the system to use for vector search. In this case, I'm using HNSW, which is a well-known, um, uh, graph-based algorithm for,

um, for indexing vectors. So when I run this, and I hope, uh, the, like the network was a little wonky, so, oh, there. Perfect. Um, so now I have an index.

I'm gonna get a, kind of a client to it, and then I'm gonna index data. And you can see here, this is a very simple case. These are my vectors.

Um, this is my full text, and these are my cat-- these are my categorical data bits. So you can let us do all the ingestion for you, or you can push data into the index.

In this case, I'm pushing data explicitly into the index. And once you have data, uh, if you look at here, like we have some vectors and we have some categories.

So you can, first you can just search. And because it's, in this case, I'm indexing vectors, when you search, you search with a vector. Uh, so I can, I can search for that and say these are the two closest to the reference vector that I gave.

And I can do, uh, um, a few, a few things, uh, kind of incrementally. Like for example, if I wanna combine text and keyword search, I can also, uh, search text right here.

Uh, I'm gonna go hello. Uh, and now I'm searching for both, uh, the, the text and the vectors, and then we fuse the result and rank appropriately. Um, and, and often in applications, you also need to filter stuff.

You can see here I have A's and B's in categories. So I can, I can, like, write filters. Uh, for example, here I'll say category equals A.

Uh, and then, you know, I only get the A's. Uh, and, uh, this is a trivial example, but of course, you can do full filter expressions and ands and ors and, and, uh, uh, ranges and, and whatnot.

And the filters are fast. So, so even if you have, um, hundreds of millions of documents, these are not a problem.

So in, in that one, if you had, uh, k-nearest neighbors, k-nearest neighbors equals two, how come they had three results?

Oh, because, uh, great question. Because, so what I, what I was telling this, the, the system here is from the vectors retrieve two candidates. But then I was like, I also told it, "Go to the keyword side and retrieve a bunch of candidates from there and then fuse them."

Uh, so only two of them were from vectors, but the fuse, the fusion of the two, uh, selected three. I actually, like if I, um, I can make this a larger number, and that sometimes is useful, um, to get more candidates.

And then still I can say, uh, as a result of that, get the keyword ones too and then rank the top N. So basically separate how many candidates you want from how many you wanna return.

So, so as, those are the basics of a vector engine, but there are a few, uh, key elements to, to consider when you're actually building an application in production.

The probably the most se- the most salient one is quality. Like in the end, the, your application works well for your users when they ask a question and they get the answer they're looking for.

And that is highly, highly influenced by whether your retrieval system actually found the, uh, the bits of information that you wanted, uh, that, that you wanted to produce that answer.

Um, in, in AI search, the way we do this is we do, um, what kind of most, uh, kind of more sophisticated search engines do that where you use a two-stage retrieval system.

First stage is recall-oriented and uses vectors and keywords and kind of all these recall-oriented tricks to produce as many candidates as we can find. And then the second stage re-ranks those candidates using like a, um, a, like a larger model that you maybe, let's say you have a hundred million, uh, vectors in your database.

Uh, you, you wanna use something fast to go from a hundred million to a small set. But then once you have a small set, you can afford to lar- run a larger, more sophisticated language, um, sorry, um, ranking model, uh, to create better quality ranking.

So that's what we're doing in the L2 stage. Uh, so when you turn this on, you effectively get better results. And, uh, what you can see here, there is a link at the bottom, happy to share it later, uh, with more details on the numbers.

But you can see that, um, so from the left, this is what you get when you just do keyword search using BM25. It's a well-known scoring approach. Second one is only vectors using Ada, uh, OpenAI Ada vectors.

Third one is using fusion, combined vectors and keywords. And the fourth one is using fusion plus, uh, re-ranking, um, re-ranking step, where across the board we see better results just out of the box when, uh, when re-ranking is enabled.

Yes.

So the re-ranking is the semantic re-ranking. Is that Is that another cosine similarity style thing, or how does that actually work?

Oh, great, great question. The question is, is it a cosine similarity thing? No. So the, the thing, bind coders are useful when, uh... because you can do-- you can encode all your documents as vectors and then the query as a vector.

Um, and then you can evaluate them fast because you are only comparing similarity. But that means that at no point the model sees the, the vec-- the document and the query at the same time.

So the re-ranker is, uh, these type of rankers are often called cross-encoders. And, uh, these are transformer models that you feed them the document and the query, and you ask them to predict a label of the correspondence of the query to the document.

So they are much better positioned to produce a high quality result. Uh, but they are, they're, they're, they're... At, at inference time, or rather at query time, you're running an inference step.

So, um, you can do it only on a smaller set. You couldn't do it on the entire data set. It wouldn't be practical. Uh, at least not for interactive performance applications.

And speed?

Speed?

Yeah.

Uh, this is like a hundred milliseconds, give or take, for a model like this. Depends on the cross-encoder that you, that you use. Ours, our trade-off between, you know, make it fast enough but make it very high quality, we landed on a hundred milliseconds ballpark.

Um, uh, and we found that that works well in terms of interactive performance because the majority of the latency ends up hidden on the LLM call. Uh, uh, and you still get very high quality results.

Um, the other thing, I won't drill into this, but, uh, often, uh, one second. Often the other dimension of getting quality out of the system is to narrow the data set.

If you know discrete, um, metadata elements that can help you narrow the data set, that's the most effective way. Uh, then do all the ranking tricks on top of the resulting set.

But, uh, first, uh, scoping is a, is a very effective way to, uh, to get quality up. Yes, question.

So my question is, do you, uh, extract the keyword from the prompt, or like how do you set the keyword?

The, for the keyword part of keyword search?

Yeah.

Oh. So if you look at, um... Let me show you this example. So if you look at what I did here, is, uh, you give us keywords, you give us text-

Yeah

... and then you give us a vector as well. Or you just give us the text, and we'll turn it into a vector too. Uh, so we need the original text.

The vector alone is not enough because we can't go back to a keyword.

No. So, I mean, if you have a RAG application, let's say you have a chat bot.

Yeah.

How can you know how to find what kind of keywords could be relevant, let's say, to the prompt that the user is giving?

Oh. So, um, u-usually in a RAG app, w- the question is, what of the conversation so far do you wanna use for search? Is that, is that the question?

No, like I understand, but my question is, so if you wanna use... Because in your method you showed that the fused search-

Yeah

... plus

Yeah

... uh, prompt. So how, how do you... Like for vector search it's easy, right? You vectorize the prompt and then search.

Yeah.

But if you wanna like search and vector fuse-

Yeah

... is there a way you can extract like from the user's element to, uh, extract the potential keywords for you to search? Because you cannot pre fix those keywords for every potential chat or, or prompt that the user can have.

Yeah, I... Maybe we should talk after the talk because I'm not sure I understand the question. Like usually, like you, in the index you have the se- the text and the vectors.

Uh, and then from the user, you extract a candidate search for, for that. This is before you send it to the LLM. Uh, but maybe we should chat right after the talk on the details.

Um, so let me go back here. So the, the other dimension of scaling is, um, is, uh, just, just how many vectors and how much content you can fit in one of your indexes.

Uh, and, uh, for that, in, in AI search, like one thing we learned last year, uh, or in the beginning of this year, is that it went from everybody had tiny indexes to everybody put all their data in these systems.

Uh, so we re- we, uh, accommodated for that by significantly increasing the limits. Like for most of the ser- services, uh, you'll get anywhere ten, between ten and twelve times the vector density on, on the same SKUs, and we didn't change prices or anything, just so you can build larger applications on the same setup.

With these new limits, you can build multi-billion vector apps by just provisioning a service and uploading the data. And it's surprisingly straightforward. Like, you know, a year ago, the billion dataset vectors was the thing...

It was a curiosity used for, uh, for benchmarking. And now you can just create an index and upload them, which is very impressive to, to see. Um, I'm not gonna drain the slide.

I put it here for reference. Uh, these are kind of all the new limits that, uh, that we have, are significantly higher than the, than the ones we had before.

Um, one of the things that has been exciting for us to watch, uh, to watch grow is, uh, you know, among our customers, one of them is OpenAI themselves, uh, where they have a lot of RAG workloads, like when you have files in ChatGPT or when you use the Assistance API.

All of those, you can create these vector stores inside their system and, uh, all of those are backed by AI search. Uh, and when we increased these limits, um, one of the things they did is they increased the limits they give to their users by five hundred times.

Um, so, uh, it's been impressive to see how fast they grow, and it's been fun to kind of see, uh, kind of first, um, uh, from up close how a system can scale at, at like this big and, uh, and still be fast and responsive and whatnot.

Um, so finally, the, the other thing you can do is sometimes the limits, higher limits are enough. Sometimes you wanna push even more data into it. So the other thing we've been working on is quantization, where we, we...

you can use narrower types. Like instead of using full floats, you can use ints, uh, like int eights. Uh, and they just simply use less space at the trade-off for a little bit of quality.

Uh, and interestingly, you can even do single-bit quantization. And I confess that when people said, "Hey, like we're gonna do metrics for single bit," I felt people were wasting their time.

Uh, but it actually works. Uh, it works surprisingly well. Our evaluations show that they are still, for some models, in the mid, uh, low to mid ninety percents of the original performance.

And other companies have seen the same thing. Like for example, this is an evaluation from Cohere, but separate company. Um, and they also see like about almost ninety-five percent of the original precision is preserved.

When, uh, when using bit encoding. And you can get the precision, remaining precision back by re-ranking, uh, when you're done. So surprising that it works, but you go for float thirty-two to one bit.

That's thirty-two x the vector density. Um, and it's faster because you just do Hamming distance mu-much faster than computing on a small number of... a smaller number of bits instead of cosine similarity or something like that on a wider, wider set.

So because of this, we now support all these types in AI Search as well.

Um, uh, I'm gonna skip this slide in the interest of time. Um, so and, uh, you can do quantization yourself, or you can just enable it, and we will do quantization, uh, for you.

And, uh, if we do quantization for you, we will also store the original, uh, precision data there, which means we can do oversampling, where we query at the quantized kind of compressed, uh, version of the vectors, but we have the full precision stored, stashed as, on the side.

Uh, so later, we can re-rank at full precision. Uh, and, uh, so you can effectively choose between you want a highly compressed, uh, index that is, uh, a little lower quality, but, but, but larger, or it's a little slower but larger.

Uh, or you just don't compress it, and then you get the quality up. So effectively, you can choose any of the three, uh, and it's effectively up to you, uh, what you wanna prioritize.

But we give you control for all three dimensions.

And then the last thing I wanted to, uh, touch on is the, the other challenge is you have to keep adding data sources that you bring into these RAG systems.

Um, and, uh, each of them, you connect to them differently, you enumerate changes differently, um, and, uh, that's, that's just not where you wanna spend your time. Um, so we have this ingestion system that includes integrated vectorization as part of AI Search, where if the data is in Azure, whether it's in Blob Storage or OneLake or Cosmos

DB, we will connect, deal with all the security and all of that. We will automatically track changes, so it's not a one-shot thing. But as, as the data changes, we'll pick up only the changes and will only in process the changes, so the cost is also incremental.

You don't pay as you update the stuff for the entire set. Uh, and then we'll deal with all file formats, you know, PDFs, Office documents, images, un-unpack the nested formats and whatnot.

Do chunking, do vectorization, and land it on an index all in one go. And this is a, like, industrial strength pipeline that you set up once, and then it continuously runs, uh, af-after that.

As your data changes, your, it reflects the changes. So you can focus on the RAG stack, you know, the workflow, how you query the system, but you don't have to think about how the data makes it there.

If the data is anywhere in Azure, we'll index it and create, like, an index that follows the original data automatically.

All right. And, and with that, uh, I know I raced through this content in these twenty minutes. I'll be hanging out outside if anybody wants to chat or have questions.

Um, and I would encourage you to go try AI Search today. Here's a link to the starting point. Um, uh, and Azure subscriptions include a, include a free instance of AI Search, so you can even give it a shot in a minute without, uh, without having to pay for any of this stuff.

With that, thank you. [clapping] [upbeat music]