Vector Isn't Enough: Hybrid Search & Retrieval — Jeff Vestal & James Williams, Elastic

Thank you for coming. Uh, my name is Jeff Vestal. I work at Elastic. I am a AI architect. We picked a new title this year, so make it relevant. This is... James, you wanna introduce yourself?

Yeah, I'm James Williams. I'm a, uh, solutions architect focused on search, so I help customers with all the technical stuff you're about to walk through today.

So this is a very hands-on, as in I do not intend to talk much. I've been talking a lot lately on the road, so we're gonna have me not talk a lot. Um, we're gonna go through hands-on stuff. I will say in the upper right corner, you'll see that short URL, um, it's ELA Elastic. This is the workshop environment. Um, quick note. So they never told me how many people were coming, so I took a random guess of seventy-five. Looks like I undershot a bit. So there, um, seventy-five of you

will start up pretty quickly. The other handful that are a little slower on the, on the trigger here, it's just a few extra minutes to start up. Um, but basically, this is gonna take you to an Instruqt environment, and then I'll give a little intro. I'm just giving this spiel while it goes. Um, we'll go to

this. You're gonna go here, and then you'll hit Start, and that's gonna spin up an environment for you. So while that's doing it, um, so how many people are familiar with Elastic, the company, Elastic, Elasticsearch? All right. Pretty good. It's a good turnout. Um, and how many of you have used Elastic to build a, like a, a RAG-backed application, retrieval-augmented generation? Great. How many of you are sick of hearing the term RAG from the past two, three

years? Fantastic. I know I said it about a thousand times at many conferences, and um, my boss, who no longer works here, actually used to get really mad when we'd keep bringing up RAG. Now, the thing is, um, retrieval augmented generation didn't actually go away. Luckily, it just kinda fell to the background. But as agents got more and more, uh, capable and, you know, effective, they just kinda took over the management of being able to, you know, retrieve data and, and analyze it. So the models are getting certainly smarter and smarter.

Um, you know, Fable came out for a minute and teased us and then ran away. But, um, there's still a, a need for pulling in information into these models, be it, you know, things that are time-based, that change, proprietary information, um, price data, anything out there that the model hopefully hasn't been trained on, whether they tell us or not. Um, but there's a need for that. And so that's where things like Elasticsearch come in. It's the retrieval layer that helps make these, these agents very capable.

And the, the trick or the catch, I guess, is if you feed these models, no matter how smart they are, garbage data, and you confidently tell them, "This is the answer," pretty much they're gonna believe you because that's what you're telling them. So we wanna be able to give accurate information and, and give it. So this is kinda what this workshop is about. I'm sure there's 20 other workshops that explain how to build agents, and, uh, we actually have some too if you wanna go to our website. But this one's gonna talk to, kinda remind about,

um, vector search, semantic search, why it's important, uh, where it's useful, but also where it breaks down. And then, um, we'll go into, um, the kinda lexical search, BM25. So that's that keyword search we've been doing for, for decades, um, where it's still relevant today and useful, and then also again, where that kinda breaks down, and then putting them together into hybrid search. So being able to kinda get the best of both. Um, we'll do combinations with, uh, RRF, which is an out, out-of-the-box combination, and

then we'll do some little bit more advanced, um, linear combination where you can tweak those balances a little bit. And then should you have time, we'll actually put it into, um, a little bit of Elastic's Agent Builder platform, where you can see what happens if you retrieve bad information and tell the agent it's, it's good. It'll-- doesn't work. I mean, it works, but it gives you a bad answer, and if you give it the correct answers and things like that. Um, should you be very quick and get all the way to the end, there is a

bonus, um, one about, um... Sorry, re-rankers, so re-ranking, where it comes in and useful of that. Um, the workshop is kind of basically four plus a bonus, um, chapters, challenges. Each one has two parts. There's the part in the DevTools console where you just run API commands within Elasticsearch. Um, each one of these environments, if you're in it and hit Start and everything, what's happening is a serverless instance is being spun up for you. Um, and then you'll load some data. It's a small

dataset. You know, it's a workshop, so nothing crazy. And, um, you'll run the, kinda the, the short version of each one of these in the DevTools console. And then you'll hop over to a Python notebook tab, and you'll run the more in-depth one. What I would say is I'm gonna stop talking in a minute and let you work on the workshop, and in every s- 20, 25 minutes, we'll kinda check in and see where people are at. I'll tell a little summary about the, the cha- the chapter challenge you're at. Um, to get the most out of this workshop, I would

definitely spend time reading through what's in the, the Jupyter Notebook especially. It kinda talks about why the documents come up in the orders they are and how to tweak it and how to tune it and why it's important. Um, if you just blindly hit Run All in the notebook and fire through, you'll be, you'll be done in 10 minutes. But, uh, if you actually spend through it, it's gonna probably take longer than this workshop session. So, um, with that, I'll let you get started. So some of you should start up right away. Some of you will take a few more minutes. And in the meantime...

Has anyone actually started? Sometimes these take a few minutes. That's the thing. Yeah?

We got it.

Yeah. Okay.

And if you get stuck, raise your hand. I'm basically the runner, so I'll- Check out what you've got going on on your, on your system and, and, and potentially help unstick you if, if it's something trivial. And if you have any questions, feel free to raise your hand. Happy to provide a little bit more detail about what's going on in the exercise if, um, you're looking at something and you're, you're a little bit confused and want, wanna ask a question there.

And this is all also available, um, in a repo that actually should be public. Yeah, I made it public. Um, so you can run the notebooks, um, outside of Instruqt, either after or, or whenever if you want.

Kinda glossed over this, but the one big thing that I think has changed with RAG over the last year, year and a half, is some of the data providers for vector database, and we're in that, in that bucket, have brought inference into the vector database. So you will see embedding generation happen within Elasticsearch Serverless. We have an inference service called Elastic Inference Service, very, um, creative

name, that powers the models that are being used for these workshops. So we've got Jina models from, from an acquisition that we made, so we actually do our own model development now. And, um, and it's focused mostly on dense vector models and re-ranking models. So you're gonna get insight into some new tech that wasn't part of Elasticsearch a year ago.

That's, that's all I have to add there. We've heard of chunking.

We do he- we do, we do help with chunking. So Iragensis, and feel free to jo- jump in, he's the manager of the search specialist team, I just wander around. Um, so as James mentioned, uh, we have, you know, the text field semantic text in Elasticsearch. So there's a couple ways to do this. Semantic text is kinda the automatic easy way on road. Um, so by default, we will chunk, uh, sentence boundaries and break into a certain amount. I can't remember offhand what the number is.

Two hundred and fifty.

There you go. And so that's, you know, configurable. So by default, breaks on sentences. You can break on word boundaries, you can change it. Um, the, the Jina model that James mentioned, we're, you know, part of Jina now, or Jina's part of us, whatever. Um, it has a very, very large context window, but you're not always gonna wanna fill that for various reasons. Um, and so you can configure chunking down to the word, to paragraph, page boundary, things like that.

We have a semantic mapping field in Elastic, which basically takes a piece of text and chunks it. So if you send us five pages of text, we're gonna chunk that into two hundred and fifty token chunks with one sentence overlap and store that in the, in the system for retrieval. There are frameworks that talk about semantic chunking, and there are other labels for, uh, what I would call, um, large context window

chunking. We actually, and, and this is talked about in the lab in the later sections, uh, we do have a technique that we use called late chunking that can work on large context windows. That is not exposed via Elastic Inference Service yet, but it is something that the Jina models do support. That's the closest thing that I would say we would have to semantic chunking, but we don't really have a need for that construct because the semantic text field itself, uh,

allows us to deliver the same thing you would get with semantic chunking just by chunking the data and then, and then running searches across it, um, in a, you know, in a semantic flavor. And you, and you'll get exposed to that in the lab. So we would, we would solve the problem with traditional chunking and hybrid search, not any exotic semantic chunking scheme. You're welcome.

All right. Yep.

Will this be available for a few days or-

Yes. So I made it available for a week-

A week

... 'cause I know how things go. Um, so it is available for a week there, um, and this isn't a, this isn't anywhere. Um, I, the workshop is in my repo for better or worse. Um, and so the, the notebooks are all in here. So if you, if you wanna run just the notebooks, um, you know, you can hook it up to your serverless cluster or ele- any kind of

cluster should, should work that has semantic text, um, in there. But yeah, the, the workshop itself is good for a week.

And where was I?

And yeah, so like I mentioned, the, the first part, the first half of each lab is the Dev Tools where you run a, a mapping. The left side is the input for your API, the right side is the output, and then you'll get down to part two,

where you pop into the Python notebook, and it'll automatically connect to the, um, the Elasticsearch environment that's spun up for you.

I see they're letting folks in as, as folks kind of go out to make sure the room is still full. Um, the top right-hand corner is really the URL that you wanna hit. That's where all the workshop content is. And then there's a second URL that is the source code repo for all the notebooks for the workshop today. Those are the two most useful resources. So if you're coming in and you haven't fired up the workshop yet, top right-hand corner, and then the bottom

right-hand corner if you're interested in the Python notebooks, the power of the content that you're walking through today.

I'll probably repeat that message every time I see someone walk in.

That is a really good question, and I'll repeat it so the other folks can hear it. But is, it is a major point of this, um, lab sequence to let you know that BM twenty-five based search, which is traditional lexical search we've been doing for a long time, and dense vector semantic search, do those two things need to be put together to form hybrid search? That is the core thesis right now that we have as a company, and the only thing that

we add to that, well, obviously that, that's of value, is the option of adding re-rankers on top of that. But yes, that is a core thesis. We're firm believers in that approach, and that's in line with everything that I've seen since before even joining Elastic. Even if you're using a dedicated just pure vector database, you often have to pair that with some other retrieval system and do the equivalent of hybrid search. The advantage to Elastic is, and you're seeing this in the

labs, it's one API call. We handle all the inference that happens from ingest chunking to real-time search, where the query gets translated into a query vector, and we do all the work for you under the hood. And that's wh-- we, we consider that of high value, especially when you're talking about large scale use cases where latency, accuracy, tunability matter.

They are configurable, the chunking settings. Are they configurable? They are. The documentation has insight into how you can change them and what chunk settings can be changed, so we've got very detailed, um, API docs that walk through that. Uh, they're not described in detail in the workshop, but if you go to the Elastic docs, you'll see what the, what the configuration settings are for chunking. And some of our customers do bring their own vectors and chunk on their own. We support that through standard, um, SDKs. So if you're using

Python, Java, .NET Core, like, pretty much any TypeScript, any modern programming language, um, you could still use Elastic in the format for these workshops. You're just doing more work as a developer.

And so I'm sure people are moving along, and you can continue on as at your own pace. Um, like I said, I'm gonna check in every once in a while and kinda talk about the, the labs a little bit. So the, the first one, right, the, the whole point is essentially this little chart right here talking about how vector search works and, and semantic search and what you're getting at. And so when, when you use an embedding model that's, you know, hooked up to Elastic, and, you know, we talk about Jina as the out-of-the-box one, but you can certainly bring your own. You can hook up to other inference services.

Um, we just make it easy for you to get started with Jina, but certainly bring your own and use semantic searches or semantic text as well. But the, the nice thing about this particular way it sets up is, you know, when you create an, an embedding, a query, you know, at in-ingest time, you create the embedding from your, your document. But then at query time, you need to use the same embedding model to do the query time. Otherwise, it's, you know, like me trying to speak Portuguese. It would not go well. Um, so by... When,

when I started-- When we started doing this a few years ago, um, in the early days, early, um, you know, you had to keep track of the inference ID. You had to keep track of the model, and then the model, you know, then you have developers all, and you have to, like, make sure everyone's using the right one, and there's all these things. So by embedding it in the ID in the, uh, field itself, it helps in two ways. One, you only have to set it once, and as the person who's setting up the index, um, you know, there might be other people using that same index to ingest their own data or query it, right? So you can set that up

once, again, whether it's the Jina model or, or some other model. Um, anytime you use that field, it just becomes like another field. You just either in-- send data in or you query it. So then at query time, what you're seeing up here is, um, the text comes in. It may or may not get chunked up, depending on the, the size and settings. It goes to the, um, maybe the Elastic inference service, which is default out of the box. Maybe it goes to another inference service. That's fine. And then we generate the embeddings, and we

then pass it back to Elastic to do the, um, HNSW ANN. We do have other discount. We have disc ANN, but by default, we're using HNSW, the hierarchical navigable small worlds graph. Um, basically, it just allows quick traversal to find nearest neighbors of, of relevant documents. And then you get back your top K, five, ten, whatever you want, documents, and that allows you to, ideally, if everything's working well, find documents or, you know, chunks that are semantically

similar to what you're asking. And as you learned in the lab, it's not the exact keywords. It's the idea of what you're asking. So if, if all of us in this room ask, you know, how to set up security on, you know, TLS on Elasticsearch or something, we're all gonna phrase it a little differently, but the idea is all the same, and so hopefully you get back the same documents. And you kinda saw where sometimes that breaks down. Um, one of the key places is if you have a very specific keyword, um, certainly

if you have a keyword that is not a word, like a product ID that has makes no sense, it's very specific, um, semantic search then kinda falls apart 'cause it doesn't really know that. And that's what the second lab is, where bringing in, um, BM twenty-five lexical keyword filtering and all the other traditional so-called search, uh, methodologies we've been doing for a long time help, can augment that. And so I'll give you... I'm sure some of you are progressed past that one, but I'll check in in a few minutes and review lab two. And

again, if there's any questions, feel free to ask.

Right, so pa- yeah, paging, so if you have, like, a hundred results, you wanna get back ten at a time. Is that what, is that what you're asking for? Or-

Oh. Yeah, so you can do that. There's, um... I forgot the, the exact thing, but there's a, basically, essentially a checkpoint ID, and so then it, it continues on. And so, yeah, you're not researching aga- you're not re-running the search for that part. It goes on. So you can... That's, uh, what our pagination API, uh, part does. So if you know that's a possibility, you use that, and then we send you that key, and it's basically like this is where you start and continue on.

I think the official label for it is point in time.

There you go.

If you're looking in Elastic docs, uh, just search for point in time, and it shows you how to use point in time to do, to do just that.

All right. Good question. Um, so the, the summary field is generated just for this workshop, and then it was generated by an LM. It, it's not exactly part of it. The real reason is when I first ran through this, and you print out the whole body field, it was way too long, and it was cluttering everything up, so just having this little summary field. Now that said, you can use, uh, for example, an ingest pipeline in Elasticsearch and make an inference call and say, "I do wanna generate a summary of, of this, you know, document,"

or, "I do wanna pull out the possible queries that could do that." So you can do that on, uh, ingest. It's just for this particular thing it was just to make it easier. But so the, the vector field you're searching is the, is the title, yeah.

And as we get deeper into the labs, we don't use that, that construct, but there is LLM completion API calls in, in the later labs that it would be the exact same thing. You just wire it in your ingest pipeline. Question? In memory, do you have a search- It's a great question. So when it comes to work that we've done to make HNSW more efficient, um, that started two years ago, actually two plus years ago, where we introduced int eight quantization, which I still, I think

is still the default. So by default, we're gonna take float thirty-two vectors and quantize them before we store them in memory. We still s- we still keep the full vector, um, but we're only storing the quantized vector in memory. We also have, uh, BBQ, which is... Yeah. And we've got a blog that talks about BBQ. This is a little over a year old, maybe even a little bit longer than a year old, uh, where we've compressed the HNSW, um, graph vector that we use even more, all the way down

to, uh, basically a bit and, and there's sophisticated stuff under the hood that supports that and still keeps the recall high. And then we just introduced Disk BBQ, which pretty much eliminates the memory overhead for, um, for vector search. The only, uh, difference I would say between Disk BBQ and BBQ is that we actually have an option in Elastic to not use HNSW as the underlying mechanism for search, um,

and the, and the newest form of that is Disk BBQ, which is, um, proprietary to Elastic. No one else implements that. Graph-based search. You wanna take that one?

Not really. So graph-based search. Uh, so to be fully clear, we are not a graph database. There are some very specific graph databases. Um, graph has its place. It usually has a lot of overhead to keep it maintained. If you do that, great. Um, for the vast majority of the people, though, it's-- we find that it's not really needed, where the ability to, especially with quantization, where you can search across large amount of documents quickly, um,

now, and especially with the agent world too, where you can do multiple queries in, so you can... One of the labs later on, it does a couple round trips, I think just two as an example, but you can go in, get your kinda first phase, get a sample of documents, and then see, okay, so they're all related to this tag, and then go searching. So that's where things like filtering and, um, some of our other search methodolo- methodologies help. Um, and so, you know, I mean, it just, it comes up, but we haven't seen it r- you know, too often as an actual blocker where you have to

set it up.

And, and the last thing I'll say about that is what we see is far more common is, um, machine learning models that do classification on the fly, which is another way to solve that graph challenge. Works in some situations, not, not in all. But Neo4j and other graph databases have been around for a long time. I don't know of any that scale really well, and we're trying to target multi-billion vector data sets, which means that, um, we're very careful not to, not to intro-introduce a construct

that keeps us from being able to scale at that level. And we, we, we haven't found the need to go there yet, although it comes up a lot. It was a big theme last year at this conference. I remember being here and seeing GraphRAG was, was heavily talked about. Probably will continue to be. Um,

I haven't seen a technology nail that yet.

And maybe there's a Neo4j person in here that would contest me on that and say, "No, no, no, we have it figured out." I think they do, but not at, not at the scale that we operate at.

Yeah, I just came from that session. Yeah. Oh, so you came from their session. Nice. Okay.

In the back.

Yeah. You mentioned that-

I think the question was we can use classification models to, uh, um, to achieve some of the same results you might get with, with a, with a graph traversal. Um, so we have the concept in Elastic of running, uh, machine learning models in, in Elastic on what we call ML nodes. Um, you could basically upload whatever classification model that you've identified that works and wire it into Elastic that way. The other mod- the other option would be to create your own model that you have

your own external inference to and, and potentially use that, um, in place of, uh, of our ML technology, and I'm hoping that we'll introduce more sophistication down the road in that area, but we don't have anything noteworthy to announce right now. We're focused on taking the Jina models that we have and pushing the limit to what we can do with handling unstructured data and, and dense vector embeddings and re-rankers. We'll probably end up pivoting off of, off of that once we nail the, um, the re-ranking piece and,

and the, and the dense vector models that are, that are in development and, you know, doing quite well in the market.

I'll add in too that along with that, so there's classification models, and you can r- also run things like NER models, named entity recognition. So, you know, if you pull out the entities, if you're trying to identify person, location, en- um, business names, things like that, or product names, you can run those on the fly as part of a, you know, part of the query where you s- as the, the query comes in, you run it through the NER model. And NER, NER models are very fast 'cause they're, they're small. They're really small models. And you pull out these entities, and then,

you know, again, now with agents and things like this, you don't have to wire up crazy, um, painless scripts. The agent can take those and then create a query that's more tuned for that. So there's a lot of ML capabilities that we don't talk about as much as we used to because, again, like Jim said, we've kind of pivoted to focus on other things, but they're still there in the stack and very capable. All filtering in, that's great. There's Sunil. What's up, man? Um, one of our distinguished SAs over in the corner. You can ask him all our

questions. I'm gonna f- now redirect. Um, for the people who came in, aside from Sunil, the upper right corner is the short URL for the workshop, and you can spin that up. It opens up a kinda lab for you, lab environment, and also spins up a, um, Elasticsearch serverless cluster all set up for you. So feel free to run through there.

So there's, um, there's one main knob for r-r-reciprocal r- uh, uh, RRF, and it is-- And I'm, I'm spacing out here off the top of my head. I do not know. You're, you're pulling it up.

It's something constant equal to eight.

I don't know if we expose it in the, in the RRF, um, attributes for the retriever.

What's the default? What's the question again?

What's the default constant in-

Constant. The rank constant. I- it is something that you can configure. Um, I don't know what the range is.

It's a denominator, right? RRF is an equation, so-

That's what lab five is, the optional one. It's all about, uh, well-

Sorry.

Lab five is re-ranking. RRF is in there somewhere. Um, sorry, brain melted for a sec. Um, it's in there somewhere about, talks about what, what Sunil was talking about, a little bit more example, and you can change things like that, so...

So I don't know. Did that answer your question?

Yeah.

Okay, cool.

Thanks.

There's a question here.

Fire away.

Um, so what is the recommendation in using the re-rankers when your corpus of data is small? So how effective is it or how effective is it not?

That's a great question. There's actually a table that kind of explains some of those decision trees in the, uh, later workshop. I wanna say workshop number or lab number four. But the short answer is for a small document corpus, and, and, and my rule of thumb is always see what you can accomplish with a combination of BM 25 and semantic search. If that gets you the recall or precision, uh, target that you're looking for, whatever relevance targets you have, stop.

Anytime you add anything on top of that, you're going to be taking on an additional latency hit. Only do that if you need to. And the next, uh, you know, extension point would be to, um, to use a re-ranker model. And, um, and then it's just a matter of picking one that, that you think is gonna work best for your corpus and doing some testing and validating that it moves the needle enough. Um, that would be kind of the standard happy path approach. There are some folks that would change that, uh, recommendation slightly and, and

say, "Look, just do BM 25, just regular search, um, and, and then, and then add semantic search if it's not, um, uh, relevant enough." But I'm a big fan of dense vectors, so I would just go ahead and wire them into the architecture right out of the gate, and, uh, that way I don't have to worry about things like s- you know, you, you, you get into these weird situations with lexical where you're starting to add synonyms or get into advanced, um, lexical tuning techniques that you might not need if the

semantic search is able to, to pair well with the lexical search.

Thank you.

Welcome.

All right. So now people are moving along, which is great. Um, just give a little wrap up on lab two, which again was about your favorite p- lexical search. Uh, but hopefully you did learn, uh, you know, get across the point that it does, it does have its, its points. Um, we just barely touched the surface on lexical because, you know, for fifteen years or so, you know, a dozen years at Elastic, that's what we had. So if you've ever been on the Elasticsearch doc sites and you go

to the search part and you click on the search API, there's an insane amount of ways to change and tune and tweak lexical search because to Sunil's point, before we could do semantic search, we had to do all these, uh, crazy things to get relevant search. Um, luckily now we don't have to do quite as much, at least usually, once in a while still. Um, but with lexical search, you do-- it does have a, its, its point, um, but it does tend to break down, um, any time you break out of looking for

very specific keywords or a very specific phrase. Um, the upside of it is it's, I mean, uh, I don't even know what I wanna say. I was gonna say it's generally faster, although now with quantization, things like that, that's not always the case. But, um, by default, it's kind of faster. So, um, in lab three, if you're-- most people are probably along there, but you move on to combining them.

And I will say that e-even though I'm a big fan of dense vector search, that lexical search usually does have lower latency profiles in almost all use cases, and that's because there is no ML inference happening. So anytime you go semantic search with, with dense vectors, you're introducing, um, a call to, um, you know, an inference endpoint, and more often than not, that inference endpoint is gonna be more of a bottleneck than the rest of the system. And then that's another reason why we wired, um, the inference

into the, into the data platform because when you're doing end-to-end testing, it doesn't matter if your vector database returns vectors, you know, really fast. If the inference that you're depending on is slow, then your search is gonna be slow. I think that's a question that is shape-shifting as the data platforms wire more capability into the systems and these models improve. The, um-- And every vendor's gonna have their own specific take. Our take is that everything starts with traditional

search, which makes a lot of sense given our heritage, and then you add semantic capabilities on top of that, and, and then you layer in re-ranking if you, if you need to. In terms of where that stuff happens, that's where the devil's in the details. As, as a developer trying to build an architecture around this stuff, your options are, are quite vast, and depending on the stack you start with, you're gonna have perceptions of what you should be doing in the data platform versus what

you might have some sort of middleware around it that, that is, that is wired up. And, and I'm using the word middleware kinda loosely, but there's a whole application runtime construct that's went away ever since this, this AI boom started that is gonna have to come back, and it's either gonna land in the data platform itself where we're doing inference and some of these other things that you would normally have to coordinate with these external services or, um, there's gonna be, you know,

platforms that try to, try to accomplish, um, that, uh, and then use the data platform as just kinda this thing that you plug in. And I'm thinking about, like, old school stuff, like back in the old days when I started my, my tech career, we were doing object relational mapping, and the database was, you know, supposedly pluggable. You swap out your credentials, and it doesn't matter because you're not using SQL anymore. Um, I think that we're in the very early days of AI where for retrieval purposes, as soon as

you, um, as soon as you start writing code, you're locked into whatever stack you've picked. And, and that dictates a lot of different design decisions that, that you have to make, especially as you start to scale. And where w-we kick in and have an advantage against a, let's call it a simpler stack, would be when the, the data volumes get larger, and the latency requirements become more, um, you know, more, you know, more-- I don't wanna say severe, but the SLAs are really tight. That's when we're gonna shine because we

can do things that the other platforms can't do, and we've got a level of tunability, especially on the relevance side, that, that can't be accomplished with other data platforms.

That's a very good question that we do talk about in the lab, but I don't think we provide that authoritative answer key of here's how you construct your weights based on, you know, things that you can tell from the shape of your data or the shape of the searches that you know users are gonna execute. The, um, the short answer to that is today that's actually hard to do. Um, the longer answer is, and, and this is something that I think you'll see come from us, is, um, there's a

missing element to relevance tuning that hasn't been, um, addressed in a way that makes this turnkey yet. And, um, I've actually got a reference architecture that I put out a year ago. It needs to be refined. It's based on Mashro, uh, about auto-- that covers auto-tuning and that concept. Um, but in a perfect world, you would actually be able to take the searches that are coming through the system and the telemetry from those user experiences and things like click-through traffic

and, and, and thumbs up, thumbs down, basically telemetry from the users, and then also, um, any other business metrics that you have wired into your use case to, uh, to actually tune the relevance, which might introduce different weights for linear. Um, it might, um, toggle between RRF and linear with specific weights depending on the type of query coming in, or, um, you have all sorts of other types of techniques. The one that I think is more common than even those two is just coming up with, uh,

different sets of, of filters, um, based on the, the query that's coming in. So, um, Sunil built out the original architecture for this. We called it an intelligent search layer, um, and it's something that Azure Search has tried to kinda wire into their product. It is something that you can actually do today with Elastic using the agent builder construct, which is covered in the last lab. Um, but that's, that's a little bit different technique than just tweaking the weights on the fly based on what we're seeing. You, you would

actually have, um, the, um, the LLM interrogate the query and then, and then execute searches with different filters based on that interrogation, which I actually love that technique. I think it's fantastic, and I'm glad that we have a construct for it with, with Elastic, and it's-- that's all, you know, stuff that agent builder is designed to solve for. Yeah. It's almost chicken and egg, right? Yep. You have no amount between the weights. You might just end up with the same thing. Exactly. So you're like, "Okay,

I did the search." And that's where I'm hoping, and, and this isn't something that, that we've announced that we're gonna do yet, but I'm hoping that at some point you're gonna see, uh, the concept of auto-tuned relevance, um, be something you can enable and, and have attributes or settings that you specify to, to turn that on and maybe even turn it up depending on the, uh, the use case and, and what you're seeing with actual usage of the system. And, and just to be clear with everybody in the room, we have not released any technology that does that yet, but it

comes up often enough in the field when-- where I'm working with customers that It's something that if we had it, it would move the needle big time and make developers' lives a lot easier. Would you agree?

I would. Yeah.

Okay, I'm gonna, I'm gonna bring it up again to our CTO and make him, make him say no again.

All right. So we're gonna just walk through this one, and most of you are probably along with that. Passed it. If not, it's all right. Um, so, right, so we're trying to combine the two, the two different methodologies here, and the, the crux of the problem is, and it's in here somewhere, right, like BM25 has a scale, it scores documents from zero to kinda whatever, whereas vector search normally it's zero to one. And so when

you have these scales that do not, that aren't the same, obviously a poor result in a, a lexical BM25 search is almost always gonna beat out, from a pure number perspective, a good semantic search. So then it's the fun game of how do you combine them? And so there's a couple of different ways. Um, we don't get into the kind of choosing one on the fly

here, but there are, you know, different ways of getting to that, kinda getting the basics here. So RRF is kinda designed to be the out of the box, works pretty well, um, until it doesn't. But it's the, it's a good first start. You don't have to know how to tune, you don't have to really know, um, the details of your data. But the idea then is that essentially, forgot there's a table in here, you, you throw out the

scores. So you, you run the two searches in parallel, you take the order, there's a little bit of that rank constant in there. But essentially, if you're high, if your result document is high in both, it's probably a good ma- it's probably a good match. When they're not, that's where the kinda combination gets into play, and you have this pretty simple, where is that? I scrolled past it, um, scoring in here, and then you have a table. And so essentially what you're returning is not a,

a, uh, list of, uh, score, documents scored order, it's just the rank order. And so that's why there's a rank constant there, um, because you're just getting the rank of the documents. And you can see in here, where is it working?

Yeah. Um, it talks about it in this table where the, the documents come in in different, um, different orders, but by ranking them, then you hopefully get the best answer for you. But that doesn't always work, and actually James mentioned this a second ago, filtering is a really powerful and effective way to help along. So if you can filter your data by keyword,

geolocation, date, anything that gets that dataset down to a smaller candidate pool, you're gonna have better results usually, and because you have a smaller thing. You're not just casting a net over everything, you're filtering it down and you're, you're narrowing it down, right? It's like if, if I want pizza tonight, you know, I'm not just gonna search for food, or maybe. But I can, if I did search for food, I can then filter by pizza, and at least I'll just get pizza results or f- you know, things like that. So,

um, that's where filtering comes into play, and it's, it's something Elasticsearch does very, very well and, and very quickly, and we have a lot of different ways to do it. But then it gets into that doesn't always work, and so you can get into linear combination. And there's, we talk about min-maxing here, there's different things you can do, um, but this is where you're normalizing the scores. So instead of taking the order, you're gonna then instead just normalize the scores so everything has the same scoring range from

zero to one, and you can put them together. And there's a little bit of the, the tuning examples in here. It's, uh, again, it's kinda the, um, intro level to tuning for sure. But you can tune the weights and you can see, um, which one works well, and this is where judgment sets comes in, and this is where things like, um, you know, we have a lot of different workshops and things like that and the, the intelligent query layer and things like this do this more

automatically, not in, in sophisticated way. But essentially you have your judgment set that a human, these days you can get an LLM to help you, but, um, is based on this query, these are the exact documents I want, and then you run your experiments and you see which weights work for you to give those results as, as accurately as possible.

Uh, am I through here?

Do you have something you want to add there? Sorry.

And I will add one more item, because this is actually a technique that we used in the intelligent, um, intelligent query layer architecture that, um, that, that we were talking about quite a bit, uh, last year and the year before. Um, Elastic supports the concept of search templates, so you could technically have three different search strategies, one that uses RRF, one that uses linear, and maybe another one that uses, um, a combination of linear

or RRF with a re-ranker. And, um, if you have a layer on top, let's call it an, an LLM that's got a simple prompt that says, "Based on this type of query, I want you to route to this search template. Based on this other type of query, I want you to route to this search template," you can literally have the equivalent of a what I'll call poor man's, um, auto-tuning, where you're literally toggling between different search strategies based on what the search query that's coming in looks like, and you have something, and it doesn't

have to be an LLM, but you have some way of interrogating that search to decide which search template to invoke. Most developers that are building out an advanced use case where they don't have a single strategy that they can use for all queries are gonna end up with something like that under the hood. It's just a matter of how hard it is to actually code that up. And in a case of Elastic, um, a lot of folks don't know about search templates, but that is a powerful mechanism that can be used, uh, to reduce the amount of boilerplate code you have to write for that type of architecture.

Not covered in the labs, but noteworthy to at least have an understanding that that, that concept exists in Elastic.

So the table at the end, again, is kind of the take it with a grain of salt, but it's a good way to get started, right? Like, if you don't know your, your data too well or it's all, you know, kind of an unknown query mix, you can start with RRF. It does work pretty well. It's not something-- We didn't invent RRF, but we do use it, other places uses it. Um, and then beyond that, you can start getting into, um, other ways to com- to combine the data. Um, but you don't always have to do hybrid search. Sometimes semantic search is enough, sometimes

BM25. Um, but usually we, we find that there's some combination of, of semantic and, uh, lexical search that helps the, um, the retrieval work most effectively.

And then lab four is where you, um, despite the title, you do put it into a model, but we're gonna show you how... Again, as I said at the top of this, LLMs are extremely, you know, smart's not their best word, but, you know, they are e-effective at giving you answers. Um, but you-- anytime you have to feed them data of-- that they're outside their training knowledge or anything, you can pretty confidently

get them to give you the wrong answer by providing them the, the set of data and say, "The sky is orange." And then usually it'll tell you the sky is orange. Um, there's ways to around that, but this is where we kinda show you that retrieval is still important to agentic, um, answers.

Quick show of hands. Are there any folks that are not actually past lab three yet?

Okay, good. People are cruising along. Awesome.

And so it's worth mentioning too, this is where we actually get into Agent Builder, um, Elastic's agentic platform, which has, um, the ability to, obviously to run agents, to run tools and skills. Um, you can hook into lots of LLMs. The Elastic Inference Service has, you know, all your favorite LLMs available to it. Um, but you can, again, bring your own, um, your own LLM. And s- and it's able to then run these tools directly against, uh, the data within Elastic.

And actually now, it's not covered in the work- in the, the workshop, but we have w- uh, workflows. And workflows essentially are anytime you have a s- an ordered set of tasks you want completed, you can have decision trees and things like that, um, the agents can call these workflows, and they can also reach out. So you can call out to other systems, you can call MCP tools, you can call APIs, call anything to your heart's content. Um, but again, outside of the scope of this workshop, but since we have a little time here, this is where the agent has the ability then

to start doing things. So not just retrieving your answers for you, but there's a whole other workshop that you could go to where because of, uh, you know, an incident, for example, the, the observability, the, the OTel trace data you bring in, it can decide, "Yeah, the system is not-- is running out of, of resources. It needs to scale." So you can call a workflow, for example, to auto-scale your Kubernetes environment. It's just one example. But this is where we actually get into Agent

Builder and, uh, it walks you through a little bit of the, um, the capabilities. So yeah.

And one of the more interesting things about Agent Builder for folks that are software developers is the fact that everything that you see in a Kibana user interface in Agent Builder is, is programmable, programmable via API. So if you decided that you wanted to do a quick prototype and you're using our Agent Builder mechanism within Elastic, and you think, "Oh, I have to rewrite all this code from scratch now and build out a real app," you don't have to do that. You can reuse those same endpoints

in your application and make the same calls in a custom user experience that, um, that you control. And then you have the added benefit if you decide that you wanted to pull those Agent Builder components out and wire in another framework or another, you know, another set of APIs that you either build or use a different framework for, uh, we would be compatible with that as well. So the goal for us is to provide as much turnkey experience as possible but not, uh, lock you in so that you're forced to build applications the way that we want you to build them. We're very flexible,

almost to a fault. So when you ask us a question, oftentimes the answer starts with a, "It depends."

Did anybody come in recently that doesn't have the URL? Sorry. Thought so, but start out there. Um, so it's in the upper right. It's that short E-L-A-S-T one. And in case anyone, um, came in later and didn't catch me say it, this will be available for a week, the actual workshop itself. And then the blue, by James, um, link is the, the repo that has-- it has all the, the notebooks in there. It has

all the other code in there, but the relevant stuff you'll use without Instruqt is the notebooks. You can hook it up to any Elasticsearch cluster you have, free one, uh, well, sort of, if you, if you hook up an inference service. Um, but it'll load the data, and you can run through it all. So that, that'll be available until someone deletes it. Deletes me.

By the way, thanks to our distinguished architect over here, Sunil, we, uh, wanted to make sure that we give you bonus content, bonus workshop materials. There are additional workshops available online that have the same format that we used here today, where there basically is an Instruqt environment. It's pre-pre-built so that you can walk through the concepts and learn more about how Elastic works under the hood. We highly recommend that you hit that URL and, and use the

workshops. It's essentially free training, and once you complete those workshops, you'll probably increase your market value by, I don't know, hundred, two hundred thousand dollars a year. It's a guaranteed pay raise.

Okay, so while people are working on that, I'm gonna do two things. So one of the r-- Sorry. Sorry. I kept talking a couple times about, um, you know, providing new information and up-to-date and, um, fresh and whatever proprietary information to the, these LLMs as a way to augment... That's where the augment, the A in RAG comes in, augment their knowledge. The other thing that comes up a lot, and this is coming up more and more in the f- in,

well, everywhere, especially as agents start running around and, and doing things even faster, is being able to cite the sources. So models are really good at coming up with information, and oftentimes it is correct. But what they're also great at is lying to you because, not because they're specifically trained for it, but what happens is if there's a s- very thin li-line between, say, two other lines that you ask a question about, and it doesn't know that specific answer, but it knows enough o- information

around the answer and about other topics and related, it can come up with a extremely convincing, um, you know, response that, unless you're an expert or really know the actual answer, can, can trick people. So, all that to say, one of the things that's in this lab, it's kind of in the middle and it's up on the screen right now, is the ability to have an LLM cite its sources. Most models do not have the ability to cite their sources because it's not a database. It's, it's-- these

models are not doing a table lookup and pulling information. It's generating tokens based on its training. Anyways, so by providing the actual answers to the LLM and having it parse it out and put together a human-friendly answer or whatever format you want it in, it can also then cite its sources. This is extremely important in things like legal, which is, you've probably seen in the news come up a lot of times where lawyers are getting in trouble because they're having cases, and then they're making up answers, citing cases that don't exist

or twisting the answers. So if you can have the model process, you know, large amounts of data, complex data, but still cite the answer, um, it makes them a lot more, uh, useful and trustworthy to a point. So when you go into... I have to restart this.

As James mentioned, um, one of the nice things about the Agent Builder platform is that it is all also API-driven. So that's kinda why the, the lab in this one was a little bit reversed to show that, where you can build all this programmatically. So you can build skills and tools and agents and, you know, through... I built this all programmatically, and that way you can create it, tear it down, scale it, change it as you need, um, but

still use all the capabilities built into the stack and things like that. Um, so we'll go in here, and then you can, um... Do I have to ask a question? You know, we can go into overview. We have a lot of skills out of the box, so these are the skills. And I'm sure everyone here knows this 'cause you're here at an AI conference, right? But a skill has a lot of different things, but essentially it's explaining to, um, you know, a model or an agent how to perform a task in a certain way. Um, when, when

tools should be used, how to create presentations, how to create other services. Um, but it's anything you can write down to kinda help it out, so it doesn't have to just figure it out by itself. Um, some things it's just ways to make it faster. Some things it's when you want very specific outcomes, um, if you wanna give it core instructions as, as shown in here. Um, and so we provide a lot of out-of-the-box things, um, skills for you, like if you wanna create graphs, visualizations, searching.

Um, but we also create tools. So the difference then is a, a tool is performing an action. So a tool that we prov- you know, provide here out of the box are things like, uh, running searches, listing workflows. I talked about workflows. There's the way to actually reach out and, you know, query external services or run things in a very specific, um, deterministic manner. But you can create your own tools. So if we go into the, the agent that you created, for example, and you look at the overview,

um, it's a very, you know, simple agent, again, for the purposes of today. It has one skill. It talks about how to diagnose an, an, um, an, an issue that you're looking for. So it's pretty, pretty, um, pretty short here compared to the out-of-the-box ones. But this is where you can provide as much information to, to help it out. So it helps to find the cause, find the fix. It talks about, um, looking at the error codes. The other things that it can do is... I think it has one tool.

Yeah. So here we're, um, we have a search tool. But if I add other tools just to show... Can I create a tool? Create a new tool. Um,

this is always a fun question I like to ask. Who here knows or has heard of ES|QL within Elastic?

Okay. Oh, we got one. Nice. That's about what I expect. I was happy to hear, though, that a lo- a lot of you are familiar with Elastics-- in Elasticsearch, so that, that's good. Um, ES|QL is a newer, um, query language within Elastic. It's the ES pipe QL, so it's a, uh, SQL, near SQL-like, um, language. So you have a SQL-like syntax, and then essentially the pipe is in there to, you pipe commands together. You, you run the

thing, it passes it to the next pipe, and so on and so on, um, do that. So you can have ES|QL tools to query, um, you know, documents in Elasticsearch. Index search is kind of an interesting tool, or, uh, yeah, tool. So ES|QL is you're running a specific query. You're, you're querying, you write down the query, you get back specific fields. Index search is kind of this, it-- I almost think of it as a documentation tool. So you're, you're telling it for a specific index or set of indices, this is what the index is

for, this is what the field mapping is, these are what the fields are. If you have, say, a product index and a pricing in-inventory index, it tells how to combine them. If, like, with ES|QL, you can join, uh, indices together. So it talks about that. And agents, because they can do round trips and do exploration, they will generally be able to figure out, um, you know, the indexes in your, in your index, in your cluster. So if you just ask a question and it doesn't know anything,

it's gonna do this query search tool, and it's gonna look at the mappings or some mapping tool. It's gonna find all the indices and then the mappings, and it's gonna try to guess, like, what sounds like a product thing, and then it's gonna try to put them together. Anyways, it'll round robin for a while, and that costs tokens, and we all love token spending. So this is a tool that helps speed up that dramatically and kinda help point the agent in the right direction. Workflows, I think I talked about this a few times, but we can run things in a, uh, orderly fashion

and reach out to other systems. And then MCP, if you wanna call out to an external MCP server, you can do that. I will mention at this point also that the tools in Agent Builder, both the out-of-the-box one and custom tools, are exposed as an MCP, so Kibana exposes an MCP, uh, server. So any of the tools in there you can use, you know, if your agent has, um, wants to connect via MCP. And the last one, so

you do this in the lab, but this shows basically how the agent can then do multiple, um, multiple loops. So it has to do, first it loads the skill to see how to answer the question about, um, keeps dying with a one thirty-seven, why does it happen, and how... So this is a two-part question, right? Why is this happening? Why is my cluster dying with, you know, the one thirty-seven, uh, exit code? So that has to figure out the answer to that. And

then how do I change my settings to, to prevent it? So it's a two-part question. And again, so it runs that skill, and then it starts to find the answer. So goes a couple loops, and it diagnoses your problem of, you know, tells you why this one thirty-seven's happening. It's actually, it's the O, the fun out of memory manager running and killing you. Um, then it talks about how to, uh, tune your JVM options to, to fix

it, but talks about preventing so. Because the answer within, was within Elastic, and it had the ability to, you know, query it and then run multiple tools and things like this, it's able to kinda get you that answer and, uh, get you, get you on your way. You had something?

Yeah, and the one thing I'll add is that if you're, if you're in active prototype mode and you're trying to, to get to, um, search logic or query language, um, logic that, that actually solves for your use case, this is actually a, a pretty nice developer tool and debugging tool to find out what, um, what searches will work to retrieve, uh, the data that you need. And the last thing I'll say about, uh, I'm a big fan of ES|QL. I started out a little bit skeptical

because there were some features in Elastic Query Language that weren't in ES|QL. I think that it's feature complete enough, uh, for you to, to at least take a look at it, especially if you're building agentic search use cases, 'cause there's some really cool concepts in ES|QL, uh, that lend themselves well to, um, to solving for, for certain search, um, tasks better than Elastic Query Language, which you should consider a superset of what ES|QL is today. Um, and the other thing about ES|QL that makes it fantastic is,

uh, the ability to have parameterization wired into the MCP tool calling chain and, and wired up with Agent Builder, which is, um, a fantastic add that technically is doable outside of ES|QL, but the way that we have architected ES|QL and our Agent Builder and our MCP endpoint, it is the happy path to a lot of the, um, let's call them agentic search use cases that you might have. So if you have not looked at ES|QL yet, highly recommend you do

that. And if you're trying to pay attention to some of the innovative stuff that we're doing around search, first place to, to look is actually going through the ES|QL docs and seeing what new novel, um, retrieval mechanisms we're coming up with, because we're doing some very interesting things with federated search and potentially calling out to different systems within an Elastic Query Language, um, you know, query that, that is not considered traditional Elastic. Um, and that's stuff that's in development now, but you're gonna see that

hit, hit the product, and when it does, ES|QL is gonna be your gateway into that.

And I'm talking about things like being able to do, like, parquet queries on things that are external to Elastic with, with an ES|QL call, which could be incredibly powerful if you're also joining that with data that comes from Elastic.

You can actually see that the, the agent used ESQL for this, uh, this call. And so this is what it looks like. Um, so it can do things like fork off, and then, so f- this is where it's combining, uh, two of the, two of the search threads. That's what the fuse is down there. And, um, so it's able to ask these questions and then return the answer. But you can see it's, you know, there's the pipes that can, um, y- again, you run a query, you run a phase of it, essentially, get the answer, and it

moves on to the next and next, uh, pipe, um, language like that. So it's, um, yeah, it's very parameterizable. Is it?

Parameterizable, and it's-

Thank you

... it's trying to accomplish all the things that the Splunk query language tried to, you know, to do with, with Splunk in its heyday. Uh, and then, um, and then also marry that with SQL concepts, which are obviously tried and true, and I think that the, uh, the last thing that you'll see with ESQL is the queries are, are more terse. They don't have as much markup in them, uh, to actually get, to get the retrieval right, which is one, one added benefit to that piped, uh, construct. And there are other, other

companies that, that leverage the same technique in order to build out, um, you know, search. Usually, it's for unstructured data sets, logs. Um, and, you know, and I'd say that that was the primary reason we started building ESQL was to build, you know, the best, best-in-class search experience so people could get rid of all their Splunk stuff. And, and now we've pivoted to where this construct is so useful that it should be available in all the pillars of Elastic, search, security, and observability. And it's a first-class

citizen construct for, for, for all three pillars going forward with the primary, it being the primary mechanism for where we're gonna innovate.

And so you can see when it's doing its, one of the, the hybrid search query, it responds. It shows you... So you have the, you have the audit trail essentially of how it got to these answers, so not only citing the answers all the way at the bottom, and you can click on them, and if you wanna deep dive into the fun world of JVM settings, you can do that. Um, never doing that again. But, um... Just kidding. One of the nice things too is then it'll show you, so we're limiting five, and it, it does five, sh- it tells you it

found five results. Um, doesn't clutter it all up in the, in the UI here for you. But you can click on that, and you can hop over to... This is Discover. It's kinda where we discover documents. It's the kinda normal searching around interface of Kibana. It's changed a little bit over the years if you haven't used it in a little while, but now you can do ESQL in there. And you can see here, then it has the five documents that it came back with. And if I close, like, some of these panels, builds a nice

little chart for you. I don't know if that's the most helpful chart, but you can see then with the query up at the top, you have the actual documents, and go in and see the full thing. And so here's the body of it, and so we were searching, um, part of this. Oh, here it is. Searching for the URL. So the URL, again, this is because we provide the URL as part of the document, we're able to, you know, provide essentially a link

to more information without having to just guess.

I think this room closes out at the four twenty mark. Is that correct?

Yeah. Timer says 18 minutes till the-

Eighteen minutes. Which means you have-

Don't let the hammer come and hit us.

Yeah, which means you, you do have a little bit of time to ask questions.

Yeah, there is the-

Yeah.

Yeah, feel free to ask questions. Sorry. Um, there is the lab five about, um, re-rankers. These lights are, like, burning my brain. Um, so re-ranking in there is, is an interesting, um, thing that I think a l- a lot of people don't kind of fully understand why you wanna use it, and so that's why we added this on, um, to kind of explain why you wanna use it. And if this

spins...

Uh, see if that comes up. Maybe it's my thing. Re-ranking is taking the... So if, if the queries that we've been doing here and everything is the first phase, call it, of, of retrieval, the second phase you can do retru-- uh, re-ranking. And so if you're running vector search, semantic search, hybrid search across a first phase and get back a large set of data, then you can take that data, and you can

re-rank it. And re-ranking has... I wish this thing was, was working. I think it's me.

Re-ranking combines the, the query and the, the documents at scoring time together. So normally with, with vectors, you, you know, you ingest all your data, whenever that is, and then at query time, you query it, and it just does, um, HNSW or, or one of the disk algorithms and returns a set of documents. But the, the query and the, the documents aren't really scored together. Re-ranking combines those

together, but because they're running together at, at that query, that pha-- uh, second phase time, you can't pre-embed. So you pay the penalty of having to run inference at, you know, at, at query time. Now, the way you do this then to kinda help this along is by not re-ranking your entire corpus of data. You run the first phase to get a smaller set of data and then re-rank with a model to

then, you know, re-rank that, that small set of data.

Sorry. There we go.

I was trying to show an example instead of just talk.

Oh, no.

There we go.

My lab was dying.

So if you get to this in here, you can see toward at the end, um, again, kind of the point I hear about when you can add a re-ranker if you have that, that, the latency budget. If you don't, if you can spend a, a little bit of extra time to let this second phase run in generating embeddings because that precision of the, you know, if it's answer one versus answer two, if that is a critical step, that's where we're paying, you know, the, the slight, um,

latency penalty comes into play. If you just get your documents good enough in the first phase and let the agent go after them, then you probably don't need a re-ranker. But it's when you need to have that or- that specific order come into play, um, come in and, um-

And we see that a lot with agentic search use cases. We're seeing a lot of situations where folks are testing the evaluation based on the first, second, and third document chunk coming back, which gives you a very small window of opportunity to get the answer right if you're just handing a small set of candidates to the, to the large language model to answer the question versus a larger candidate set and then, and then asking the model to get it right and, and that's where a lot of relevancy issues come in. You, you're passing in too much context.

It's a multiple choice where two or three answers look equally right to the LLM, and it just picks one and, and if it picks the wrong one, then your, your users and their beta testers for your new agentic search system complain about the search not working. Very common. Very, very common and, um, you know, oftentimes a re-ranker like this will, will actually be very, very useful.

And so there's two points or two types of, um, of re-rankers, at least, you know, that we support. So there's pointwise. These are what, uh, most re-rankers are today, and this is what the, the V2 model is. And so the pointwise, it, it still combines the-- it takes the query and the document together, but it just scores them independently. It's only looking at the, the query and the document, nothing else. List-wise, which is what is in the Jina v3

model, takes your candidate set and I think it right now, for example, up to sixty-four documents or sixty-four candidates, and it looks at them all together. And so that way, way if the, if the answers essentially or the, the potential candidates are all very similar, like they're all pretty much talking about the same thing, that's where having the list-wise can edge out pointwise because it can see the nuance, see-- it can see the nuances,

um, within that list because it sees the multiple documents together. Um, it does take a little bit of extra, um, uh, memory or, or time because, again, it has to generate embeddings for all those candidates all at once and hold them together in memory and run, um, run its evaluation across them as opposed to just firing one at a time and do that.

So if your, if your data is pretty diverse and where the answers come back and it's, you know, it's a pretty distinct set of candidates where, you know, one, two, three, four, five are all pretty different enough that the pointwise is hitting it... Oh, sorry. Sorry. There we go. Um, then pointwise usually, pretty much usually works out. But again, if you have that set where the, the potential candidates are all extremely similar,

list-wise is probably gonna help you out for the best.

And somewhere in here, because we wanna be helpful, you don't have to run, um... While it is a, a two-step, a two-phase, um, you know, phase one is query, phase two is your ranking, you don't have to manually, um, have those as separate, separate API calls. We can do it all in one shot. Um, you do it in the lab. It's in the retriever. You can, you can do it-

Can you run SQL?

Can you re-rank any SQL?

Yes.

Yeah. Okay. Um, so you can do it in SQL too. But one shot, basically you have the re-ranker. You tell it what re-ranker you wanna use, you know, the model and the query, and it runs both phases for you. So that way, again, you're not having to manage the inference setting separately. You're not having to call the inference API, then query, then re-rank. You can just fire it all in one shot, and we'll take care of it for you.

So I'm gonna put up that slide again with a few of the links. Again, so the upper right is what we've been doing today. That's available for a week. The bottom blue reference repo is where the, the code for this lives, the notebooks, and things like that. And then there's the two red ones in the middle, um, Sunil's secret Elastic environment.

Oh, man, I typo'd it too. Um, everyone's gonna take a picture of it. So and then the other-- God, I'm gonna blame-

Just proves that we're in a live session.

I'm gonna blame the lights.

And that we're just, you know, we're doing stuff on the fly.

I wrote it y- uh, still got it wrong. Okay. Um, and then the other workshop's online. So we do have a bunch of other workshops that are online that are completely free. And so things like working with streams, if you have streams of data coming in, it's extremely, um, efficient way to handle that data. There's another RAG one, lexical search, vector search tutorials.

If you have anyone's doing, uh, observability work, so things with open telemetry data or looking for root cause analysis, things like that, we have labs on that. So a couple observability labs. Should be some security ones in here. So security's a big platform, or a big part of our platform. Um, doing anything from like attack discovery and detection rules. I know there's new ones coming out too, um, as we add in

more and more agentic capability into the stack.

Any questions? All right. So l- let me ask a question real quick. If you hang out in here, do you automatically get a seat in the next workshop?

I'm just curious if that's why people hang out. Okay. I'm trying to judge who's still working and who's just hanging out, which is totally fine. I... Yeah. I get to leave either way.

We're gonna assume that most of y'all are hanging out and that you're not, you know, waiting for help on anything. But if, if you are, you've got another seven minutes and fifty-seven seconds.

Sunil, you wanna demo anything for seven minutes?

Hopefully, this was a good use of your time for the folks that are still in the room, and we'd love to get your feedback. We didn't tell you this, but there is a feedback, um, uh, data entry point at the end of the workshop, so when you finish it, you do have the ability to tell us how awesome the workshop is or, or how much you didn't like it. And if, and if you say bad things, then it will hurt our feelings, but, but we will make the workshop better for others after the fact.