Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop) — Taylor Jordan Smith

[on hold music] Hello, everybody.

Um, my name is Taylor Smith. I am an AI developer advocate slash evangelist slash technical marketing and other titles depending on where I am. Um, I work at Red Hat in our AI business unit 'cause yes, we do also do AI too now.

Great. Um, I'm really happy to be here. This is my first time at this conference, so I'm new to the conference, new to speaking at this conference, ready to have a good time and also relax in about an hour and a half and just learn stuff.

Stoked that this was at nine AM. Um, my presentation clicker isn't working, so I'm gonna be a little bit glued over here. But to kind of overview what we're gonna do today, I'm gonna talk a little bit about kind of the issues of setting up a large language model in production and the reason why you need evaluations

and benchmarks and all of these things, and why that is so critical. And then I do have some hands-on activities that we'll get to do to use some evaluation, um, methods and benchmark tools to kind of get a sense of what is out there to use, how we can use those tools, what that might look like to

put it all together for an actual production system. So

ready? Are you excited?

Mm-hmm. Yeah.

Yay. First thing in the morning, love that. I'm glad it's nine AM 'cause that feels like a reasonable start time. I don't like that eight AM stuff that happens.

Um, setting up generative AI tech in production, if you think about it, crazy pants, okay? And there's so many things that could go wrong. This is such a complex type of technology that is very creative.

Setting this up to be scalable and reliable and safe is challenging, okay? That's why we need evaluations, that's why we need all of these tests and we need to be careful, we need to understand the technology we're working with when we're looking to implement this as well.

So hard to do. Most organizations, so they don't typically start off with a multi-agent framework, right, and go crazy. Typically, they're gonna have a kind of standard repeatable path to maturity.

They might start out with, "Okay, how can I automate some things with AI? Can I have like a chatbot type of situation?" That's kind of the standard that everybody starts out with.

"Okay, I maybe wanna implement a RAG setup. All right, let's start to look at agents." To-- For, uh, on the Red Hat side, we're still dealing with probably like the first three phases of this maturity with our customers, and that's where a lot of people, um, still are.

But we get to talk about all the cool advanced stuff at this conference, so we can plan ahead and think about what we want to implement. But enterprises, they need to take an incremental approach to do this successfully.

GenAI models, they have a number of drawbacks, right? And we all know about these things. Policy restrictions. Typically, if you're a developer at a company or have whatever type of role, you're restricted in the type of AI that you can use.

Red Hat was just, um, allowed to use ma-- what-- Gemini was just made available to us, but before that, we weren't really allowed to use anything

officially. [chuckles] Um, so the company policy restrictions of the tools you can use and the AI you can use is pretty locked down typically. The legal exposures and risks of these models, you know, there was the glue incident.

Um, they're gonna say crazy stuff sometimes. How do we guardrail against that and how do we protect our customers against that and account for that when that does happen because it's hard to, um, completely avoid?

There's the bias and discrimination issues. Most of the internet data is still largely Eurocentric and US-based, so the models trained on this public internet data, of course, is gonna be a little skewed, right?

So we need to be aware of it, just like in re-regular everyday life. We know that bias and discrimination exists. How do we account and kind of provide guardrails to make sure that we adjust and prevent what we can?

Cost and performance, that's probably kind of a baseline big one. How much these GenAI models cost to run at scale and production, and the performance, throughput, latency, et cetera, that we need to account for when we have these production systems set up.

And then the knowledge cutoff as well is another limitation. You know, these large frontier models have a knowledge cutoff 'cause they're not consistently trained, so you might be working with a model that was cut off a year ago, so it's not gonna have that up-to-date information, which is why they, you know, implement RAG systems and agent systems

to look out into the internet for more up-to-date info. These are just some of the kind of drawbacks we need to be aware of and account for.

Inference at scale. No matter how-- I'm gonna go kind of into these categories a little more in depth. No matter how good your model is, if it's not fast, if it's not reliable, if it's not affordable, you're screwed a little bit from the get-go, okay?

So this graphic, it shows a classic bottleneck type of scenario. You got concurrent user requests, which might be represented by those green, yellow, and orange dots flowing into a system.

But then traditional inference runtimes typically can't handle that kind of load officially. Um, to serve real world traffic, whether you're powering a customer support agent, a developer type of copilot system, or maybe a RAG pipeline, you need an inference engine that's purpose-built for scale.

So that's where inference runtimes like TRT, SGLang, which I know we have a session on or at least a couple sessions on, vLLM, um, that's where Red Hat's kind of focusing.

That's where those type of production-grade inference runtimes really need to be utilized.

There's a lot of pain points with inference that I just want to double down on, and some of the activity will be benchmarking and evaluating, um, these types of metrics.

Model inference performance evaluation under enterprise-level workload scenarios, it's very complicated to actually evaluate this appropriately. It requires manual setup of evaluation runs with various parameters you have to test.

The compute load just for performance evaluations as well is also pretty taxing. Um, you have to make sure the datasets that you're using for benchmarking are compatible with the models that you're using.

The resource optimization and identifying sizing so that you're efficiently using your hardware appropriately for whatever model size you're using, that's a big challenge, um, for enterprises today, making sure they're efficiently using their GPU investments.

And then actually cost estimating is a little bit of a black magic thing. Like, it's really hard to do that appropriately. Um, you have to, like, backwards math map inference performance to tokens, and it's a whole thing.

So these are what enterprises are trying to achieve, and it's hard.

Um, just a little kind of more examples of the challenges. So we have, you know, this is just an example of stable diffusion bias. Like I said, most of our data is Eurocentric.

This is going to be there, so how do we-- what tools can we use to provide guardrails against this type of behavior?

The glue incident, this was because there was re- something on Reddit was, like, it-- like a joke, um, satire, and, uh, this AI overview tool used that information and didn't have the right mitigation techniques in place to identify that's satire.

So then it came out in that, um, AI overview, uh, suggestion.

And then we have this, um, like, mad situation where a lot of the-- we're getting into a lot of synthetic data on the internet, and each generation of these AI models that come out are consuming more and more AI-generated data, which over time is going to get you-- Oh, there's music happening.

Um, further away from that original human-anchored data. So this is going to lead to a loss of output diversity, a loss of pr- a loss of precision. This would be an area where you would need to use those kind of accuracy evals to mitigate aga- and identify that this is occurring.

So just to cover, of course, Google and the Stable Diffusion project, they have m- introduced additional evaluation frameworks and mitigation techniques to address this. Um, just like any time we have any story like that, right?

They are certainly working to make sure that that does not happen again. We don't know with the closed source, um, AI offerings how exactly they're doing that, but we can speculate.

You know, the AI overview technology maybe introduced more RAG mitigations to where it, uh, helps to identify that that was satire or whatever the case is. Um, it has more safeguarding triggers.

Um, the Stable Diffusion model likely introduced some level of bias mitigation guardrails. So we need-- So of course, they've been working to fix this, but ideally, we don't run into this, and we prevent it ahead of time before a model release or before an application release.

So how do we prevent these kinds of issues at scale in production environments? I want to just look at a couple of definitions because sometimes evaluation and benchmarking are terms used, um, kind of-- they kind of conflate a little bit, and people kind of use them for whatever they want.

So benchmarking is just a subcategory of evaluation. Evaluation is a comprehensive process to assess a model end to end, and it could include a lot of different kinds of evaluations about a lot of different components.

Benchmarking is very specifically controlled specific datasets and specific tasks typically used to compare models against one another. So this would be like a latency score that compares different hardware setups and different models or, like, the MMLU benchmark scoring, things like that.

We'll look at both custom evals that aren't benchmarking and also some benchmarks in our hands-on.

These are just some examples of what is typically considered a model evaluation versus a benchmarking specific test, just to kind of get a little bit of a sense. Um, but again, like, there's so many types of evaluations, so many tools.

You can customize it in so many different ways, but this kind of helps hopefully a little bit with the definitions.

So hopefully kind of seeing all these challenges with GenAI, we understand that this is a critical process. Um, you need to manage risk for customers. Like, there's less of a concern, right, when I'm tinkering on my laptop, whatever, you know.

I see something weird, who cares? But when we're talking about a production-level environment, we're serving thousands, whatever, customers, we need to think about these things obviously more in these types of scenarios.

The credibility of the company, when those stories come out, that takes a hit for a good chunk of time. Um, and you obviously also need to continuously improve your evaluation frameworks as well because you're not going to catch everything.

You need that CI process to make sure you are continuously improving your evaluations and benchmark setups.

It's also going to very much depend on the type of system that you have, what you set up. So again, it's very much there's tons of tools. This could look a lot of different ways.

We'll get a sense of it today. If you have a RAG setup, you're going to be maybe focused on a-- the RAGS, um, benchmark or evaluation tool, agents. You need to look at function tool calling capabilities, et cetera.

You can kind of get a sense there's going to be specific metrics that you need to set up and look for depending on the system that you have, which requires a lot of planning in advance and kind of architecture scoping.

I'll give an example of a RAG use case, and also it's incremental too, 'cause you could-- And I'll, I'll talk about this a bit. Like, you could literally evaluate every single part of things, but that's going to be, um, time and resource extensive to set up immediately.

So you likely wanna take an incremental approach with these types of setups. So you might start out with, "Okay, I'm just going to evaluate the chunk retrieval, my retrieval application in a RAG system.

I wanna set up some kind of evaluation test there. I might just want to set up a latency throughput eva- um, benchmark test for my LLM output." You can start with those kind of incremental approaches for specific components, and then from there, based on priority levels as well, branch out into a full system eval that covers all

the components, the integration layer of how the components work together, the UI end-to-end experience, and kind of have a software engineering kind of test period approach to this, where you have that unit test layer kind of approach at the bottom, um, integration layer in the middle, and that UI end-to-end at the top.

And you can take that layer by layer as you're building this evaluation framework for your systems.

So there-- we created this-- or we-- there is this pyramid [chuckles] also for model evaluation that represents the same kind of setup for, um, that, that software engineering pyramid, um, represents as well.

So the base layer and, uh, very base layer is the system performance, 'cause like I said, no matter how good your model is, if you don't have fast throughput, you aren't able to handle concurrent users, you're gonna be in a bit of a pickle.

GPU utilization, et cetera, you need to make sure kind of the basics are handled, uh, the, the-- as the main kind of event. Um, and we'll talk about-- That'll be the first hands-on activity is, is evaluating the system performance.

Formatting might be making sure it's, um, religiously giving you JSON output that you need for your application, something like that. The factual accuracy, which we'll also talk about, um, in one of our hands-on, that would be like the MMLU benchmark, so evaluating that it's performing well on various subjects, kind of standard large language model accuracy, as well

as if you've fine-tuned a model, potentially making sure that it's accurate based on the information that you fine-tuned that model on. And you kind of go up from there into, you know, safety, bias.

S-- There might be specific custom evaluations that are very specific to your application. So gives you kind of a sense of the tiered approach that can be taken here.

So we're gonna talk first system performance, and we're gonna have our first hands-on around this. We're gonna be looking at GuideLLM, which is kind of a new project that is associated with the vLLM inference runtime project.

Um, we're gonna use that for system performance benchmarks like latency, throughput, and you'll get a little bit of hands-on there. The general user flow there is like you, you know, you select your model, you select your particular dataset that you wanna use to test throughput and to test inter-token latency, time to first token, those types of metrics.

And then GuideLLM allows-- gives you a nice kind of, um, in-terminal UI to visualize the results of that. Uh, and then once you get kind of the results that you want based on your use case, again, then you're ready to deploy.

We-- You'll see this in the hands-on as well, but you wanna test based on the use case, and the pri-- one of the primary ways you test via, via GuideLLM is adjusting the input and output tokens.

So if you have a chatbot use case, a RAG use case, you can adjust the, um, input and output token levels based on your use case, and you'll see that in the hands-on and have an opportunity to kind of play around with that, depending on what you're most interested in.

So we're gonna start with the hands-on. That is the link. That red.ht/evals is the link to the workshop. You will be signing in with your email. I have no marketing game.

It just requires you to do that, so I'm not gonna haunt you after this. Um, you'll put in your email, and that is the password.

And let me... Hold on. I just wanna show you. Well, I don't wanna take up one of the systems, um, otherwise I would show you. But you're going to get instruct-- Once you are in the workshop, you're gonna get your instructions on the left-hand side, and you're gonna have two terminals avail-- two terminal sessions available to you

on the right-hand side to the same system. It's a RHEL system. Um, the instructions will overview what the system includes a little bit for you so you get a sense.

Um, they each have an L4 GPU, as an example. Um, anything else I wanna call out before we get started? That'll be primarily what you use. It has Tmux enabled, if you like to use that to open up different things, and it gives you some flexibility.

We have three different activities. I'm gonna pause after each activity, so we can have a little bit of a discussion in between. Um, we'll kind of get-- This is my first time running this particular, uh, activity, so we'll gauge kind of the time it takes, but I'm gonna give it about fifteen, twenty minutes for this first one.

Okay, so I'm also pulling up a system, and I'm just gonna like walk through some of the stuff. The initial page is just gonna give you... If, of course, if it loads.

Jesus. The internet. Um, the initial page is just gonna give you a little bit of background and preparing your system instructions, and then my terminals are on this second tab.

And everything is gonna be glacial pace. Um, so the first thing I have to do, these systems don't have the container toolkit installed, so I just gotta do that.

Just some, uh, system logistics because I'm gonna be running vLLM inference runtime in a container, and I need that to work. So that's what that's doing. And then I'm gonna deploy a model with vllm.

I-- So you're gonna have to grab a Hugging Face token. Probably have gotten there by now. Probably most of us have a Hugging Face token, but just disclaimer there.

So incognito window, if you're hitting the Hugging Face rate limit to grab a new token.

So vllm you can also install locally, like if you have a Mac or whatever, um, Linux machine, but we're deploying it as a container here.

So I'm gonna get that deployed. And I'm, um, you can see the vllm serve command at the end, so that's just the vllm CLI tool, and I'm using an IBM Granite model because, you know, Red Hat, IBM.

Um, so we'll be working with that for a chunk of the activity. And it takes a bit for vllm to load the model, so you're gonna be waiting for info, the words "info" like four times in green, and then it's deployed.

What's nice about vllm is that it is, um, it's compatible with the Safetensor format. So... And has anybody used TRT to load up a model? Okay, anyway, it's crazy, [chuckles] um, because it requires you to also convert the model formats initially.

Um, so there's less kind of configuration steps with vllm. Takes up less space too.

So when you do these kinds of system and performance, um, benchmarks, you can make a lot of adjustments like the, um, the input and output tokens that I mentioned, um, for the GuideLLM configurations.

But there's a lot of also configuration opportunities for the inference runtime itself, um, depending on what you're trying to do. So sometimes we'll reduce the max, um, the context window of the model so it runs more quickly, because if it's a big context window, it's gonna be pretty beefy.

There's a lot of knobs you can use for vllm. We're not really gonna touch that, um, this particular time, but just so you're aware. So I have my, my three in-- green infos, so that means it's working and the model is successfully deployed.

So I'm going to, um, get into my virtual environment, which is already in place. And then I already have GuideLLM installed, but I'm going to pip install GuideLLM. And these are copy buttons, by the way, so you can just easily copy-paste things over.

So once I have that up, then this command is set up to just work with the, um, model deployed by vllm, and that I'm just keeping up here in the top terminal.

You can run this in the background, but I'm just not doing that. And this is... So I have my target. Um, the rate type is a sweep of, uh, various benchmarks like intertoken latency.

There's various types of benchmarks that it'll run that you'll see in the output. Um, but these are all things that can be adjusted. You can run one particular benchmark at a time, for instance.

You can take a look at, I think it's, uh, GuideLLM dash dash help, typical type of commands. You can kind of see what the-- where all the knobs are.

And the documentation is pretty good too. So that'll take a couple of minutes to run because I have it, um, set at a rate of five to reduce the amount of time that it takes to process.

So you can kind of get a sense of the output here once it all processes. And I have explainers on the left-hand side of kind of how to read some of this.

Um, the mean performance for each... So we have the, the benchmark info on the top and then benchmark stats on the bottom. Um,

so the-- for the constant rate, the-- on the le- very left-hand side, so those are the number of requests sent to the model per second at that particular rate.

So three-- the constant at three point six three, six point nine three. If I did written, um, the GuideLLM command rate and it was five, if I did rate ten, you would see more lines of that at, at, um, more progressive rates.

And, like, whether or not these numbers are good also totally depend on your use case as well. And I would be comparing... I would have a better hardware configuration obviously in production as well, because again, I'm j-- I'm running an eight billion parameter size or two?

Two, hmm, two billion parameter size model, um,

on an L4, which is, like, okay. But obviously if you're doing anything concurrently and at scale, like, that's going to go bonkers pretty quickly.

So you get like the mean performance, the median performance in P99, which is like the, the extreme level, um, which matters for SLOs and things like that.

And you can also output this into, um, JSON format as well to take a closer look.

So once you reach this, you can try also with an additional... Like tweak the parameters and then maybe compare the results and kind of see what, what that changed if you do a RAG setup.

Um, so I think we had-- I forget what we had, what the initial command said. Um, but you can adjust those based on a different use case and kind of compare and contrast what the stats look like after because it just takes a couple minutes to run.

Who's done with this first exercise? Okay, good. Great. We're a few minutes away from the additional, um, ten to fifteen systems being ready, so.

Just for the sake of time, I- Am not gonna do breaks for kind of discussion in between, if everybody's okay with that, and we can kind of just converse independently, and I'll just awkwardly walk around the room.

Um, so if you're done with activity one, move on to activity two, because there are three activities, and I wanna make sure everybody has appropriate time for each that they would like.

Um, but we will also have the systems up until probably about noon. I don't know, early afternoon. Um, we'll keep them up all-- if you wanted to also go back and look at it after.

So is that good with everybody? We just kinda steamroll power through. Steam through. Okay. That's a different problem. Yes.

So I have a new URL where we have three more available so far, but others are provisioning [chuckles] and I just wanted to go ahead and put the URL up.

It'll come... They'll appear here. And it's the same password, right, you said? It's the same password. Sorry? It's the same password you said?

Yeah. Okay. Okay, so that's the new URL. Three more systems, but they'll-- more will appear.

Um, I put descriptions on the left, um, to explain that, but I just wanted to heads up, we're kinda moving from system performance to now that kind of factual accuracy part of the pyramid, um, so you get a sense of...

So, so we're gonna be doing the MMLU Pro, um, in the second activity. And then the third activity is gonna be focused on safety and bias and more custom evals.

So that's the trajectory of activities. Feel free-- If one is also more interesting than the other, feel free to skip around. Totally fine. Can you explain more about the MMLU Pro?

I'm curious about [audio cuts out] or, you know, like, uh, proprietary datasets, if that's a thing. You can cu- you can basically customize everything because every-- like, all these things are open source, so you can create a similar type of eval in that multiple choice format that MMLU does with your own dataset.

Um, there's different ways to do custom accuracy evals with your fine-tuned data. So we kind of do it in a-- We have-- So as part of one of the products, we, we incorporate an eval for, um, our fine-tuned models on your proprietary data, and we do like a branch of MMLU, essentially.

So there's kind of-- there's a lot of ways to skin a cat, even though I love cats, um, in regards to how to set up the evals and a lot of tools available. [faintly]

Yeah. So yeah, yes. Okay. [chuckles] You can just like fork it and ch- and change the data sources. Yeah, yeah. Okay. Oh, great. Yeah, that's all.

Yeah. Put that link back up. The instructions don't-- Some of the instructions don't look updated as I expected, so I'm going-- what I'm going to do, um... Is everybody in the Slack for AI Engineer World Fair?

I created a, um, Slack channel called Work- Workshop Beyond Benchmarks, um, and I'm going to put

some of the... So activity three, does anybody see activity three? No. Okay. I'm putting content in this, um, in this Slack. Yeah. If people can-- The-- It's a public channel.

If people can go there, my, um... You'll see starting I'm-- the link is at activity two, but then you'll also see at the page for activity three.

It didn't-- Um, approaching activity three and are looking for that. The systems for some reason didn't render my latest changes yesterday where I improved on activity three, but I put the link to it, to my repo in the Slack channel, in this channel, if you search for it.

But there's information about the Slack in the emails that we got about the event, but also on the back of our badges as well to navigate there. And this will be good for after if anybody has any questions, and I can send more info about any particular tool here as well.

I wanted to have a wrap-up moment because we ha- we have about eight minutes, so please feel free to continue working. I put the link to the activities in the Slack channel.

These environments will be available, um, until the end of the day today, so you have time also to tinker around with whatever you want. Um, so you-- Just to kind of recap, we went through the fir-- This feels-- [chuckles]

Sounds funny. Um, the first was at that system performance latency throughput level. Did everybody kinda get through that successfully? Of course, there's-- I tried to include reading material and stuff to kind of look more into things after because it is, it is a very big topic, and there's a lot going on, and there's a lot of terms,

and it is very complicated. Um, so hopefully you can use it as a learning resour-- my GitHub repository as a learning resource to kind of poke around after. But we started there, and then we moved into the MMLU Pro with ML Eval Harness, um, which also allows you to do a lot of other evaluation, um, benchmarks as

a part of that ML Eval Harness framework. Um, I happened to choose MMLU Pro because it took the least amount of time, even though it took still ten minutes.

Um, but there's other ones also that you can play around with, um, within that ML Eval Harness repository. You can see the different, um, evals you can run there.

And then we ended with a safety evaluation with Prompt, which-- with Prompt Foo, which, um, is a tool that allows you to do a lot of customizing, um, and do your own evals.

Like, you can do all kinds of custom tests with Prompt Foo. So I wanted to get you exposed to that tool so that you can start looking around there as well.

That repository on GitHub also has a lot of different examples. So we used that particular, um, safety-focused example. But if you look at the Prompt Foo repository, it's very easy to play around with other types of examples as well.

Um, so we kind of moved up the pyramid throughout the activity, so hopefully you get a sense of kind of how you can layer this approach when you're looking at and trying to plan for how to strategically implement evals across your entire system.

Um, does anybody have any kind of questions or general what they experienced,

notes of import? Does this feel, like, valuable? I'm curious also about use cases and happy to talk to you after too. Yeah.

So, um, I'm, like, super new to evals and stuff.

Mm-hmm.

Uh, so, like, when I'm doing evals, it's like, uh, testing my prompts and, like, my data sensor if I wanna switch, like, models out and stuff. For now, I'm actively thinking about, like, those evals, connecting those actually to, like, my production, like, running, like, use cases to, like, track that, like, my real performance matches kind of like

the drive-

Yeah

... to see what is, like-- is there, like, a word for that concept or like-

Um, so for me, what I hear is the kind of CI/CD automation implementation of an evaluation framework, just like with software engineering testing, is kind of what I hear from that.

Um-

Exactly. Yeah

... you know? Yeah. So I don't know. It, it's evaluation, but in a more CI/CD format.

Like, when it's actually running for, like, customers and stuff.

Yeah. You should have a CI/CD framework that includes these evaluation tests, just like for unit testing setups.

Gotcha.

Yeah. Anybody else?

It's on.

Okay. Thank you, everybody. I really appreciate it. Of course, feel free to... [audience applauding] Thank you.

Feel free to, um, continue to use the repo. [laughs] Message me on Slack. Again, the environment will be up until about 6:00 PM... 5:00 PM tonight. So thank you, everybody. [outro music]