Ensure AI Agents Work: Evaluation Frameworks for Scaling Success

[on-hold music] Well, thank you so much for being here.

I'm gonna start off by saying apologize my voice a little bit. It's a little hoarse today, but you guys are gonna hang in there with me. Today, we're gonna talk about a really important topic, which is, um...

Slideshow mode. Awesome. We're gonna talk about a really important topic, which is about evaluating AI agents and assistants. This will load. Just to set a little context before we, we jump in,

a lot of you have probably heard today about different agents that are being built, how to build it, what are the cool tools out there to go build agents, and today we're gonna actually talk about when you put those agents into production, it's important to actually know how they're doing and evaluate them.

It's super important to making sure that they actually work in the real world. We're probably gonna get a little technical in this talk, maybe a little bit more than some of the other talks.

But hang in there, I think this is important, even at the leadership level, to understand how to make sure what you're putting out actually works in the real world.

Um, so a little bit about me. My name's Aparna. I'm one of the founders of Arize. Uh, fun update on us, actually. Today, we announced our Series C raise. [cheering] [applauding]

So [laughs] um, have a lot of folks who are using us to evaluate agents. So with that, let's jump in. Okay. Well, everyone here has probably talked to you about text-based agents.

So you have this chatbot, whatever, it's making an action, and it's, it's figuring out all these things to do. The cool next frontier is actually voice. AI is already taking over call centers.

There are over one billion calls made in call centers all around the world with voice assistant a-- with, with voice APIs and the voi- the real-time voice API, if any of you guys have played around with it.

We're actually already seeing these types of, um, agents start to take over and revolutionize call centers. This is actually a real production application of a travel agent. This is the Priceline Penny bot.

You can go in and actually hands-free, no text, book an entire vacation using Priceline Penny today. So we're not just talking about text-based agents anymore. We're talking about multimodal agents, and it's important to address these because the way that you evaluate these types of agents, it's not just evaluate an agent, but also if it's on voice, there's

specific types of evaluations you're gonna need to do. If it's multimodal, there's additional types of evaluations you need to consider. So we're gonna break all that down and hang in there with me for a fun one today.

So before I jump in and talk about how to evaluate an agent, let's talk about what are the components of an agent. You probably have heard different versions of this today, but I'll tell you the language we're gonna use.

One, um, there's something typically called a router, uh, which is essentially what's deciding what the next step an agent will take. There's skills, which is the actual logical chains that do the work, and then there's something that stores the memory.

This is important because i-i-- there might be different architectures of how you're seeing people build these agents out there. Doesn't matter if you're using LangGraph or CrewAI or LlamaIndex Workflows.

There's all sorts of agent frameworks. They all have slightly different ways of building an agent. You might not even use a framework. But what you're gonna see is these common patterns of, okay, that's a router, that's a skill, and that's a memory.

And these different components are gonna have different ways of how you actually evaluate it. So let's first talk about the first one. What the heck's a router? So you can think about a router almost like the boss.

It's kinda deciding, hey, well, y- it's very common to have e-commerce agents in... You probably are all talking to e-commerce agents today to purchase things. Amazon has one. All these e-commerce companies have one.

When you type in a question, like, "I wanna make a return. Give me an idea of what to go buy. Are there any discounts on this?" That user query funnels into something called a router, and that router's goal is deter-- is really to determine, do I call this skill about hitting up a customer service agent?

Do I call this skill, um, to suggest all the discounts we have or suggest products? The router is really kind of the boss deciding who do I tap on to go actually execute the, the ask that the user made.

And the router might not always get it right, but you want it to get it right because then it goes down the pathway of a specific skill within an agent.

So in this case, it will call a skill. Um, so if I asked, "Hey, uh, tell me the best, um," I don't know, "leggings to go buy." So it'll go in, it'll do a product search, and then this is actually the entire skill flow of execution that the agent needs to go through to execute, you know, whatever

the user asked for. Some of these might be LLM calls. Some of these might just be API calls. It just really depends on how people actually implement them. And then lastly, this is an important piece, is there's always something storing the memory because these are usually not just single-turn conversations.

They're multi-turn conversations, multi-turn interactions, and so you don't wanna be talking to an agent that forgets what you previously said. So there's really memory which is storing what it previously asked for and keeping all this in some sort of, um, i-i-- in some, some sort of semblance of state.

So with that, we're gonna get a little fun here. I'm gonna show you, um, an actual example of what this could all look like, a router, skills, and memory.

So this is an open source project, um, that actually looks at the inner workings of an agent. These are called traces for folks who may not be familiar. If you're a-

You know, in leadership or your, your-- this is really what your engineers are looking at when they're actually building and troubleshooting your agent. They're actually understanding what the heck went on under the scenes.

So this is actually an example of a code-based agent. Somebody asked a question like, "What trends do you see in my trace latency?" AKA, what's making my application slow?

This is the router call that we were talking about earlier, where it actually decides, well, how do I then go ask-- you know, how do I then go tackle that question?

So first, you can see here there's multiple router calls. There's not just one router call. This is pretty common as your application grows. You can have multiple times where it comes back and has to decide, what do I need to go do?

So the first time it calls the router, what it does is it actually-- so the router then makes a tool call, um, which is essentially the skill that you need.

The first time, it actually makes a tool call to then go run a SQL query, go collect all of my traces of my application, and go, go run a SQL query.

Then it goes back up to the router, and then it calls the second skill, which is actually the data analyzer skill, which takes all of the traces and the application data, and then it passes it to something that actually analyzes that data.

So in this case, you can actually see there was a router, there was tool calls. We actually have memory that's actually storing everything that's happening under the scenes. And so really just shows all three of the different components that I actually just walked through.

So now that we have an example of a, of an agent with a router and skills and memory, let's talk about how to actually evaluate these agents. Every single step that I just walked through here actually is an area where the agent can go wrong.

For routers, typically, what teams end up caring about is, did it call the right skill? Because if it didn't call the right skill, you know, user asks for, "I asked for leggings, but then it sent me over to customer service," or, "It sent me over to, um, you know, s- uh, something about discounts and deals."

So you actually wanna make sure that the router within an agent is correctly doing the right skill and calling the right skill. So that's the first piece that you'll want to make sure that your teams are evaluating.

So if your teams are building agents, you wanna ask, "Well, hey, what's the ultimate control flow? What's the control flow? And are-- do we have something like a router, and are we evaluating it to make sure that it's correctly calling the right skill between A, B, C?

And is it calling the right skill with the right parameters?" So not just, um, it calling product search, but actually making sure that whatever way you've designed that skill, you're actually passing in the correct things like, um, you know, I want this type of material, I want this type of whatever cost range.

You're actually passing in all the right parameters into what the user actually is, is asking for. Can I get a raise of hands? Have any of you guys heard of-- D-do any of you guys evaluate your agents today, actually?

Is that something you know your teams are doing? Okay, awesome. Are any of you guys evaluating this router level internally? Okay. Awesome. Wow. This is a great group. Okay. [laughs]

This is impressive. Um, okay, let's next go to the next one, which is actually evaluating a skill. This is actually the part where it gets really interesting and tricky because there's many different components in a skill.

There might be-- In this case, I have a RAG type of skill, so I wanna look at things like evaluating the actual relevance of the chunks that were pulled.

I wanna look at the actual correctness of the answer that was generated. But the skill itself can have many different LLM-as-a-judge evals, or it can also have code-based evals that you might wanna run to actually evaluate the skills, the skills of the agent.

And then lastly, this is kind of a really important one that we're seeing teams probably have the most trouble evaluating, which is actually the path that the agent took.

Because, well, ideally, you want it to converge. You call the same skill hundreds of times, and it always takes about five steps or six steps to actually query what the user asked for, put in the right parameters, call XYZ components of the skill, and then ultimately, um, take the right-- you know, generate the right answer.

But sometimes this can be a little longer. We've seen some times where the same skill, uh, it-- I don't know if you all have done this experiment, but you can put the same skill and build it with OpenAI, and you can also build it with Anthropic, and sometimes they have wildly different number of steps that the path

actually takes. And so the goal here is how do you be succinct, and how do you also make sure there's reliability in the number of steps that your agent takes to actually consistently complete a task?

So we call this convergence, um, but probably one of the hardest to actually evaluate. Is anyone evaluating convergence today, or at least counting the number of steps? Awesome. Okay.

You're awesome, dude. [laughs] Cool. Well, with that, I'm gonna go maybe two more minutes, and then I'll hop into one more demo here. So if any of you guys watched the movie Her, this is from Her. [laughs]

Um, uh, this is where, you, you know, the, the main character asks, like, "Who else are you talking to?" And, you know, the-- Samantha says something like, "Eight thousand other people are in a conversation with me right now."

And so the future of voice applications is that these are probably some of the most complex type of applications that have ever been deployed, ever been built. It's gonna require one more additional pieces to actually evaluate voice applications.

And the interesting part about these is that it's not just the text that needs to be evaluated or the transcript But it's also the audio chunk that needs to be evaluated.

Um, in a lot of these voice assistant APIs, you have the generated transcript that happens actually after the audio chunk is really sent. And so that's a whole nother dimension around is the user-- how-- what's the user's sentiment?

Is the speech-to-text transcription actually okay? Is the tone consistent throughout the entire conversation? And so you actually need to evaluate not just the audio piece and the flow of the conversation and everything else you're doing for all your other ag-- you know, other parts of your agent, but also make sure that the audio chunks are getting their

own evals defined on, um, you know, intent or speech quality or speech-to-text accuracy. Um, so this is important for, for voice. So with that, um, I'm gonna actually show you guys how we evaluate our own agent so that you can get a little bit of a example of, of what some agent in the wild actually does.

Um, this is our own agent, so let me actually show you what it looks like. Um, you can actually go on our product today, and there's a little copilot.

And our copilot does something similar to what other copilots do, where as people are spending time [clears throat] in our product, we actually help them do things like, "Hey, help me debug this," "Help me summarize this," "Help me look at this."

Um, "Can I search with natural language?" There's kind of this copilot integrated throughout our entire product. But we're an evals company, so what do we do? We actually dogfood our own tool, and we decide to...

What you're looking at here is actually the traces of our entire copilot actually in the wild. And every single step of this copilot, we actually run evaluations of. So in this case, we have an eval at the very top actually evaluating something around was the overall response that was generated, this was actually a search question.

Is the overall search question actually correct or incorrect? And then we also have one around once it actually called the search router, did it pick the right router? And then did it pass in the correct arguments into the router?

And then finally, ultimately, did it complete the task or the skill correctly in the execution of this, this entire skill? And so evals aren't just at one layer of your entire trace.

If you take anything away from this conversation, the goal here is really how do you make sure that you have evals throughout your application so that when something goes wrong, I can debug if it actually happened at the router level, if it happened at the skill level, or if it happened somewhere else along the flow.

Um, and I think that's it from me. Any questions? [audience applauding] [upbeat music]