AI Engineer Summit 2025
Agent Evals: Finally, With The Map
Read the talk
Agent Evals: Finally, With the Map
Evaluate what an agent represents, what it does, and whether it reaches its goal—then apply the same operational scrutiny to the system judging it.
From a talk by Ari Heljakka
Before you start: Familiarity with language-model applications, retrieval-augmented generation, and tool calling will help you apply the evaluation map.
What would establish that an agent does what you expect?
Before launching an agent into production, how do you establish that it does what you expect? A convincing response is only part of the evidence. You also need to examine the actions it takes, whether those actions advance its goal, and what they change in its environment. Ari Heljakka of Root Signals organizes that problem into a map with two sides: semantic evaluation and behavioral evaluation.
Semantic evaluation asks how the agent’s representations relate to reality. Behavioral evaluation asks how its actions and tools contribute to its goals and affect the environment. Each side can be examined locally or across a sequence:
| Scope | Semantic evaluation | Behavioral evaluation |
|---|---|---|
| Single step or turn | Coherence, consistency, response quality | Tool selection and usage |
| Multiple steps or turns | Conversation history and reasoning | Task progression and planning |
The distinction determines what evidence an evaluator needs. One response can expose an inconsistency within that response; consistency across a conversation requires the history. Likewise, examining one tool call is different from assessing whether a sequence of calls makes progress.
Both sides need grounding. On the semantic side, grounding representations in data—often through retrieval-augmented generation, or RAG—supports truthfulness. On the behavioral side, available tools connect the agent’s intentions to actions it can actually perform. Heljakka treats this symmetry as more than a diagramming convenience: representing the world is itself an activity, so representations can be understood as a special case of behavior. The map separates these concerns to make evaluation manageable while keeping their connection visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate response quality from retrieval grounding
Start with the quality of a single response. Is it consistent? Is its content safe? Does it align with stakeholder values and adhere to their policies? These properties are not inherently agentic: an ordinary language-model application needs them too. They belong on the map because they establish the semantic baseline against which the additional demands of agent behavior become clear.
Retrieval introduces another set of relationships to evaluate. Heljakka places RAG within the broader problem of attention management: what information is brought into the agent’s context? Evaluating that process means checking whether the retrieved context is correct and whether it comprehensively recalls the relevant information. Those questions concern the material supplied to the model before they concern the answer it produces.
Once there is an answer, three distinctions matter:
| Quality | Relationship being checked |
|---|---|
| Faithfulness | Answer against supplied reference data |
| Answer relevance | Answer against the user’s question |
| Factfulness | Answer against reality beyond the reference data |
Faithfulness is not a guarantee of external truth. An answer can accurately reflect its reference material while that material is incomplete or wrong; it can also remain faithful while failing to answer the question. Keeping these relationships separate makes a failed evaluation more informative than a single undifferentiated quality score.
The same principle extends through the RAG pipeline: different evaluators inspect different relationships among its parts. Retrieval quality and answer quality are connected, but they are not interchangeable. The map helps identify which relationship a test actually measures before its result is used to judge the whole system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate meaning across a sequence
Moving from one response to a conversation changes the unit of evaluation. Now the questions concern how the history develops: whether the conversation remains consistent and coherent, and whether it stays on topic when that is required. Topic adherence needs a policy, not just a similarity measurement. If the user is allowed to change the subject, following that change can be correct behavior; in a constrained interaction, the same shift may be a failure.
Reasoning adds another kind of semantic sequence. Motivated by the progress achieved through investment in reasoning, Heljakka includes the evaluation of reasoning traces where those traces are available. These are sequences of representational activity: how the agent develops its understanding before taking an external action. Conversation evaluation follows the exchange with the user; reasoning evaluation follows the available sequence through which the agent works toward an answer or decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check the action before judging the trajectory
Crossing to the behavioral side starts with individual actions. Does the agent follow instructions? Does it correctly interpret the characteristics of the available tools and choose an appropriate one? Are tool outputs of sufficient quality, and are tool-use errors handled correctly? These checks concern behavior even before there is a chain of actions to assess.
The structure of the call is a separate check. A tool request may select the right operation but supply malformed JSON or arguments that do not satisfy the tool’s contract. Conversely, a structurally valid request may select the wrong operation. For an illustrative order-lookup task, a proposed call might be represented as:
json
{
"tool": "get_order",
"arguments": {
"order_id": "ORD-1042"
}
}
This record makes the distinctions concrete: JSON validity concerns its syntax; argument validation concerns order_id; tool selection concerns whether get_order fits the instruction. The proposed call alone does not establish the quality of a returned order record or the handling of a tool failure. Those require evidence from the subsequent execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure whether actions converge on the goal
A sequence of individually acceptable calls still needs evaluation as a sequence. Are the actions converging toward the goal? Is the plan consistent, and is it good under the quality criteria that matter for this task? Local correctness cannot answer those questions on its own. The evaluator must relate the chain of behavior to the intended outcome.
This returns both halves of the map to external reality. Representations are ultimately grounded in truthfulness; behavior is grounded in goal achievement and utility. In Heljakka’s framing, the intermediate checks are proxy metrics for those ends. A tool-selection score or plan-quality score is useful because it helps explain or anticipate successful behavior, but it should not replace checking whether the agent actually accomplished something useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add efficiency, diagnosis, and live evaluation
The map describes what to evaluate, but operating an agent introduces additional dimensions. Cost and latency matter because the objective is to reach the goal as cheaply and quickly as possible. The number of steps also belongs in that optimization: the route to an outcome has operational consequences as well as behavioral ones.
Tracing and debugging provide the evidence needed to locate where the agent went wrong. Error management, in this part of the discussion, specifically concerns errors in tool usage. It is distinct from semantic errors made during inference or reasoning. Keeping that distinction visible helps separate a failed interaction with a tool from a mistaken interpretation of the world.
Evaluation timing cuts across the map:
- Offline testing: What can be tried and evaluated during development?
- Online testing: What can—and should—be evaluated during the agent’s actual activity?
These are separate dimensions, omitted from the simplified diagram to keep it readable. They still need deliberate treatment in the evaluation system; a development-time test and a check during live operation answer questions under different conditions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use conventional tests where the contract is concrete
Application-specific evaluation extends beyond the general map. Some specialized topics are mature; others remain active research areas. The useful additions depend on what the agent does. Tool-specific metrics can be comparatively straightforward when the tool is an API call with behavior that can be measured independently.
Those tool contracts can often be tested with traditional software-testing methods. Many other measurements use LLM-as-judge techniques. An evaluation system can therefore combine conventional tests of concrete tool behavior with model judgments of qualities that require interpretation. Introducing a judge, however, creates another system whose behavior and operating costs need attention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimize the judgment flow as well as the agent
A common starting point is a single-tier approach: optimize the operative LLM flow—the agent—and use evaluation scores to guide that work. The omission is that the judge introduces its own cost, latency, and uncertainty. Receiving a score does not remove the need to understand the system producing it.
Heljakka recommends accounting for judge optimization early. His double-tier approach treats both flows as optimization targets:
| Approach | Optimization scope |
|---|---|
| Single-tier | Operative agent flow |
| Double-tier | Operative agent flow and judgment flow |
The second tier makes evaluation itself an operational concern. The agent needs to perform its task well; the judgment flow needs to assess that performance with appropriate attention to its own uncertainty, expense, and delay.
Heljakka calls this work EvalOps. Evaluations can become sufficiently complex, expensive, and slow to require dedicated operational effort. In his framing, EvalOps is a special case of LLMOps that works with different kinds of entities and therefore needs different implementation choices, ways of thinking, and resource allocation. Building the judge into the workflow is only the start; getting the evaluations right is work that must itself be supported.
The objective is to make agents measurable and controllable so that their behavior follows the operator’s intentions. Evaluation serves that objective when it connects representations to reality, actions to intended goals, and the judgment process to a system that can itself be understood and improved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- EvalOps 101Article
Explains how to operate and improve a separate evaluation system, including judge costs, calibration, human requirements and production feedback loops.
Original research on using model judges to approximate human preferences, including their biases and limitations in conversational evaluation.
Updates since the talk
Current documentation covering retrieval metrics, custom evaluators, calibration datasets, nondeterminism and conversation evaluation.
Read the complete timestamped transcript
- 0:01
Hi, this is Ari Heljakka from Root Signals, uh, presenting agent evaluations finally with the map. So, the agent evaluation is rather art than science, and ultimately it is nonetheless required to actually ensure that your agents do what you expect them to do, especially when you're launching them in production.
- 0:26
So, without further ado, let's just, uh, dive right there into the map and start to get an understanding of what are all the things involved if you want to evaluate all the aspects of your agents.
- 0:42
Uh, so the agent evaluation can be actually neatly divided to the semantic and behavioral parts of what the agents do. The semantic part is all about how do the representations, uh, of the reality that the agent has actually relate to the reality.
- 1:02
Whereas the behavioral parts, uh, are all about how do the actions and the tools that the agent is using actually contributing to the achievement, uh, of the agent's goals in its environment, and ultimately what kind of effects it will have on its environment as well.
- 1:23
The semantic part, uh, is, uh, further divided into what could be called the single step or single turn, uh, items, which are, uh, like coherence, consistency, et cetera, so various semantic virtues, and what can be called the multi-turn
- 1:42
aspects of the same, so these relate to chat and reasoning. Whereas in the behavioral part, you also have this distinction whether you're looking at the task progression and planning or then individual selection and usage of tools.
- 1:57
So let's, uh, dive into each of these and paying also attention to the fact that truthfulness, uh, is ultimately achieved by grounding, uh, the representations in your data, often with RAG.
- 2:14
Whereas, uh, the goal achievement is what actually, uh, is achieved by grounding to the tools that the agent has available. And this is sort of a symmetry that is not accidental because in a sense, there is similarity and analogy between, uh,
- 2:39
representations and behaviors because representing the world is a kind of activity, so you can see that, uh, that the representations are, are in a sense a special case of tools, special case of, uh, uh, of behaviors.
- 2:58
And, uh, let's then dive into these individual parts of the map.
- 3:08
So first, looking at the semantic quality for the single-turn case. So here we have these universal virtues. There's a long list of these that you can look at, uh, and these are actually non-agentic in a sense, and we are only covering them here because of completeness, so that you can sort of understand these agentic parts through this
- 3:29
contrast. So these are things like, uh, uh, is the,
- 3:36
uh, the reply that the agent is giving to the user, is this consistent? Is the, is the content actually safe? And so on and so on. And then there's the inter-interesting and important part of whether whatever the agent is actually saying, if it's, uh, aligning, uh, with the values of, uh, of the organizations or people, uh, who
- 3:58
are the stakeholders, and also whether it adheres to the policies of the same.
- 4:04
RAG, or what could be more generally called the at-attention management,
- 4:10
uh, is then something that need to measure through specific evaluators, such as those looking at whether the retrieved context, uh, was correct, whether all of it was comprehensively recalled, uh, and so on and so on.
- 4:23
And ultimately relating the answers, uh, to the external reality through what is often called the faithfulness, which is then separate from, uh, from the both the answer-question relevance and also from the notion of factfulness in general, which, uh, relates to the reality beyond just the reference data that the RAG is using.
- 4:46
So now moving forward, uh, we should also pay attention to the fact that the RAG evaluations actually come, uh, in many forms, and they're, they also have certain symmetries, uh, that are not that difficult to understand when you look at them, uh, through, uh, a map like this.
- 5:06
So they are all just, uh, essentially looking at different relationships between the parts of the RAG pipeline. But this is not the, the main topic of the presentation, so you can look, look up more information through the, uh, background material links that we provide.
- 5:23
So then moving to the multi-turn case. The multi-turn in the semantic qua-quality sense means essentially chat, conversation histories, how do these develop, what sort of things we want to look for in these, consistency, etc., adherence and sticking to the topics when that is actually necessary.
- 5:41
Sometimes we want to allow changing topics if the, for example, the chatbot, uh, user is, if user wants to change the topic, or sometimes we don't, but we need to be able to, uh, be aware of this.
- 5:53
Then super important, uh, and almost groundbreaking progress has been achieved by
- 6:00
uh, by, uh, investments in reasoning. So you can also, uh, actually evaluate the reasoning traces, the reasoning, uh, chain of thought, uh, uh, if, if allowed to use that term here.
- 6:12
And that's sort of another way of looking at the kind of, uh, sequential or mul- well, multi-turn and sequential, uh, kinds of activities that the agent is just doing in the course of its reasoning and representations of the world
- 6:26
before even taking any kind of actions. And now we move to the action side. So here we have, uh, things to evaluate, such as whether the agent's actually following instructions.
- 6:37
Is it extracting the, the tool characteristics, uh, correctly? Is it selecting the right-- Is the right tool being selected? Is the output quality of the tools correct? Are the errors it uses handled correctly?
- 6:50
Are the structures, co- in the JSON tool usage, uh, format, are they, uh, correct? So these are all, all the things that relate to, uh, to the behaviors, even before you have a chain of, uh, behaviors.
- 7:04
And when you move to this chain of behaviors or multi-step or, uh, multi-turn, uh, cases, so then you actually start to look at, like, are, are the actions that the a- agent is taking converging towards actually achieving its goal?
- 7:20
And, uh, is the plan that the agent might have actually consistent, and is it actually, uh, high quality in whatever sense you want to measure that? Uh, so then both of these are actually grounded, uh, on external reality.
- 7:36
So, uh, like mentioned, the representations are grounded on truthfulness, and, uh, the activities and behaviors are ultimately grounded by goal achievement and utility.
- 7:50
Uh, and this is, uh, what sort of, uh, is the ultimate, uh, metric to everything else and, uh, that the agent is trying to do, and all these other things along the way are sort of just proxy metrics, so to speak.
- 8:08
Uh, moving to other practical considerations, we can only scratch the surface here, so I'm just sort of listing not to leave these out, but they don't neatly fall into this, uh, general map that I presented, but should also be taken into account on the background.
- 8:23
So most important of which is the cost and latency optimization. So generally, you want to-- the agent to progress towards its goal as quickly as possible and as cheaply as possible.
- 8:34
So cost and latency optimization, the number of steps optimization, and then moving to the, what you often are going to need, which is the tracing and debugging, so being able to actually see where did the agent go wrong.
- 8:49
Error management here refers to, to dealing with the, with the errors of the tool usage, so not the actual, like, semantic errors that, that the agent is doing in the course of its, uh, inference and thinking processes.
- 9:04
Uh, very important distinction to keep in mind is the offline versus online testing. So what are the things that, uh, that you can actually, uh, try out a- and evaluate during development, and what are the things that you can do and must do or should do, uh, during the actual online activities of the agent?
- 9:24
And these are actually going to become like, uh, two very distinct di- dimensions that could be also included on the map, but, uh, but that would make the map, map more complicated, so that's why I'm just mentioning them here now separately.
- 9:40
And then we have various special cases that I don't have time to cover. Some of these, uh, are, uh, more refined and more advanced than others, and, and some of these things are sort of, uh, more research-y, and there's a lot of papers on each of these topics.
- 9:55
And, uh, uh, depending on what you're doing, some tool-specific metrics might actually be useful to add to the mix, even, uh, uh, and they might even be rather simple to implement because the tools are often something as easy, a- as straightforward as, like, API calls and, uh, and such things that can actually be just measured separately, uh,
- 10:19
uh, using more traditional software testing methodologies. Uh, one important part to mention here is that a lot of these measurements are going to be implemented with LLM as charge techniques.
- 10:33
And now the caveat, uh, behind this is that, that quite often when people start implementing these evaluation methodologies, they are looking at this kind of a single-tier approach. Single tier here means that you're focusing on optimizing this operative LLM flow, meaning that you're optimizing your agent, which makes sense.
- 10:54
But if you only are concerned of, uh, with getting some scores on what the agent is doing, then you're sort of forgetting that what about all the cost, latencies, uncertainty related to the charge itself, the charge that is on the background.
- 11:09
So you should be actually, uh, taking into account as early as possible that you are going to need to also optimize the charge itself, and this is what, uh, what we could call double-tier.
- 11:22
So you need to optimize both the operative LLM flow that powers your agent and then this judgment flow that actually powers your evaluations. And, uh, this is a rather complex situation in general, and we are calling this EvalOps because, uh, this seems like a, like a separate kind of thing that involves, uh, evaluations that themselves are so
- 11:46
complicated, uh, so expensive and slow that they sort of earn their own, uh, category of, uh, of activities. And this is, uh, something that, uh, that, uh, we have written about, and, uh, and you can also find more about this on the source materials.
- 12:04
Uh, the general, general gist of it is that, that, uh, EvalOps is, uh, kind of a special case of LLMOps and, and actually operates on different kinds of entities that the, that the LLMOps, uh, uh, in general.
- 12:20
And, uh, and then it also requires different ways of thinking, different kind of software implementations, and also, uh, essentially a different kind of resourcing to, to get those evaluations right.
- 12:33
So thank you very much. This was only a brief glimpse on the, on the general landscape. I hope the map is helpful to you. And, uh, please take a look at the source materials, which will give you more depth to each of these, uh, these topics.
- 12:48
And, uh, happy to discuss any of these things, and let me, uh, know if I'm forgetting something crucial. And, uh, this, uh, presentation will of course be obsolete by the time, uh, it, it goes out.
- 13:01
So, uh, probably when, when you're watching this, there's already some major developments have happened. But that's, uh, that's how it is on this general life of, uh, of, uh, AI engineers.
- 13:13
Uh, so let's go out there and make our agents measurable, controllable, and let's make sure they are actually doing our bidding and, uh, not, not rebelling against our, uh, ultimate intentions.
- 13:27
Thank you very much.