How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI

I'll go ahead and get started. Uh, my name is Xander Song. I'm an engineer at Arize AI, where I work on Phoenix, our open source observability, experimentation, and evaluation platform. And today I'm going to introduce you to Harbor, which is an open source framework for evaluating agents on real tasks with real data in isolated sandboxed environments. The theme of today's talk is shipping based on evidence rather than vibes,

and my goal is to give you a conceptual overview of how Harbor works as a framework, and to share some of the ways in which my team is using Harbor to iterate more quickly and with more confidence on our agents, harnesses, skills, prompts, and tools. This talk is going to be a discussion rather than a interactive workshop, but my hope is that you walk away with a concrete sense of how Harbor works as a framework, and also a sense of how you might use it in your day-to-day work to make your

life and your agents' a little bit better. A disclaimer before I begin, I am not a maintainer of Harbor. Um, I do, however, consider myself to be an avid power user at this point.

So, so what is Harbor at a high level? Harbor is a framework that allows you to define realistic tasks in the form of natural language instructions to an agent. These are tasks that you would like your agent to be able to accomplish. Harbor feeds those tasks to your agent running in a fresh sandboxed environment with real data relevant to the given task. And then it enables you to define what are called verifiers. These verifiers

are specific to each task, and they just check that your agent produced the intended side effects inside of the environment. Verifiers produce rewards. Examples of rewards are binary scores, such as zero or one, or pass-fail. So this picture is, like, really, really simple, right? But I want to just take a moment to appreciate why it's powerful. Because if my task and my data are realistic,

and if I've done a good job of defining verifiers that accurately describe what constitutes success or failure on the given task, what that means is that I can hold my task and my verifier constant while I change the implementation of my agent. And by simply executing this pipeline from start to finish, I can quickly and efficiently gain a signal telling me whether or not the change I made to my agent produced an

improvement or a regression for this particular task. Um, and not only that, I can take it a step further. I can compile not just one task, but many tasks that cover the broad surface area of the capabilities of my agent. And this lets me close the loop here so that now I'm actually able to start iterating with a degree of confidence that I might not be able to, um, if I was shipping based on vibes alone.

So that's really the idea of Harbor in a nutshell.

Let's break it down a little bit further. I'm gonna talk about, like, the architectural details of how Harbor works. So on this slide here, I have the task that I was talking about on the previous slide. Each task specifies a set of instructions in the form of a markdown file. And once again, that is just a piece of natural language that you're going to feed to your agent.

The task also describes an environment in which the agent will run, and that typically takes the form of a Dockerfile. And that Dockerfile typically contains data that is relevant to your task at hand, and we'll give examples of this in just a minute. But the thing to realize here is that, um, Harbor is actually agnostic with respect to what kind of data you give it. So some examples could be, like, a SQLite file. A very common example for coding agents would be, like, a Git repository at a particular commit.

And Harbor also lets you define a set of verifiers inside of a test directory. And as I said, verifiers are just scripts. They're just scripts that run on your environment after your agent is done running that verify that the intended side effect took place.

And like I said, typically we don't have just one task, but we have many collected together in what is called a dataset. And for each task in the dataset, Harbor allows you to run a configurable number of trials. These trials are just independent executions of the task to account for the fact that LLMs are stochastic, agents are stochastic. They don't behave the same every time. So each trial is going to build a fresh environment in the form of a brand

new Docker container seeded with that data we were just talking about. Harbor is going to drop your agent of choice into this environment. It's going to feed it the instruction markdown file, and it's gonna let it do its thing. And of course, the agent under test is going to produce side effects in the environment usually. So if it's a coding agent, it's gonna make changes to source files. Um, it might make changes to a database if you put it in there. It could do any number of things, right? And after the agent finishes,

the verifiers are going to run on the environment. And again, a verifier is just a script that checks the environment. Those verifiers produce rewards, which get written to a reward.txt file.

And that, that's the most important thing if you take away nothing else. This is like the architecture of how Harbor works. There's another thing, which is kind of an additional optional thing you can do, which is you can configure your agents to emit so-called trajectories in the form of ATIF. ATIF stands for Agent Trajectory Interchange Format. ATIF is Harbor's specification for a JSON-based tracing format, which basically lets you peer into the internal details and the

internal steps that your agent took to produce the side effect on the system. So it's basically there for debugging purposes and for logs and, and recording and all of that.

This whole pipeline executing end to end, where we take all the tasks in the dataset, run a certain number of trials for those tasks, run our verifiers to produce rewards and collect trajectories, this whole thing all taken together is what Harbor calls a job.

And there's one more thing I wanna point out while I'm on this slide, which is you'll notice right here, um, I said Har- I said agent/oracle. Um, so Har-Harbor also has this notion of an oracle. You can drop not only an agent into the environment, but also an oracle. And an oracle is just a solution to your task. It's just a script that solves your task. And we'll talk about oracles more in a minute, but oracles turn out to be like this really useful thing because oracles basically let you work out all

the, the, the kinks in your pipeline before you ever throw an agent into the mix. It lets you make sure that your data is good, that your environment is configured properly, that your verifiers work as expected, and you get to do all that before you spend any money on tokens for an agent.

All right, so that is like the high level. Um, before I pause for questions, I will, I will just continue on for just a second to kind of illustrate it with a concrete example. So I'm gonna break down a concrete task. Let's suppose that our agent that we care about is a coding agent, um, or a coding agent with a particular set of skills, or a coding agent configured with like a custom MCP server that we're running, and let's suppose that the task is to fix a failing migration. So if that's the case,

the instruction markdown file would, you know, it would, it would be some set of instructions that you would pass to the coding agent as a user. So it might say, "Hey, I'm noticing that this migration is failing. Could you please try to fix it?" Right? And in the environment, the environment would contain perhaps an actual, um, SQL file. Maybe it would contain some DDL, um, and maybe it would contain some source code containing an up migration. And we've, you know, we've seeded the data so that the up migration fails. Um,

in addition, we'll have our tests, and again, the tests, they could be just scripts that run the up migration, that ensures that the up migration exit with status code zero, and then it, maybe it checks like some tables. Maybe it just checks the tables to make sure that the columns are there that we expect, the indexes are there that we expect, to make sure that the, um, migration has actually had the intended side effects on the system.

And you'll also notice that here optionally we have the solution, right? So the solution is just gonna be the literal solution, the literal s- script that you could run, um, that would fix the, the migration. And again, why would, why would you go through the trouble of doing this? Um, the reason again is it kind of lets you quickly and confidently build a little bit of a assurance that you've got your system set up, that your verifiers are correct. And you can-- It, it kind of lets you, um, vibe a little harder, I think, with the

system because you can gain a certain amount of confidence in your verifiers without necessarily, like, reviewing them line by line. At least that's my experience with it.

And so after the, the agent runs the task in the environment, um, it's gonna edit that broken migration file. It's gonna leave a couple modifications on the file system, and then the verifiers are gonna run. And typically, if you're running, um, Harbor in Docker, which is like the default local setting, those verifiers are gonna mount in the Docker container after the agent finishes, and that's to prevent the agent from

cheating. So the, the agent doesn't have access to the verifiers and the solutions, um, during the actual execution.

Those verifiers are gonna run the sort of checks that we talked about. Maybe it literally runs the up migration. Maybe it, um, you know, checks the tables, checks the indexes. And I think the thing to really take away from this slide is, um, the verifiers are agnostic of the implementation of your agent. They actually don't know anything about the agent at all. All they care about is what the agent leaves behind. So it is possible to, like, write some verifiers that are not like that, that peek into

the ATIF trajectory, that kind of look at the implementation or the execution of the agent and evaluate it that way, but that's not really the canonical kind of verifier. The verifiers, you know, canonically treat the agent as a little bit of a black box, and they're grading the agent solely based on its side effects and its output in the environment rather than the exact manner in which it produced those side effects and outcome. And there's kind of a natural parallel

here with integration tests because you can imagine that if you have some integration tests for an interface or an API, those integration tests, um, if they're really well-written, are hopefully agnostic of the implementation that is behind that interface or API. And in the same way, if we have really good verifiers, our verifiers are kind of describing what we expect the agent to do, the side effects we expect it to produce in the environment. And they're doing so in a way that is decoupled from the implementation.

And that's a really nice feature to have for a set of tests because it allows you to refactor under green, for example. It allows you to hold your tasks constant while you, like, completely change the implementation of your agent, and then it allows you to, like, fire this pipeline really quickly to figure out if everything still works. That's kind of what we're talking about when we're talking about iterating quickly, um, and shipping confidently. So before I go any farther,

let me just pause right here for questions or comments or, um... Yeah, let me just pause before I go any farther. Yeah, yeah. You just... Yeah. Again, it's just a Docker container. Yeah. So you just pop whatever you want in there, right? It's a primitive that you're, you're proposing. Yeah, yeah, yeah. Yeah, yeah, yeah. Yeah, yeah, yeah. So it, you know, it could be, it could be any framework, right? Uh, it could be any language. It could be any piece of code, any executable. Concretely, like, you know, I'll give you a little sneak, sneak peek or sneak preview, right? Like, we've had to, like, change tools, like, um, you know,

adjust, adjust prompts, right? Um, I'm going through a process right now where I'm trying to consolidate our skills, for example. So we kinda have, like, uh, several different agent experiences that we try to support. Um, there's a lot of overlap, and then there's, like, certain differences, and we're trying to, like, basically get our skills all kind of, like, reusable, right? Um, and agnostic of the particular, like, direction that the agent is coming from, whether it's MCP or a CLI or via, like, our, our own, like, server-side agent. We're trying to get it

agnostic. And, and so those are the kinds of things where it kinda helps to be able to specify this set of tests that is, like, completely agnostic of the underlying implementation, you know? And we, we kind of... We came to this, like, maybe, um, through the pain of doing it the other way, where we were like, kind of like, had a set of tests that were, um, that were coupled, and it was becoming difficult to maintain because, you know, I would change a tool and the tests would break, right? Even though the, noth- even though

the true functionality of the agent was still intact, right? Things like that. Um, we'll get to more of these, right? We'll get to more of these at the end. Um, any other questions before I go on?

Okay. Okay. So, uh, this, this slide is gonna talk about scale. Um, so as we, as we already talked about, right, each task and each trial, uh, is independent. Runs in its own environment, its own Docker container. Um, and that makes this whole pipeline completely horizontally scalable. And that means that it's very compatible with a sandbox provider such as Daytona or Modal or E2B or any of those. Um, and it means you can kinda rip through these tests,

like, really fast if you want to, which is kind of fun. So in this diagram, arrayed vertically, we have a set of four tasks in a dataset, including the fixed migration task that we were just imagining. Um, we've got a couple of different coding agents that we might wanna test out, Claude Code with Opus-5, Codex with GPT-5, and then we've got, um, a couple of different trials. Let's suppose that we're doing three trials for K equals one, two, three. Again, the point of that is just to kind of handle the, handle the fact that these things are

stochastic and they, they're not perfect. They don't behave the same every time. Um,

so because all of these are independent, um, you can run them all in parallel on these sandbox providers. So we use Daytona, um, and it runs quite fast with, like, 200 concurrent workers. So even if you have a large test suite, you can rip through it quickly. Um, and then from these individual trials, you are able to compute aggregate metrics for the kinds of things that you care about. So for example, you could compute a pass@K metric that just tells you, in this case, did I

pass after three attempts at the, at the task? And so I can see that, oh, for, like, this particular task, the fixed migration task, both of my agents, like, pass@K, right? Pass at three. Uh, for this one, only one of my two agents pass at three. You can actually also compute those kind of aggregate metrics over the entire dataset, right? And so this kind of like gives you this very quick picture of how your change is performing, 'cause again, if you have, like, a, a good dataset that covers the set of functionality that you care about, you can get

these aggregate metrics pretty, pretty rapidly. Yeah, go ahead. Each... Yeah, that's a great question. So this is a really important point, right, is each of these boxes is a totally separate Docker container, identical in the beginning, but then you, you put an agent, a separate agent into each one, and you let it go do its thing, and they do different things, right? Um, so each one is totally independent. They're completely isolated, right? They're independent trials of the same task with the same agent. Um, you're basically ru-

Yeah. You're basically running Claude Code three times and saying, "Hey, try to fix this migration." And then you're saying, "Oh, after three tries, did Claude Code fix the migration?" Okay, so now, now I've, now we've talked about Harbor at a, at a high level. Um, but I wanna switch gears and I wanna talk about how Harbor plugs into Phoenix. So again, I'm not a maintainer of Harbor. Um, I'm just a fan. Uh, but I am a maintainer of, of Phoenix, uh, which is our, which is Arize's open source AI observability tool. And the point of this slide is just

to basically convey that we have analogous concepts for the primitives in Harbor. So we're able to map your datasets to our notion of a dataset. We're able to track your experiments. We're able to, um, record your trials. Uh, we're able to convert your rewards into our notion of reward, which we call an evaluation. Um, and I guess the other thing I, I wanna take away from this slide is also we, we do something

where we convert those trajectories that I talked about into so-called OpenInference format. Um...

So OpenInference is a open standard of semantic conventions, so-called semantic conventions, which is built on top of OpenTelemetry, which is an open source framework for telemetry data such as logs, metrics, and traces. And

these conventions basically define data structures that, uh, your AI application can emit, which then downstream consumers such as Phoenix can ingest, and we can use those to build certain kinds of UI experiences and workflows that wouldn't otherwise be possible. So full disclosure, I am a maintainer of OpenInference. Um, we recently donated this, the spec to Cloud Native, right? Um, and to the OTel working group. So it has some traction outside of Horizon, so that's

maybe a reason you might want to, um, use OpenInference. Um, so this diagram on the left, you can see the ATIF trajectory for that fixed migration task. You can see the individual steps that the agent might take, um, including an LLM call and including a tool call. And over on the right, uh, this is kind of what it winds up looking like in OpenInference.

And in the product,

like I said, OpenInference allows us to build out the kinds of opinionated UIs and workflows that you see here. So we can do things like we can render your trajectories with some opinionated UI, some specialized UI for LLMs and tools. We can version your tasks and your datasets so that at any given time, you know exactly what the task looked like for a particular job in Harbor. We track your experiments, and we help you

visualize metrics and rewards as you iterate. So in this case, the reward, you could imagine it being pass/fail, um, zero or one, and you can run a sequence of experiments, and you can kind of see over time, is my reward moving in the right direction? We also have skills, CLIs, and an MCP server so that Claude and Codex and Cursor can easily get access to this data. And at this point, I do wanna note that Harbor does have its own open source UI, um, that you can run

locally on your machine. It also has a hosted web service. Um, so you, you definitely don't have to use Phoenix to use Harbor. Um, but if you're interested in some of the features I just mentioned, that would be a reason to give Phoenix a try.

Okay, so hopefully at this point I've given you kind of a bird's-eye view of how everything works. Now I wanna talk about some of the ways in which our team has been using Harbor, and hopefully this answers, um, answers your point over here just around getting some concrete examples. Um, so in order to do that, I just wanna give kind of this high level of like what our architecture works with. So again, this is the Phoenix architecture. This is particular to our team, right? This is the... This is what our team is kind of like dealing with on a day-to-day

basis. Um, on the left here, you have an end user's AI application or agent. And again, this application is going to have been instrumented with OpenTelemetry and OpenInference, and it's going to be exporting and emitting traces to the Phoenix server over OTLP, which is the OpenTelemetry Protocol. So it's basically sending data to Phoenix, right? Saying, "Hey, I'm doing this, and this is like what my-- these are what the sequence of steps that I did to accomplish this task."

This could be like a production application. It could be an application that you're testing out. Those traces are ingested by Phoenix, and they are written to one of our SQL backends, either SQLite or Postgres. And once the data is in the system, we have a set of core services for things like traces, spans, sessions, datasets, experiments, and evaluations. And these core services are exposed via various kinds of APIs, REST or GraphQL. But our goal is

not only to, like, make those programmatic interfaces, but it's also to make it easy for our end users and for their coding agents and other kinds of agents to get access to the data and use the data. And so we support several different kinds of agent experiences, um, the first of which are coding agents. So for coding agents, we have a CLI that the coding agent can use, which will invoke REST, or it will invoke GraphQL. We also have an MCP server that you can

configure your coding agent to use, um, which is going to do the same things. And we have a third agent interface that we maintain for the team, which is a server-side agent that we call Pixie, which also has access to the MCP server, and therefore has access to REST and GraphQL. Um, it also has like a Bash emulator, so it's, it's kind of like a coding agent, but running server side. Um, so these are the three surfaces that we, that we want to

support and that we have to test and ensure are working. The coding agent with the CLI, the coding agent with an MCP server, and a server-side agent, which is our own harness.

And when it comes time to actually, like, test all of this out, architecturally, we have a pretty big advantage. We have this really nice feature, which is that the server runs on a single node. It's kinda like a Grafana-like architecture. Um, and that means it's very easy to Dockerize. So you can just pop the whole thing in a Docker container. What we do is we just take a SQLite file, which contains some realistic data that we've tested on, um, we replace the browser with a test client, um, and we just put it in

Docker.

And once it's in Docker,

we can start using it in Harbor, so we can feed the system the kinds of tasks that our particular users care about. So our particular users might ask a coding agent, or they might ask our Pixie server agent Find some hallucinated LLM calls from the last week of my traces and add them to a dataset. That's a realistic query that one of our users might ask. And we can go ahead and let any of these three kinds of

agents go wild inside of this environment, and at the end, we can run our verifiers. And the kinds of verifiers that we can run for us in this particular setup are really quite simple because we have a server running inside of here with a SQLite file. So we could either hit a REST endpoint to verify that the intended side effect has changed, or we could literally, like, query the SQLite file if we wanted to because it's just in the container,

um, and just verify, for example, that the particular tuple for this datasets table has been written in the way that we expect. Yeah. Yeah. No, thanks for stopping me because I think there's a lot going on on this slide, so maybe I should pause and, and kind of, like, make sure it's clear before I go on, right? Um, so I think there were a couple of questions there, and I'll, I'll try and address, address them. Um,

so in this slide, there are three different agents. In any, in any given trial, I would only be testing one of them, if that makes sense. So you could imagine maybe I'm only testing Claude Code with the CLI, maybe I'm only testing Claude Code with MCP, or maybe I'm only testing, you know, our Pixi server, server agent. So just may- let's imagine, like, it's just one at a time. In terms of how does Harbor feed it the prompt, Harbor, Harbor... And this is maybe a nuance that I haven't really talked about, right? Harbor itself, um,

does not-- Harbor is a program. It's a command line tool. Um, and it doesn't actually run in the environment per se, in the Docker environment per se. It runs on a host machine, which could be your laptop if you're running it locally, or it runs in, like, a GitHub Action runner, you know, if you're running it in CI. Um, and you can run the environment in a Docker container locally. That's, like, the default behavior. But if you're trying to do something more scalable, you would be running it on a sandbox provider in all

like-likelihood. Um, so you'd be running it on a Daytona. And so Harbor is this orchestrator, um, which is running on your laptop or running in GitHub Actions, which is, um, kicking off the run. It's basically saying, you know, "Hey, start, start the agent," right? Um, and the way it does that is... And I can show you what this looks like on the file system, but it basically, you know, um, expects a particular script. Harbor expects a particular

script to be written for the task, right? It's all driven via scripts. So you-- Like, when you're, when you're setting up Harbor, you have to kind of make sure that all these scripts are in the right place in the Docker image. Um, so there's gonna be a script to start the task. There's gonna be a script to run the verifiers. There's gonna sc- be a script to run the oracles. It's all driven by scripts. So the scripts are in the Docker container, um, but Doc-- but Harbor itself is a program that lives on a, on a host machine and just kind of orchestrates things and kicks things off. That's right. And the interfa-- Yeah. So

maybe I'll just repeat what you, what you just said because I think it's well put, is like, Harbor is a CLI, um, and it has certain conventions for if you, like, make your Docker container or your Docker image look this way, it will be able to run it. Um, and those conventions are almost all, like, is there a script in the right place? So that's kinda like the contract, right? If there was a, a contract that you had to honor with Harbor so that your, your image would run, it would be like, "Did I put my scripts in the right place?" Right? That's kind of the

contract. Got it. Yeah. Does that, does that clarify? Yes. Okay. Yeah. So again, on this one, on this slide, right, basically, I think the takeaway is, um, the takeaway on this slide is, like, you have a great deal of freedom, um, and flexibility when it comes to customizing your Harbor environment, right? Um, the, the simplest, like, the simplest example that people always get started with is the example of a coding agent operating on a Git repo, but this example is different, right? This example is, like, I

literally put some data from a SQLite file in there. I threw my whole application in there with it, along with its APIs, along with its own server-side agent. It's all-- It's a running server in the environment, in the Docker container. Um, something, you know... There's a CLI in the environment. So what's not even included here is, like, there's actually a set of skills literally in the environment on the file system that the agent has access to. Um, so all that is to say is, like, you have a lot of flexibility in setting up your

environment. Um, you know, it has to fit in a Docker image, but if you can fit it in a Docker image, um, you know, you can, you can make Harbor run it, basically.

All right. And now... Any other, any other questions before I move on? Maybe... 'Cause this is a complex slide, I think. This is, and y- and, you know, this is like, again, this slide is very much, like, particular to h- to our project, right? This is, like, what our project looks like. Your project may not look like this, right, if you, if you're not running, like, a single node server or something. But yeah, I mean, that's such a good question, and I haven't had to deal with that problem, so it would require a lot of thought, I think, right? But basically, the question is like, my system is not a single node,

it's not a single, you know... I can't just, like, throw in a SQL s- single SQLite file. I have, like, a, you know, some distributed system with, like, S3 and all this other stuff that I'm trying to maintain. How would I test Harbor out with that? The answer is, like, that's probably harder. Yeah. That's probably harder. So in that case, in that case, my initial thought is, like, you know, you have to perhaps carve out your agent a little more, right? So here we've had this advantage where, like, it's simple for us to throw the whole damn thing in there, in the Docker

container, and so we did. If that wasn't simple to do, if you can modularize your agent somehow into some kind of, like, interface, hide i- hide the implementation behind an interface Um, and if you can like, if there's some, like, stable API boundary or interface that is relatively stable, right? Where you know, um, I'm feeding, like, this agent data in this format, right? Whatever that format may be. Like, eventually, this agent has access to data in this format, right?

Um,

that would be one approach, right? Um, and I realize that that is maybe not a satisfactory answer depending on what your exact setup looks like, because maybe your agent, like, literally queries S3, and then, like, what are you to do, right? Are you gonna mock out all those S3 calls? And then once you start mocking out S3 calls, you're kinda in a world of pain, because now you're in the world of mocking stuff again, right? And, and that's really, like, what we're trying to get away from with Harbor. We're trying to get away from that world. Um, so I don't know that there's like... I actually don't know, 'cause I haven't had to explore this problem

yet. Um, I do know that, you know, I've been kind of, like, eliding this particular feature of Harbor, but Harbor does have the ability to, um, like, run multiple Docker images, right? Spin up multiple things. So, like, you could, you could wonder... I would wonder about that, is like, oh, is there a way where I could somehow encapsulate my application or something? Yeah, go ahead. The question was, like, what scale do you use, right? If you have some massive SQLite file or some, you know, really large set of traces, do you, like, actually

evaluate on the whole thing? We don't currently do that, and I think it would probably be okay for us to do that. Um, right now, we, you know, right now, the kind of format is like, oh, like, we have some data here. I'm gonna have Claude kinda help me clean it up a little bit for this particular set of tasks. And again, the kind of, the nice thing about Harbor is, you know, you, you can, um, use the exact same data for many different tasks, but you don't have to. Um, and so if there's a particular question that you care about, and it only

requires a small subset of, of the data to answer, you don't have to, like, upload all of the data to answer that particular question. Um, but yeah, I, I mean, it, it just depends, too, like, on what you're trying to test, I think, right? If part of what you were trying to test was the performance of your agent when the system is full and the database is, like, quite full, then I think it would totally make sense to, um, put in a lot of data. We, we haven't done that yet. Well, I tried to... Yeah, that's such a good question. I think the question is, like, what is, what, what do you get by using

Phoenix with Harbor? Why not just use Harbor maybe? Um, and the answer is, like, you totally can use just Harbor, right? I'll, I'll give you a little demo, um, in just a second, where hopefully some of, some of what we offer is, is on display, right? 'Cause I, I do think that there are some genuinely unique things around, like, dataset versioning, around the experiments, around the visualizations, around the, the agent experiences that you get, um, the ability to, like, actually inspect and drill down quickly, right? So we've been, we've been solving those kinds of

problems, um, for several years, and, and our UI is, like, really dialed in for that sort of thing. And I'll give you a demo in just a sec. Yeah. And then you can ask me again, see if I answered the question. All right. Uh,

all right. I'm gonna go to the next slide, right? So this particular example that we're gonna think about is, um, this problem that we encountered recently. Uh, the problem was, you know, in this picture that I've shown you, I talked about REST, and I talked about GraphQL. So our MCP server, our CLIs, they had access to these APIs. But we found that people were asking questions that are not easily answerable via REST or GraphQL. So they were asking aggregate kinda questions, right? They were asking, you know, "What was the busiest day

that I had in the last few months?" Or something, right? Um, "How many traces did I get on that day?" Or they're asking questions like, "How much am I spending on, uh, each of these models?" Right? Like, which model is actually costing me the most out of all my LLM calls, right? Where am I spending the most? Um, and so what the agents would try to do is they would, uh, you know, when possible, they would, uh, look for the, the REST route or the, uh, the GraphQL resolver. Um, but these were not aggregate resolvers. So they

would try to, like, basically paginate through stuff and, and do the aggregation themselves. And as you can imagine, that results in a problem, because if there's enough data, you wind up, uh, you wind up doing a lot of, a lot of paginations, right? And a lot of tool calls. Um, and so part of what we did was we actually added this, this line right here. So this line, uh, is meant to represent a tool that we added to our MCP server, um, which basically

allows the agent to write a read-only SQL query and execute it against the database. Um,

and a lot went into making that tool. I won't really go into the implementation details. But the thing that we noticed even after we made that tool was that, oh, sometimes, sometimes the agent isn't calling the right tool, because now the agent has this other problem, right? Now the agent is here, and it's like, "Well, I got REST, I got GraphQL, and I got SQL. Which one, which one should I be using?" The agent doesn't really know. And so we realized, oh yeah, well, naturally, we have to, like, somehow update a prompt. We gotta tell it what to do, right? Um, but even after you tell it what to do, how do you

verify that it's actually doing it, right? Um, you know, it's one thing to change the prompt, but again, we're trying to, like, ship based on some evidence, right? And not just on vibes. Um, so this is, this is the change that we, we wound up testing out. It's like a small little tweak to a, to a prompt, right? So before, it, it only mentioned, uh, GraphQL. But after it says, "Hey, if you're dealing with, like a... Some kinda aggregate, like a count or a group, um, why don't you use our SQL tool rather than reaching for GraphQL first," right?

Um, and of course, the most important, the most important thing that we need in order to answer this question is, first of all, some, like, realistic data. So we throw in a, you know, semi-realistic, um, SQLite file that we've been kind of just, like, collecting as we develop ourselves, as we work ourselves, and, like, dug through the product ourselves. Um, but then here's the really... the part that I found quite cool or novel about Harbor, which is like, um, I literally just asked Cl- Claude, like, "Can you look at like

the domains of my application, you know, traces, spans, um, data sets, experiments? Can you look at all these domains? Can you brainstorm a list of user queries, some of which would be aggregate, some of which would not be aggregate, right? And then can you write an Oracle solution to those queries?" Right? Um, the Oracle solution literally just being a REST call or perhaps a database query, and the coding agent will just do that. So we generated just this data

set of synthetic questions. Um, I spot-checked the questions, right, just to make sure they sort of make sense. Um, and I have a certain degree of confidence in the actual verifiers, which again, are just scripts that are, that are running in the environment. But because those verifiers, um, are passing against an Oracle, again, those verifiers are just like, you know, kind of passing against this Oracle, I have a certain amount of confidence in them, right? Some of them were LLM-as-a-judge, some of them were SQL queries or, or REST calls

themselves. But, you know, just the ability to kind of work out those kinks before the agent ever gets to the system, it gives you a certain level of confidence. Um, and then, you know, I was able to test out this particular change, and I was able to see like, oh, you know, as you might expect, making a change like this, it causes the system to use SQL more frequently and more aggressively, which is what I wanted it to do, right? I want it to overuse SQL rather than underuse it, 'cause I don't want it to like start trying to paginate through trying to do like aggregate queries. That would be bad.

And then the other interesting thing is just by changing this one prompt, I can also see the number of tool calls in each, in each, uh, trajectory, the average number of tool calls dropping. Which again, is like also, you know, the reason that we did this in the first place, is I wanted to reduce the number of tool calls. And this dataset, it wasn't even like a big data. It wasn't a big dataset, so it didn't have to do a lot of pagination. I think it-- th-these numbers, like these modest drops, they kind of understate the issue because the data was not large. And so let me, let me show

you now what this looks like inside of Phoenix. So the first thing that you can see here, um, are the examples in our dataset, the tasks in our dataset. And again, these are versioned, so we actually have a whole version history here every time there's a change. Um, and we can inspect what some of these look like. So again, some of these are aggregate queries, some of them are not. So here's an example of an aggregate query that I might care about. What are the three most expensive models by total cost?

How much does each cost? So this question is more easily answered with a SQL than, than by paginating through like a REST API. And down here, you can see the answer. And again, this answer, um, it comes from an Oracle. Um, and so that's part of why I have a certain level of faith in it, is because it's, it's coming from a SQL query run against a concrete set of data. It's not something that I feel, um, a strong need to check actually for this particular example, because

the answer is deterministic, and I'm, I'm sort of allow-- I'm tr-- I'm placing a certain amount of trust in Claude, I guess, to like, write a good SQL query. Um, so we got all our examples here.

And then we got our, um, we got our experiments and, uh, our notion of an experiment, that's, that's what I've been calling a job in Harbor. So just imagine this to be all the jobs that we've run. I've run a lot of jobs in preparation for this talk. Um, these charts show you the kinds of metrics that you might care about, whether those are cost or run latency. Uh, the reward, as you can see for this particular task, is quite high. Um, and that's because I haven't really done anything that's gonna like, hopefully break the system. It's not like something was truly, genuinely broken. The system was getting the right

answer in the beginning, so the reward was always high. Um, it's just that it was just taking too many tool calls, right? So this was more of an optimization rather than a correctness fix. You can see tool call count. You can see, um, whether or not we use SQL. How, how do you calculate the reward? So the reward, if we go back in the slide, the reward is just coming from the verifier, and the verifier is something that you, the user, have in your control, right?

So when you're defining your task, for each task, you get to write a script that says what constitutes success or failure, and that script is going to emit a zero or a one, a pass or fail. You get to name the reward, right? So the reward could have a different name other than pass, fail. Um, that's all a reward is, though. You as the developer, you as the maintainer, you think about, for this particular task, what, what does it mean to succeed and what does it mean to fail?

How could I figure out just by looking at the side effects in the environment, looking at the file system or database file, for example, how could I figure out whether the agent succeeded? And that's what the reward is. And so that's what we're tracking over time there, are the aggregate rewards for the entire job. That's what we're tracking in this view. Yeah, go ahead. I mean, this is such a good question, right? I kinda get... it kinda like gets to one of these big, big quick, big picture questions about like, what do you use Harbor for, actually? The question was around like, well, hey, you said that, you know, typically

your verifiers are agnostic with respect to the, the trajectory or the execution path of the agent. But hey, like look, some of your verifiers actually like kinda... it looks like they're peeking in to see whether the agent used SQL. Isn't that cheating? Um, the answer is yes. So it's not-- I think the times when you would do what I said in the purest form of like your verifiers are truly agnostic, and they really don't look at the trajectory, and they really just look at the file system, you know, or the, or the side effects. The situation for that is really like, um, when you care about correctness.

Yeah. Right? So may-- you know, one of the other things that I'm not showing here, which is a different example that we've done, is like we've worked on some regression tests, and those tests are really like- Does it get to the right answer in the end? And for those tests, I think our verifiers are quite pure. They don't really look at the trajectory. They just look at like, "Oh, did it get the right answer?" Right? For this one, for this particular problem, I was concerned with something maybe subtly different. Like, I wasn't really just trying to get the right answer. I was trying to optimize a metric, right? I was trying to optimize tool

calling in particular, tool calls. Harbor's one of these things where you can use it for many things. You could use it for, uh, you know, I, I talked about integration testing. You can use it for like kind of end-to-end testing, um, for assuring correctness, um, for running in CI, right? Or you could also use it because you're like, "Oh, right now I'm trying to solve a really particular problem," which is like, "Oh, I kinda noticed that it sucks on these aggregate queries. Like, how can I make it better? How could I, in a quick and dirty way, kind of convince myself that it's actually doing what I hope it's doing, right?" Um, so like at least

for, for us, like what I've found so far is it just depends on what you're trying to do. Um, sometimes, sometimes it, it is worth peeking inside the trajectory, I think, you know? Yeah, even though, even though it makes the test less robust to change perhaps. That, that's a really good question. My... You know, my team is so small, I don't know that we can, I can really speak from personal experience. Um, that being said, like one of the reasons that you, you might use a tool like Phoenix is because it allows, you know, it has like the ability to share data, right?

To curate data from different sources, to add it, you know, to, to like add traces to a dataset. Um, it has the ability to like store data as a, you know, in an external file and like upsert it to the database and then run it. It has those kinds of abilities, so like that might help you. In terms of the, the, uh, logistics of like, you know, actually like dealing with this in a larger team, I, I don't, I don't know, 'cause really it is just me and a handful of people working on this right now. All right, I'm gonna show you one last little demo, and then I'll just take questions. I think we got only a handful of minutes here left.

So, you know, back to, back to your question around like what are some of the things that you might get out of Phoenix. I think this is like kind of the, the, one of the bigger ones, right? Is like the ability to compare individual jobs at the task level, right? So right here I have two, two particular jobs selected. I have Opus 5 before and after making that single prompt change. This column is Opus 5 before. This column is Opus 5 after. These, these metrics at the top, they're telling me in aggregate how did these like metrics that I care about, how did

these rewards change after I made this tweak to the prompt, right? And I can see that like GraphQL tool calls went down, so it's calling GraphQL in aggre- in, uh, it's calling GraphQL less than it was. Um, the reward is staying the same, which means everything is still passing, right? SQL calls are going up. Tool counts are going down. The number of turns the agent is taking is going down because it's encapsulating a lot of that, a lot of those turns that would've been multi-turn steps into a single turn because

it's using SQL. Uh, it's using more SQL. It's using less GraphQL, right? So you get those kinda like top-level metrics, which are the ones that in my mind verify, "Okay, this is a decent change." Um, and then I'm able to see in this table, like four particular tasks, right? Here's the, here's that one we looked at, like what are the three most expensive models by, by total cost. I'm able to see what I expected the answer to be, right? I'm able to see the actual answer in each job. I'm able to see the metrics and how they changed. So for

this one, um, yeah, for this one the number of tools went down. The number of turns went down. Interestingly enough, the number of SQL calls actually went down too, which is kind of unexpected. And I can do things like I can, I can filter by these, right? So I can... Here's a fun little feature that Mikio built recently, um, which is I can say, "Show me the runs where the second experiment had, um,

where the second experiment had lower

tool call count," and I'll use the exact, the exact metric to help it out, "lower tool count than the first experiment." We'll see if this works. So what this is doing is it's actually running a, a local model in the browser to convert that piece of natural language into a query DSL. I think I messed that up, though.

We gotta, we gotta use a more powerful one. The prompt API is a little bit- I'll just, I'll just copy in what I know to be the correct, uh,

correct expression here.

So I'm able to filter down. So now I'm just looking at the, um, now I'm looking at the rows where it improved basically. Like I'm looking at the ones where it didn't use SQL before where now it's using SQL. Um, and you can see here this one used SQL. It went up by one. And if we actually like look at the, uh,

look at the trace, we can actually see that, right? So those ATIF trajectories that g- get converted into OpenInference traces, here you can see it. You can see it was using Bash before. Um, and when it's using Bash, under the hood it is actually using GraphQL. It's calling, has a Bash emulator which is calling GraphQL underneath the hood, right? So I'm able to see that right here. I'm able to see in the second job the same trace or the analogous trace, and I'm able to see, oh, instead of, instead of calling Bash

and using GraphQL here, it actually issued a SQL query directly to the database. So I'm able to like visually inspect and say, "Oh, okay. In this particular example, it did the right thing." Um, and I can even, I can even ask, you know, the agent itself. So this is kinda like what we do. Um, you know, I can literally ask the agent itself like, "Hey, look at a particular, um, set of experiments," you know, "Look at this particular task," and it will tell me exactly what happened. And so this is kinda like how we get some of our data in the first place,

is we're kind of like using, using the agent, um, recording its traces, uh, getting its data, and using the agent to like iterate on itself. Yeah, I'd say with respect to the regression stuff, we're still really early days. I mean, we're trying to get like happy paths going, I would say. Um, and I think like the way that, the way that we're structuring it is, is around skills because I think in my mind, when you're doing those regression tests, they are sort of analogous to like Playwright in a certain way, where it's like an end-to-end thing, and it just includes

like an agent in the loop. It includes like LLM calls in the loop. Um, and so I think skills are really good, like, thing to write those kinds of tests for because at least in our case, like the skills are describing workflows. They're saying, "Oh, if this happens, do this. If that happens, do this," right? It's describing a workflow that you wanna have happen. And, and so right now, we're just kinda in the process of trying to nail down happy paths on our most important skills, um, and ensure that the happy path passes across

like the, the broad array of like models and providers that we're trying to support. Right, right, right. Yeah, so this, this particular comment is around like the ability of a... You could make the gr- the greatest skill in the world, but like, if somebody else makes another skill that kind of like where the front matter of the skill kind of treads on your feet a little bit, it's gonna-- it may not fire anymore, right? And then it's no good if it never fires. Um, and for that sort of thing,

it's an interesting question, right? It's like there's multiple ways of approaching it, right? Like one thing that you could do would you, you could just say, "Well, you know, I don't care which one fired as long as I have a good test, and I know that like, oh, well, like if this skill didn't fire, it's probably not gonna pass this test," right? Um, that would be one approach. Another thing you can do, and maybe this is the appropriate thing, right, is you can always, as we said, reach into the trajectory with your verifiers. You can always look in. You can always peek. You know, you

can cheat a little bit, and you can say, "Oh, well, like given this question, did it call this particular skill?" And we have had tests like that. Um, I think the challenge is maintainability, right? It's always around maintainability. Um, if you change the skill name or something, the test is gonna flake. Um, but we've had tests where the thing we were trying to test was skill firing, right? And those tests are not, they're not the hardest to write, right? I mean, don't even... 'Cause if, if you're only testing one turn,

and all you wanna know is like, did the right skill fire, um, you don't even really need great data for that. You just ask it a question and, and see did the skill fire. Um, and like you can totally, you know, maintain a suite of tests that, that do that, right? And, and, you know, then it becomes less of a end-to-end test. It becomes more of like a,

almost like a, i- it's like almost like, you know, on the spectrum between, you know, fully decoupled from the implementation and coupled to the implementation. It's a little bit more coupled, um, and yet it could still be useful to you, right? Um, so there's no reason to not write a test like that. Um, there's no reason to not write a test like that, I think. So it really just depends on like, you know, getting, getting, getting a broad level of coverage, um, on the things that you care about. We are collecting traces

in production, so when you deploy Phoenix in production, this is the picture. The traces are coming into the system from an instrumented application or agent. We're collecting those traces, and we're writing them to the database. When we're running things in Harbor, you can see that this, this thing disappears, right? The OTLP ingest, the AI agent, which is the user's agent, those disappear, and we just seed the database with data. So when you're running an... When we evaluate in Harbor,

um, we don't, we don't pretend or emulate an end user's application. Um, we assume that the end user's application exists and has resulted in the data in the, in the SQLite file. Yeah, so the question is around, um,

Harbor is canonically used for evaluating coding agents.

Right.

That's the canonical use case. And we're doing something

similar on the one hand, but the unit that we're trying to evaluation, to, to evaluate here is not the coding agent in isolation because we don't work on Claude Code, right? We don't, we don't tweak Claude Code. Uh, we don't tweak Codex, right? The thing that we're trying to optimize for is this hybrid system, which is Claude Code plus our particular bespoke MCP server, or Claude Code plus our particular bespoke set of skills that we maintain, or Claude Code plus

our, um, Claude Code plus, uh, our CLI, or Claude Code, um,

interacting with our APIs. And that is a different problem, right? And the fact that that is the thing that we're trying to optimize, this hybrid entity, which includes Claude Code, it includes the coding agent, but it's more than just the coding agent. It's all of the, um, supporting infrastructure around the coding agent which we maintain, which helps Claude Code be better at interacting with our product. Um, that's the reason that we've shoved all of this stuff into an environment together is

because all of that stuff has to change, um, and be evaluated together, um, as a unit. I think it took a second to kind of conceptually land on and understand Harbor well enough to land on this picture. Um, and then it took a second. It, it didn't take super long, and again, like I think part of that is, part of that is due to our advantage, which I mentioned, which is that Phoenix runs on a single node. Um, and that makes it easy to Dockerize, right? And so as, as some of these other

folks suggested, it's like, you know, if you're working on a distributed system, you know, it's, it's tougher. Your, your life is gonna be a little tougher, right? And so we had a certain advantage that made it easier for us to set this picture up. Um, but I would say like the coding agents are, um... Harbor's tough in the sense that like there's no-- because the contract that Harbor tries to honor is all based on like scripts in certain directories. You know, I need to put my, my task s- my task

initiation script here. I need to put my verifier script here. Um, there's no like linters that the coding agents can rely on to make sure that that works. Um, the best that you can do, uh, is you can download Harbor's skills, um, and those will help you set up your project correctly. Yeah, so the question is, uh, I have a SQLite file here. Where, where do I put it, right? Um, cloud storage, yeah. So basically, um,

it's kind of a one-off process right now. It's like we have a collection of data that we generally use, and if that thing works for the task at hand, I tend to reuse that file. And if it, if I don't need the whole file, I'll ha- say, "Claude, can you clean this up? Uh, can you, can you cut out the stuff I don't need, um, and push it to cloud storage for me?" Um, that data though, uh, the data's the hard part, right? I think the data, getting the data is the hard part, um, for, for any of these tasks, and the more realistic your data, the better the system is going to run,

uh, the better it's gonna like uncover realistic issues for you. So it's always the hard part, getting data.