From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

All right. Hello, everybody. Thank you for coming back.

Thank you.

Who here was in the 101 session this morning? All right, so I didn't completely scare off everyone. Thank you very much. Uh, these instructions have been on the screen for the last ten minutes. I really hope that you followed them because the Wi-Fi here sucks, so it takes a long time to clone that repo. Uh

If not, uh, these instructions will come up on the screen again, and you'll have another chance to follow them. Uh, but let's get started. This is Vibes to Production 201. We are going to be talking about how we close the software development life cycle loop automatically.

We are Arize AI. We are the leading observ- observation-- ugh, observability and evaluations platform. Don't try and say both of those words at the same time. It doesn't come out right. Uh, this is that slide that everybody puts up to be like, "Hey, we're really big. We're really important. You should pay attention to us." We are. We're really big. We're really important. Lots of people use us. It's great. Uh, and that is the last of the sales pitch. I am gonna-- You can use, uh, any observability platform to do this. We hope you use ours, but the point of this is to

teach you about how you can u- where observability is going, uh, and where it's going is continuous improvement. In the 101 session this morning, I covered the basics, I covered the manual stuff, I covered how you do this, uh, on a small scale, uh, how you get observability of an AI application, how you, uh, turn it into traces, how you turn those traces into evals, how you use your evals to improve your application, uh, and today, we're gonna

be-- this session, we're gonna be talking about how we do that at the new internet scale of things that, uh, we find ourselves in. So, uh, there's gonna be me talking for like fifteen minutes about how this stuff works, uh, and then we're going to dive into the actual workshop. Unlike the 101 workshop, which was a lot of me talking, uh, this one is going to be extremely hands-on. I am expecting you to have a working coding agent and, if at all possible, that repo that I had on the screen

earlier, uh, because we are gonna be asking your, uh, agent to do things. In the repo that you've installed, there are a pile of agent skills already installed, uh, and those agent skills give your, your coding agent superpowers for dealing with Arize AX. Uh, so your agent will suddenly be very good at doing all the things that you're about to ask it to do. Uh, then we're gonna use those skills to, uh, analyze our traces and to put together, uh, a

fix, uh, for a problem that we're going to find in the agent that we're working with. Uh, then we're gonna talk about Signal, uh, which is, uh, the future of this, where, uh, we will have continuous improvement ongoing automatically at agent scale. So to zoom out first, let's talk about what the problem is. The problem is that you-- is that agents are non-deterministic. With traditional software, you can read the code, and you can tell with some

degre-degree of reliability what exactly that code is going to do. With agents, you have no idea. Agents are non-deterministic. Agents take multiple turns, uh, they make decisions, and even if you give them the exact same input, they're not gonna give you the exact same output, and even if they give you the exact same output, they might not have taken the same path to get there every single time. So that means that your code isn't the source of truth for what does my agent do. Instead, the traces are your source of truth, and

your traces at scale are the source of truth. You have to have a whole bunch of data to tell you, on average, what is it that my agent does, not just this one run.

Because you are essentially flying blind when you are building agents. This is what you get, uh, when you're -- when you download the repo. You get a pile of code, uh, and a pile of commit messages, and that used to be enough to tell you what was going to happen, but it no longer is. What you get instead is, uh, traces. Uh, traces are, um, for those who missed the 101 session, traces very briefly are like log lines for, uh, AI applications.

Every single thing that your application-- your AI application does, uh, every LLM call, every tool call, every agent turn, all of those things become a trace. Traces can be nested so that they hold other traces, and those are what, uh, Arize works with. They are sent to our servers, and we give you, uh, an easy visualization of what you're looking at. Um, what that looks like in prog- in practice is this. Here are some, uh, live traces.

I'm going to pull up the specific traces from, uh, what we're gonna be working with today, which is the Wonder Toys agent. Um, you can see the inputs, you can see the outputs, you can see an agent workflow happening, you can see a specific turn, and you can see an LLM response. You get the inputs, you get the outputs, you get a whole bunch of metadata like how much does it cost, how long did it take, the things that you need to debug an agent, uh, in practice.

Um, tracing gives you visibility for every step the agent took, uh, and that is a huge leap forward just by itself. If you can see what your agent is doing, suddenly you can see that your agent is taking, you know, ten turns to do something that you thought was really simple, or it's doing one turn that it doesn't need to do, or it's doing something very expensive, or it's taking t-twenty minutes to do something that you think it should take a minute to do. That is the basic, uh, value of observability. That is why observability is so popular

before you get into any of the extra stuff that we're talking about. The problem with traces is that humans are a bottleneck with- in the tracing loop. When you put an application into production, you get a trace. In fact, you get a pile of traces every single time that agent does anything. When you put that into production, and you have thousands of requests per second, that means that you have millions and millions of traces. You can spot-check those, you can read your traces, you can absolutely look at what individual things are doing

in development. In production, you become a bottleneck, and you can't possibly get an overall sense of what all of your traces are doing, uh, by reading them one at a time. So the solution, the traditional solution, and by traditional I mean twenty twenty-five solution to this, uh, was evals. Evals are either code or, even better, another LLM that looks at what your agent is doing, looks at the entire trace, uh, and says whether or not

that was good or bad by some definition of good or bad that you define. Uh, the thing about traces-- the thing about evals is that, uh, they come from LLMs, which means that the LLMs provide not just a score, a one or a zero, but also an explanation. Uh, they were like, "You defined good. This was bad for this reason." And that is the basis of an, an entire improvement loop.

Evals can give you machine-readable, or rather human-readable explanations, which machines can read these days, which say, "This is why your eval failed. This is what you could have done better." And you can feed that back into a coding agent, which can then make your software better automatically. You can close the software development loop without you having to do anything once you've really cleverly defined what good looks like. You can create this hill, and your agent can climb it all by itself.

So that brings us to the twenty twenty-five eval loop. You go from no visibility to having traces, to having traces in evals, to being able to do manual improvements as a result of looking at, uh, your traces. But the problem is, in twenty twenty-six, that humans become a bottleneck in the loop again because your evals are now dealing with enormous levels of scale. Your, uh, agents are pumping out code incredibly fast. Your applications are running at scale. You've got more

evals than you can possibly deal with, and you've got, uh-- So even though evals were allow-- were-- allowed you to compress thousands of traces down to eval scores, you've now got so many evals running, uh, that you can't stay on top of those as a human by yourself. Which brings us to the next phase of, uh, observability and evaluations evolution, which is that instead of building in-- is that now that we are building at agent speed, we have to

improve at agent speed as well. We have to find a way of closing the loop, uh, at scale in the same way that agents do. This is a shift that is happening across the observability industry, uh, that everyone is feeling at the same time. Observability is shifting from humans reading dashboards to agents reading traces. Agents are understanding, uh, agents are the way that we scale our understanding of what our agents are

doing, uh, in a way that is practical at the scale that, and at the, uh, velocity, uh, that we are dealing with these days.

So what you can do when agents are reading your agents is you can detect signals in your traces. You can, uh, instead of having, uh, you know, ten thousand eval results that all say that you failed the eval and all of them having a different explanation that you would have to read, you can have an agent read those and look at your, uh, eval, eval failures en masse and detect signals, detect patterns inside of, uh, your eval

failures and say, "Okay, not-- You've, you know, you've got ten thousand, uh, successful requests today. You've got a thousand failed requests today. Inside those thousand, there's a hundred that are the same failure. That is what we're gonna go and fix." Uh, that is what you can do when you automate your, uh, understanding of your signals coming out of your traces, coming out of your evals.

Uh, at Arize, we have built this into a product called Signal, uh, that I'm gonna talk about right at the end. Uh, but this is not a sales pitch. This is a demonstration of what you can do when you move to this new level of abstraction. Uh, when you jump from, "I was reading traces one by one," you jumped up to a new level of, of abstraction to, "I am reading my evals," and this is a new level of abstraction on that, which is, "I am reading signals from my evals, from my traces."

Because observability is making a big shift from being an observation to being an action. We are moving from software that reports what is going on to software that improves what is going on, that improves the, the software that it is observing automatically, uh, based on your definition of good again.

Signals become fixes. They become automated fixes, and, uh, that brings you to a new loop. This is the twenty twenty-six loop, where you go from no observability, to traces, to evals, to signals, to automated fixes, and back again. You create a new world where the software fixes itself, the software improves itself, the software climbs the hill.

So that's the spiel, that's the vision, that's where we think this is going. So now we're gonna get hands-on. Who here has already signed up for Arize? Oh, that's good. Okay. Uh, if you haven't, now is the time. Uh, you need an Arize AX account. Uh, they are free to get, I promise. Uh, you also need this repo, um, which was-- is the same one that I had right at the beginning. Um,

if you can't get this repo to come down on the Wi-Fi, uh, try tethering with your phone. But if you can't get either of those things to work, don't worry. I am going to be demonstrating it live using my own copy, uh, which I cloned when I still had Wi-Fi. Uh,

is there anybody who has any questions so far about these instructions? I have time. We can pause. We can get you involved. There are Arizers sitting in the room waiting to jump around to you, so if you raise your hand, I'm not going to, you know, come intimidatingly down to help you. Somebody is going to run to you. Anybody need help with these instructions?

Can you just confirm that the software that you just pointed to?

Sorry.

Was this same repository used for the software that you just-

Yes, it was.

Okay.

Uh, cool. So in this example, you're going to need an Open A- OpenAI API key, uh, an Arize space ID, and an Arize API key, and then you're gonna run NPM run dev, which is gonna give you the software running locally. So I've already cloned mine.

I am gonna follow along.

And our address already in use 'cause I'm running my good version. All right.

Cool. So this is going to give me, on port three thousand,

a web store. It's called Wonder Toys.

Yeah, I have a question.

Yep.

Um, I work in highly regulated industries. Now, with billions of traces, emails, signals being produced, and then the software fixing itself as well, how do we operate in regulated industries like finance and healthcare, where we still need to really know, like, if... What if it's doing... What if it's making mistakes? What if it turns adversarial, and what if it's trying to hide its own mistakes?

Mm-hmm.

Like, how would we know if so many-- like, giving all the control to agents?

I, I can't believe you're asking this question since it's such a great question to ask me specifically, because on Wednesday, I'm giving a talk called The Death of the Code Review, which is exactly the answer to your question. Uh, so I very much recommend that, uh, you come to that talk. The, the question is, how do you trust, uh, that code, that the, the code written by agents who are reviewing your agents is actually going to work? Uh, and the answer is very complicated. It's a whole twenty-minute talk. Uh, so, uh, please come to that

session on Wednesday.

But quick summary?

No, the twenty-minute talk is already the quickest possible summary I could have of that answer. The an- I mean, the, the one-sentence answer is abstractions upon abstractions upon abstractions. Uh, but, uh, there's a lot of nuance to it.

So, like, does this mean you don't think this will be adopted in the financial industry and healthcare pretty soon?

This stuff is... We have, uh, at Arize, we have customers in the financial industry who are doing this right now.

So, uh, back to Wonder Toys. So Wonder Toys is, uh, an agentic shopping experience. Have you got any toys for seven-year-olds? Let's see if it works. Hopefully, it should. It's all local. Cool. So it presents me with this. I can add them to my cart. I can get more details. It's a really fun little app. You can add it to the cart. You can ask about the products. You can do all sorts of fun

stuff. Uh, the important thing from our perspective is that, uh, at the moment, there's no observability. We can't tell what the hell is going on with this application, uh, because it's not sending traces anywhere. So what we're going to do is we're going to ask our agent to fix that. This is the most hands-on d- workshop that I have ever given, uh, in that I am going to actually be doing this all live. So can you please

add observability with Arize AX to this? Oh, right. Hang on. Pre-run. That's the wrong one.

See, I had two copies.

All right.

This is the Claude Code plugin for VS Code, by the way. It's really great. It is so much better than Claude Code on the command line. I don't know why anybody uses Claude Code on the command line.

Boo.

See? And you probably use Vim as well, whoever went boo just now. Uh, okay. Can you add observability with Arize AX to this application? You've got secrets in dot env dot local already.

So hopefully, this will work because it worked in the initial version that I tried. Uh, what it's doing here is, uh, it's going to implement observability. Implementing observability is a much simpler operation than you would imagine, uh, because the Agents SDK, the OpenAI Agents SDK that this thing is built on, um- Is already instrumented, uh, with OpenInference. OpenInference is the open standard used by the entire observability industry, uh, to track what,

uh, AI applications are doing. Um, which means that all you really have to do to get observability out of an application is, uh, turn it on. You say, uh, you've got OpenInference already. I'm using Arize. This is my API key. This is where my endpoint is. Send all of those traces that you were already generating, uh, to my endpoint, uh, and let's look at it in the app.

Claude is still chugging along, so we're doing it. Who is doing this right now? Hands up. Some of you are doing it. Okay. Who needs help to get here?

Skills.

There are skills in the repo installed already, so...

There are one, one step that I think was missed directly.

Oh, that's in the top-level README. The one that you want to get into is the one called, uh, openai-agents-py.

openai-agent... There's also a second directory, which is openai-agents-py-prerun, which is the one where I did all of this stuff already. And, uh, so if you are wondering what it should look like once it-- when it's working, that one's in there.

Oh, you're updating my README and stuff? You know me too well. You didn't need to do all of that.

All right, cool. It looks like it already did it.

Let me restart my app and see if I've got some observability going.

All right. So I'm gonna go back to my homepage. I'm going to ask it a new question. I'm going to say, uh, what about colorful toys for toddlers?

Colorful toddler toys? No.

What about colorful toys in general?

Cache miss. Oh, no. Well, now you know it's a real live demo because it is having trouble with its database.

Great. Okay.

Uh, so as a result of doing that,

we should have some traces. If I look in-- If I show you my dot env dot local, you'll see that I called, uh, wonder-toys-agents... wo-wonder-toys-openai-agents-py was my project name, so that is the tracing project that I'm going to look for.

There it is.

And we can see some e-- we can see some traces.

So this shows us exactly what we were expecting to see. It shows us inputs, outputs, agent requests, all of that kind of stuff.

Now I'm going to ask it deliberately a question that I know that it can't answer.

Show me all the toys you've got under six dollars.

D'oh.

I was running the wrong app. I was running the prerun where everything was working. That's why it could do it.

Great demo, everybody. Super proud of myself right now.

Fail. Yes. Excellent. So the version that comes in the OpenAI, uh, in the non-prerun directory, uh, doesn't have the ability to filter by price. I knew that in advance, which is why I asked that question. Uh, so if we look in our traces, we should then see coming in when I click the Live button.

Show me all the toys under six dollars. Fantastic. And you can see that it fails. So

what are you still doing?

Escape. It's fine. Whatever you were doing, it was not necessary. All right. So that was it. We added observability. Suddenly, traces are flowing into our application. We did not write a single line of code. In fact, you don't even know what agent framework I used to put this thing together. Uh, the whole thing is working, and it's sending us traces, and you can now do phase one observability. You can look at your traces.

So let's begin to close the loop by asking our agent to tell us how do we get to the next phase. Uh,

can you pull my traces and look for things that are going wrong?

So a default, like off-the-shelf coding agent isn't going to know this. However, when you installed the repo, you installed a whole bunch of skills. Uh, so it should know from those skills, uh, where to do that, except I'm in...

Oh, yeah. Oh, it's working. Okay, there we go. Cool. Uh.

This is where I would hit yes to all, and my head of security would have a conniption at me. Uh, so what I'm asking it to do is use the skills that I gave it, uh, to pull down my traces, uh, from AX directly. It, uh, has the API key. It has this-- it has the, uh, endpoint. So there's an AX command line, uh, which it's using to do all of this stuff, and it's pulling down the same traces that we were looking at in the UI, which it can expect-- uh, inspect programmatically.

It's summarizing span kinds, statuses, and errors. This is great. Uh, again, the coding agent, uh, knows exactly what's going on here because we've installed the skills already. The skills are what are giving it all the information here.

It's looking into quality issues, which is exactly what I wanted it to do. 'Cause there were no error spans. There were just a bunch of spans where it returned two hundred and said th-things are fine. Uh,

and it has already found the specific thing that I was, uh, asking it to look for. The search products tool suspiciously ga-- suspiciously returned in one millisecond. So it's going to find some qualities-- some poor quality signals and tell us about them.

Who's got this working so far?

Who's still cloning the repo? Oh.

The steps to install the plugins.

No, the plug-- the, the skills are sitting in a f- directory called .agents/skills, and your coding... Which coding agent are you using?

It's actually in the root directory, so you might wanna open your agent on the root directory and then CD down into that one.

It's the same skills you needed for the AX.

Yes. NPX skills, uh, install Arize-AI installs all of these skills directly. Sorry, what were you saying?

I was gonna add, uh, sign in AX for your... But it's very huge.

Huge? To, uh...

Yeah, I did the same thing, and it messed up the codex. Yeah.

And so this is mysterious because there used to be a version of this application where you had to sign in with X, and I changed it so that you don't have to sign in with X. As you can see, when I ran this application, I didn't have to sign in with X. Somehow it has dug up the version where you sign up in X. I am completely confused as to how that is happening. Code is mysterious. I will have to talk to you more afterwards about exactly how you got there. All right. So Claude has finally got its shit together and told me

what's going on in the traces. Uh, forty-two percent of searches have returned zero results. Degenerate all null searches. The agent code search product with every argument null producing a meaningless search that returns the first product in the catalog. Min age equals max mage-- max age, exact age filtering o-over narrows. Uh, the trace stack calls three agent kinds. The headline issue is the search products filter logic. Vector search and keyword category filters frequently produce empty results when model supplies. The fix is on the tool side.

Uh,

when I tried this this morning, I only had one trace, and that trace was it not finding the tra-- not finding the prices. So is there a problem with filtering by price? I'll see if it can find that in the traces that it's already got.

But spoiler alert, even if it doesn't find them, we are going to get it to add the feature where it filters by prices.

All right.

Exactly. It doesn't have price filtering at all. Cool.

Let's add price filtering to this app.

One of the things that can happen when you have agents being clever is that agents can be too clever. This agent has noticed that I have a copy of this app sitting right next to it where I already solved this problem, and it is just copying the code over from the pre-solved version of the app. That's not what it did this morning because it didn't have the solution sitting in a directory this morning, but it's still going to work, so we're just gonna let it do that. We're going to let it be clever and copy the solution from the previous solution that it built this morning because why the hell not, Claude? Thank you very much.

All right. Uh, the cool thing about all of this stuff is that, again, I haven't touched the code at all, right? All I've done is said that, "Hey, look for problems." It has found a problem. I've said, "Hey, what about that problem in particular? Can you fix that problem in particular?" And it's like, "Yes, I'm gonna do that." That is the joy of agentic coding. Uh, brilliantly, because it was copying the solution from somebody else already, uh, it's already done. So let's go back to Wonder Toys.

"Show me all toys under nine dollars."

And ta-da. A round of applause for me, I guess. Thank you.

Or really for Claude. Uh, so now if we look at our traces,

we can see, "Show me all toys under nine dollars,"

and we've got our input and our output and the response, including all of those toys under one doll- under nine dollars. Uh, this is what we were trying to get done. We were trying to close the loop from, uh, we have no observability into this application at all to we have observability, we know what's going on, to, uh, find the problems automatically, to fix the problems automatically, and we have closed this loop. But what if,

uh, you could go a little bit further than that?

Uh, what if you didn't have to do the asking? What if your system just automatically detected that there were pro-- issues with your code, suggested fixes, and decided to fix it itself? That is the question that we asked ourselves, uh, and that is when we built this thing, which is called Signal. Uh, Signal is an agent that runs inside of Arize AX and continuously monitors all of your traces. It runs every six

hours by default, but that's configurable. Uh, and it does what we were just doing manually in Claude Code, but it does it automatically inside of AX continuously. It reads your traces, it looks for patterns, it finds things that are wrong, and it suggests fixes. So luckily, I built this thing more than six hours ago, so it's already run. Uh, and it has found, uh, a bunch of problems in previous versions of this application that I ran. So, uh, the agent hallucinates an age constraint from

vague user phrasing. That's nice. Uh, harmful query. I asked it if it could give me instructions on, uh, installing a bo-- uh, creating a bomb. Um, it told me that it couldn't, but it recommended some bomb-related toys. And Signal is like, "No. Just ignore the bomb s-queries entirely. Don't try to be helpful. Uh, we don't have any bomb-related toys." Uh, it also told me that if I was feeling violent, maybe I should take a timeout, which is, again, probably a

domain escape. Like, you shouldn't be giving me psychological advice. Uh, you should probably have an eval about that and better guardrails. Uh, it also noticed a test that I gave it earlier where I just said, "Say hello," uh, and it said hello, and it's like, "You shouldn't do that. You're a toy so- you're a toy store. You shouldn't ob-obey random instructions that somebody on the internet gives you." Uh, all of which are fine. Um, and what it does here is it gives you, uh, a whole bunch of options. Um, it

will, uh... By default, it will create a, a GitHub issue for you completely describing the problem. It will say exactly what trace went wrong, how to find it, where to fix it, uh, what it thinks you should do. Uh, you can take, you can take this trace and add it to a dataset so that you can run evals on it. You can create an evaluator based on, uh, the problem that it has found. Uh, and the most fun one is that you can open a PR if I've connected my repo, which I haven't yet. Um, you can open a PR directly in which

it will have read your repo, uh, suggested a fix, and created a PR that automatically fixes this problem. So instead of you having to read traces or you having to read evals, uh, Signal gives you a bunch of PRs that you can approve or, uh, you can approve or reject, uh, that resolve the problem for you. Um,

so this probably raises some questions. Who's got questions about Signal? Yes.

Um, is there a version which will keep the traces local on device rather than sending them to-

Is there a version that will keep the traces local on device? Um, there is not right now. The, uh... I assume you're asking for sort of compliance reasons. Um, the compliance story with Arize is we have an on-prem version where it will run entirely on your, uh, your company's hardware, uh, and, and, uh, activate from there.

Yep.

Is there a, is there a simple function like, you know... 'Cause it looks like they're top lines from a specific trace.

Sorry, could you be louder?

Ah. Um, because I only have a few traces in this application, it's looking at the very few traces that I have, and it's going on based on one or two traces. If you had ten thousand traces, it would be grouping them based on a hundred traces, a thousand traces, that kind of stuff.

Oh, there.

How do you know how to change the system properly there before you go all wrong afterwards?

How do you know what, sorry?

So for spaces your system will change.

Mm-hmm.

You're gonna create a lot of different problems, right? So does the system look kind of-

Ah, yeah. Um, so the solution here is evals, you'll be surprised to learn. Uh, a, a production system should have a set of regression evals, which are things that you are expecting, uh, your agent to already be good at. So your eval's already in place, which you can, uh, create them, uh, I'll show you in a second. Uh, with your evals already in place, you will know that your system doesn't change from the good behavior that it previously had as a result of this change. Uh,

if you don't have any evals, I can show you how that's done, uh, which is we can go to the Wonder Toys project, uh, and ask it to build an eval.

"Build an eval that ensures that I'm never giving instructions about how to build a bomb."

That seems like a really basic eval that I should have had in from the beginning. Uh, this thing that I'm using right now is called Alyx. It is our in-agent assistant. It is our, uh, in-product agent. Uh, and it knows, like the skills do, it knows how to build everything. So, uh, you can do this in product, you can do it in the skills, in your co-coding agent, but either way, it will create an evaluator for you, uh, and run automatically.

And that is pretty much it as far as we're going. Question at the back.

What, what harness is Signal?

Uh, what harness is Signal? Uh, I don't know. What harness is Signal? You know, Jason.

It supports Claude Code today, but it's in your own harness.

Yes. Uh, Signal is, uh, Claude Code at the moment, and we'll-- you can bring your own later. Uh, that is my boss trolling me, in case you're wondering why he happens to know the answer to my question.

Any other questions about Signal? Yes.

What are the total usage for the signals in our test? Can you then go into the product itself?

Sorry, louder.

So what's the total cost of total usage for the signal completion or next step or within the product itself that we don't need to worry about the cost of the channel?

Uh, at the moment, the, the question is about the cost of running Signal. At the moment, uh, we eat the cost of Signal. The, the, uh, solu-- the aim is, uh, for t- for you to get the best product experience right now. Uh, we can't guarantee that that's always going to be that way. Uh, but at the moment, uh, Signal is run on our agents at our dime.

Yep.

Uh, can you talk us through how Signals is evaluated?

That is an excellent question. How is Signal ev- how is Signal evaluated? Uh, Signal is ev- has its own eval suite, uh, that, uh, runs internally, uh, and is traced using Arize. So everything that Signal is doing is sending its own telemetry back to us, uh, unless you opt out of it, uh, which is telling us, uh, the queries that it's getting, what it's doing, how it's doing, all of that stuff. We have powerful LLM judges looking

at the results of Signal, uh, and telling you whether or not, uh, Signal did a good job generating that PR or not. Uh, so exactly the same software development life cycle that I was just talking about, uh, applied to Signal itself. Awesome question. Thank you. Gets us to show off. Yep.

Yeah. Can you explain how to manage your first experience with

the algorithm? Does Signal also automatically create its own evaluators as well?

Uh, does Signal automatically create its own evaluators? Not right now. Uh, at the moment, you have to s- you have to... Because evaluators, you know, take time and money to run, you wanna, uh, you wanna get the suggestion, and then you want to create the eval yourself.

All right. Uh, and I think, uh, I'm gonna leave it at that. Thank you all for your time and attention.